Adversarial Feature Map Pruning for Backdoor
Authors: Dong HUANG
Qingwen Bu
Venue: ICLR 2024 poster (2024)
Keywords: Backdoor Defense
Data Poisoning
Primary area: general machine learning (i.e., none of the above)
TL;DR: We propose a backdoor defense method that effectively detect and pruning feature maps that propagate backdoor information.

Abstract: Deep neural networks have been widely used in many critical applications, such as autonomous vehicles and medical diagnosis. However, their security is threatened by backdoor attacks, which are achieved by adding artificial patterns to specific training data. Existing defense strategies primarily focus on using reverse engineering to reproduce the backdoor trigger generated by attackers and subsequently repair the DNN model by adding the trigger into inputs and fine-tuning the model with ground truth labels. However, once the trigger generated by the attackers is complex and invisible, the defender cannot reproduce the trigger successfully then the DNN model will not be repaired, as the trigger is not effectively removed. 

In this work, we propose Adversarial Feature Map Pruning for Backdoor (FMP) to mitigate backdoor from the DNN. Unlike existing defense strategies, which focus on reproducing backdoor triggers, FMP attempts to prune backdoor feature maps, which are trained to extract backdoor information from inputs. After pruning these backdoor feature maps, FMP will fine-tune the model with a secure subset of training data. Our experiments demonstrate that, compared to existing defense strategies, FMP can effectively reduce the Attack Success Rate (ASR) even against the most complex and invisible attack triggers (e.g., FMP decreases the ASR to 2.86% in CIFAR10, which is 19.2% to 65.41% lower than baselines). Second, unlike conventional defense methods that tend to exhibit low robust accuracy (that is, the accuracy of the model on poisoned data), FMP achieves a higher RA, indicating its superiority in maintaining model performance while mitigating the effects of backdoor attacks (e.g., FMP obtains 87.40% RA in CIFAR10). Our code is publicly available at: https://github.com/hku-systems/FMP.

Published as a conference paper at ICLR 2024
ADVERSARIAL FEATURE MAP PRUNING FOR BACK- DOOR
Dong Huang'", Qingwen Bu
'University of Hong Kong, ?Shanghai AI Laboratory, *Shanghai Jiao Tong University dhuangecs hku qubu0l@sjtan edu. en
ABSTRACT
Deep neural networks have been widely used in many critical applications, such as autonomous vehicles and medical diagnosis. However, their security is threatened by backdoor attacks, which are achieved by adding artificial patterns to specific training data. Existing defense strategies primarily focus on using reverse engi- neering to reproduce the backdoor trigger generated by attackers and subsequently repair the DNN model by adding the trigger into inputs and fine-tuning the model with ground-truth labels. However, once the trigger generated by the attackers is complex and invisible, the defender cannot reproduce the trigger successfully then the DNN model will not be repaired, as the trigger is not effectively removed.
In this work, we propose Adversarial Feature Map Pruning for Backdoor (FMP) mitigate backdoor from the DNN. Unlike existing defense strategies, which focus on reproducing backdoor triggers, FMP attempts to prune backdoor fea- ture maps, which are trained extract backdoor information from inputs. After pruning these backdoor feature maps, FMP will fine-tune the model with a secure subset training data. Our experiments demonstrate that, compared to existing defense strategies, FMP can effectively reduce the Attack Success Rate (ASR) even against the most complex and invisible attack triggers (c.g., FMP decreases the ASR to 2.86% in CIFAR1O 10, which is 19.2% to 65.41% lower than basclines). Second, unlike conventional defense methods that tend to exhibit low robust accu- racy (that is, the accuracy of the model poisoned data), FMP achieves a higher RA, indicating its superiority in maintaining model performance while mitigating the effects of backdoor attacks (e.g., FMP obtains 87.40% RA in CIFARIO), Our code is publicly available at: https: /ttp://gttttt com hww-systtttteetaeeeeeeyaaeaaaass ems /PMP.
INTRODUCTION
Deep neural networks (DNNs) have become cornerstone technology in numerous fields, such as computer vision (Russakovsky et al., 2014), natural language processing (Devlin et al., 2019), speech recognition (Park et 2019), and several other applications (Bojarski et al., 2016; Rajpurkar et al., 2017). The effective training of DNNs generally demands extensive datasets and considerable GPU resources to achieve SOTA performance. However, substantial number of DNN developers may lack access to these resources, which subsequently leads them to rely on third-party services for training their models (e.g.. Google Cloud Google (2023), AWS Amazon Web Services (2023), and Huawei Cloud Huawei Technologies Co. (2023)), acquiring datasets (e.g.. DataTurks DataTurks (2023)), or directly downloading pre-trained models (e.g., Hugging Face Face (2023)).
Although using third-party services offers a practical solution for DNN developers, it simultancously introduces potential security risks. Specifically, third-party may be involved the data collection, model training, and pre-trained model distribution process, which may introduce malicious backdoor triggers (Chen al., 2017; Gu et al., 2017). For instance, once a developer uploads their dataset to a third-party service for training. the service provider could potentially revise the dataset by injecting poisoned samples containing hidden backdoor triggers. These poisoned samples, designed to blend in with the rest of the dataset, then used in the training process, embedding the backdoor triggers into the model's ' architecture. The presence of backdoor triggers in DNNs can have serious
"Equal contribution.

---

Published as a conference paper at ICLR 2024
implications, particularly in security critical systems where model integrity is crucial to preserving human life and safety (Gu et al., 2017; Wang et al., 2019a).
To mitigate the backdoor threat, researchers have proposed variety of defense mechanisms. Exist- ing defense methods against backdoor attacks can be grouped into two main categories: detection methods and repair methods (Wu et al., 2022; Li et al., 2020a). Detection methods rely on internal information from the model (e.g. neuron activation values Chen et al. (2018)) or model properties (e.g., performance metries (Chen al., 2022; Zheng et al., 2022a)) determine whether model has been compromised by a backdoor attack, or whether a specific input example being processed by the model is a backdoor instance (Zheng et al., 2022b). Detection techniques are essential for pinpointing risks and highlighting security vulnerabilities. However, upon identifying a backdoor model, it must be promptly eliminated to regain trustworthiness and reliability.
To tackle this challenge, researchers have introduced repairing methods that go beyond detection and aim to remove backdoor triggers from the compromised models. One notable example is Neural Cleanse (NC) (Wang et al., 2019a), technique that leverages reverse engineering to first reproduce the backdoor trigger. Once the trigger has been identified, NC injects it into the dataset along with its corresponding ground-truth labels, enabling the fine-tuning of the model to eliminate the backdoor trigger's impact. However, recent evaluation results (Chen et al., 2017; Li et al., 2020b; Nguyen & Tran, 2021) reveal that NC and similar reverse engineering- based methods successful in tackling simple backdoor triggers. In contrast, complex triggers (e.g. those involving intricate patterns transformations) pose greater challenge, making it difficult to reproduce and remove them. Consequently, models containing such sophisticated triggers may not be adequately repaired, leaving them susceptible potential security breaches.
predominantly
Recently, some researchers (Liu et al., 2018; Zhao & Lao, 2022) try to address this problem by analyzing the feature map's behavior to remove the backdoor from the model since they notice that backdoor-related feature maps exist, which demonstrate different characteristics from other normal feature maps. Specifically, feature maps are essential components of DNNs, responsible for extracting various features (e.g., color and texture information) from input samples (Zeiler & Fergus, 2013). The model then relies on these feature maps to extract features from the inputs and to make its final predictions. In the backdoor model, some backdoor feature maps extract backdoor information from the inputs (Zhao & Lao, 2022). Based on this observation, they believe that once the backdoor feature maps in the model are erased, the backdoor trigger will also be removed from the model. However, detecting backdoor feature maps from the model is challenging since the DNN developers, who may download pre-trained models from third party, are unaware of the information about possible backdoor triggers injected in the model. Subsequently, they can not directly add triggers into the input samples analyze and detect backdoor feature maps.
To address this challenge, we propose the Adversarial Feature Map Pruning for Backdoor (FMP), novel approach that focuses on identifying and eliminating backdoor feature maps within the DNN. Instead of directly reproducing the backdoor trigger and then using it to detect backdoor feature maps in the model, FMP uses adversarial attack to reproduce the features extracted by each feature map. By adding these features into the inputs, we can feed these inputs into the model to identify the backdoor feature maps, which can then be mitigated to secure the DNN model. Our experiments reveal that initializing these backdoor-prope propagating feature maps and fine-tuning the model can effectively neutralize the backdoor from the model.
To validate the effectiveness of the FMP, we conduct extensive experimental evaluations on multi- ple benchmark datasets, including CIFAR-10, CIFAR-100 (Krizhevsky & Hinton, 2009), and GT- SRB (Stallkamp et al., 2012), under diverse attack scenarios. In our experiments, we consider various backdoor triggers with different complexities and compare the performance of FMP against several ofte-offthhearr art backdoor defense methods. Models are evaluated with three primary metries: Accuracy, Attack Success Rate (ASR), and Robust Accuracy (RA).
Our experiment results demonstrate that FMP consistently achieves lower ASR, higher accuracy, and improved RA compared to baseline methods across different datasets and attack scenarios. For instance, on the CIFAR-10 dataset, FMP outperformed baselines by achieving a significantly lower ASR (e.g., FMP decrease 19.2% ASR average compared with baselines in CIFARIO). and improved RA (c.g., FMP increase 16.92% RA on average in CIFAR1O), indicating its effectiveness in remov- ing backdoor triggers without compromising the model's 's performance on benign samples.
2

---

Published as a conference paper at ICLR 2024
In nutshell, our contribution can be summarized follows:
We propose a novel defense strategy FMP to mitigate backdoor triggers from the model.
We conduct substantial evaluations to investigate the performance of FMP. The experiment results show that FMP can significantly outperform previous defense strategies.
RELATED WORK
2.1 BACKDOOR ATTACKS
According to the threat model, existing backdoor attack methods can be partitioned into two general categories, including data poisoning and training controllable.
Data poisoning attacks entail manipulating training data to compromise machine learning models. Techniques in this realm aim enhance the stealthiness and efficacy of attacks by designing trig- gers with specific characteristics. These triggers vary in visibility, locality. additivity, and sample specificity. Visible triggers, such as those in BadNets (Gu al., 2017), visibly alter inputs, while invisible ones, like Blended (Chen et al., 2017), remain covert. Local triggers, as seen in Label Con- sistent Attack (Turner et al., 2019), affect small input regions, whereas global triggers, e.g., in SIG (Barni et al., 2019), impact broader areas. Additive triggers, such as Blended (Chen et al., 2017), add patterns to inputs, while non- non-additive ones, like LowFreq (Zeng et al., 2021), involve more in- tricate manipulations. In contrast poisoned samples, SSBA (Li et al., 2020b) customizes triggers for each specific input.
cample-agnostic triggers that which remain constant across all
Training controllable attacks (e.g., WaNet (Nguyen & Tran, 2021)) involve attacker having si- multaneous control over the training process and data, enabling them to jointly learn the trigger and model weights. This allows for potentially more effective and stealthy backdoor attacks.
The described backdoor attack presents a major threat to the reliability of DNN models, compro- mising their accuracy and leading harmful misclassifications. To tackle this issue, introduce FMP, a new defense method aimed at detecting and neutralizing the impact of backdoor attacks by identifying and eliminating backdoor feature maps within the DNN model.
2.2 PRIOR WORK ON BACKDOOR DEFENSES.
The research community has proposed a variety of defense mechanisms to detect and mitigate back- door attacks in deep neural networks. These defenses can be broadly categorized into two groups: data-centric and model-centric approaches.
Data-centric defenses primarily focus detecting and cleansing poisoned data points in the training dataset to prevent the model from learning the backdoor trigger. One such technique the AC proposed by Chen et al. (2018), which identifies and removes poisoned data points by clustering activations of the penultimate layer. However, once the DNN developers rely on the third party to train the DNN model, they can use these strategies remove backdoor samples since they can not interfere with the third-party platforms.
Model-centric defenses aim to address backdoor attacks by directly analyzing and modifying the trained model itself. NC (Wang et al., 2019a) is an example of such defense, which identifies and mitigates backdoor triggers by assessing anomaly scores based on adversarial perturbations. How- ever, its effectiveness diminishes when dealing with complex and imperceptible triggers, limiting its utility for developers in reproducing and removing such triggers successfully. The Pine-nPunnng (Liu et al., 2018) approach, as mentioned earlier, removes backdoors by pruning redundant feature maps less useful for normal classification. NAD (Li et al., 2021a) identifies backdoor neurons by ana- lyzing the attribution of each neuron's 's output with respect to the input features. Nonetheless, the absence of backdoor or poison samples from injected datasets can impede accurate analysis, reduc- ing their efficacy in trigger removal. Several recent studies (Liu et al., 2019; Barni et al., 2019; Chen et al., 2019; Guo et al., 2019; Fu et al., 2020; Wu & Wang, 2021; Guo et al., 2021; Bu et al., 2023; Li al., 2023; Liang al., 2023) also proposed to defense backdoor for deep neural networks.
3

---

Published as a conference paper at ICLR 2024
3 METHODOLOGY
Notations For ease of discussion, this section defines the following notations for DNNs and feature maps: is DNN parameterized by 0, and there N feature maps Σi Sg in the model. The i-th feature map in the DNN is denoted as fj. (ax) denotes the feature map i-th output when the DNN input is 2. The x' means the input x added the feature generated by the corresponding feature map.
3.1 MOTIVATION
The primary goal of our pruning defense strategy is detect backdoor feature maps in DNN model. As previously mentioned, intuitively, we should use backdoor samples and clean samples fed into the model to detect these backdoor feature maps. However, since DNN developers do not have access to backdoor samples, they cannot directly detect backdoor feature maps using this approach. This limitation calls for a defense mechanism that can operate effectively without the need for backdoor samples.
The main idea behind our proposed defense mechanism is to generate potential poison samples by reversing the specific features associated with each feature map. Since each feature map in the model used to extract features from the DNN model, and in the injected model, there are some backdoor feature maps that are used to extract backdoor information for the poison samples. By feeding these potential poison samples into the model, we can observe the inference accuracy for each feature map. Since backdoor feature maps are designed to extract backdoor information from the inputs, causing the classifier to return the target label regardless of the true label, we hypothesize that potential poison samples generated by backdoor feature maps will lead significant change in inference accuracy when processing their corresponding potential poison samples.
In summary, the motivation behind defense strategy is to detect backdoor feature maps without backdoor samples by generating potential poison samples and analyzing the inference accuracies with these samples. This enables to identify backdoor feature maps, which be mitigated to secure the DNN model. With this motivation in mind, we propose Adversarial Feature Map Pruning for Backdoor (FMP) as the defense mechanism achieve this objective.
3.2 FEATURE REVERSE GENERATION
In this section, we employ reverse engineering to generate features that will be extracted by each feature map in the DNN. These features will be added to the inputs, which will then be used to detect backdoor feature maps. The detailed implementation of the Feature Reverse Generation is provided in Algorithm 1. Specifically, for each feature map Sj in the DNN model, our objective is generate the corresponding features that the feature map is intended to extract by adversarial feature map attack. To accomplish this, we use reverse engineering approach that maximizes the difference between the original input and the perturbed input x!' in the feature map space. In Algorithm 1, referred to as FRG, we have adapted the FGSM attack (Goodfellow et al., 2014) to operate at the feature map levels. The overall time required by this method can be expressed as follows:
T= L. forward N₂ backward
where represents the total number of layers the Deep Neural Network (DNN), and N, denotes the number of feature maps in the i-th layer. The term "forward" corresponds to the time taken for one forward step, assuming each forward pass takes an equal amount time. Similarly, the term "backward" corresponds to the time taken for one backward step, assuming each backward pass also takes an equal amount of time.
3.3 REMOVE BACKDOOR FROM THE DNN
After obtaining the reversed samples using the method described in Algorithm 1, we feed these potential poison samples into the model to obtain feature map inference accuracy. Then, based on the accuracy list, will return the potential backdoor injected feature maps. After detecting the backdoor-relan related feature maps, our next step is to remove the backdoor from the DNN, To achieve this, follow two-step process: initialization and fine-tuning.
4

---

Published as a conference paper at ICLR 2024
Algorithm 1: Feature Reverse Generation
input : fe: DNN model; (X.3"): Clean Dataset; e: maximum allowed perturbation; p: First
N/p feature maps wil be pruned.
output: List: Backdoor feature map list.
Function Inference fo: (X Initialize an empty list Correct_Jist.
Start traversal over all feature maps:; for i in range( (onge(())r) do Initialize correct counter O;
for x, in (X,J) do = FRG u, steps, a, e);
fo()');
y y then
correct += 1;
Append correct Correct_list,
12
Sort CorrectJist in ascending order;
13
return First N/p elements from Correct
# We suppose feature maps with lower FRG accuracy are backdoor-related featurr maps.
Function FRG z)):
16
Initialize: x' = x + random noise:
Calculate the loss: loss = ((f((z)) fa()
17
Calculate the gradient: grad = Óloss/Ocs 0x;
19
Update: x' = x + e sign(grad)
Apply image bounds: x² clamp(æ',
min min
0, max 1);
21 return z';
Initialization. In this phase, we start by setting the weights of the identified backdoor feature maps zero, effectively neutralizing their influence the model's output. By doing this, eliminate the backdoor information these maps were designed to capture. Additionally, we adjust the biases of these feature maps keep their output close zero during forward pass. This initialization process helps mitigate the impact of malicious backdoors on the model's predictions, lowering the risk of targeted misclassification.
Tone-Tuning. After initializing the backdoor feature maps, the entire DNN model undergoes fine- tuning. This process adjusts the model weights with a reduced learning rate and subset of ac- curately labeled samples. Fine-tuning enables the model to adapt to the new initialization while maintaining its performance on clean samples. This ensures the model's predictive accuracy on clean inputs and bolsters its resilience against potential attacks.
4 EVALUATION
Experimental Setting In this work, we use BackdoorBench (Wu et al., 2022) as a benchmark to evaluate the performance of FMP. Since most of the existing backdoor-related literature focused on image classification tasks (Chen et al., 201 7; 2022; Devlin et al., 2019; Zheng et al., 2022b; Zhao & Lao. 2022: Wang et al., 2019b; Tran et al., 2018; 8; Nguyen Tran, 2021), we followed existing re- search to the choice of CIFAR1O. CIFARIÓ0 (Krizhevsky & Hinton, 2009), and GTSRB (Stallkamp al., 2012) datasets to evaluate FMP's performance. Similarly to baselines, we chose ResNet18 (He et al., 2015) our evaluation model, as it has been systematically evaluated by BackdoorBench (Wu et al., 2022), widely used by baseline approaches, and also widely used by vision tasks (e.g.. image classification (He et al., 2015), object detection (Ren et al., 2015)), so we believe evaluating FMP in these data sets can provide a comprehensive and meaningful assessment of its performance. In our experiments, we selected five of-te-o-thhe-a art backdoor attack strategies, which are systematically evaluated and implemented by BackdoorBench, as baselines to evaluate the effectiveness of our de- fense strategy. These attack strategies consist of BadNets (Gu et al., 2017), Blended (Chen al.,
5

---

Published as a conference paper at ICLR 2024
2017). Low Frequency (Zeng et al., 2021), SSBA (Li et al., 2020b), and WaNet (Nguyen & Tran, 2021). We train the CIFARIO and CIFARIO0 datasets with 100 epochs, SGD momentum of 0.9, learning rate of 0.01, and batch size of 128, using the CosineAmeaaligLLR scheduler. The GTSRB dataset is trained with 50 epochs. We set the poisoning rate to 10% by default. To ensure fair com- parisons, we adhere to the default configuration in the original papers, including trigger patterns, trigger sizes, and target labels.
Defense setup BackdoorBench has implemented many effective defense strategies (Zhao & Lao, 2022; Liu et al., 2018: Wang et al., 2019a; Li et al., 2021a; Tran et al., 2018) are implemented by BackdoorBench, we believe that taking these defense strategies as our baseline can demonstrate FMP's effectiveness. However, some of these defense strategies require DNN developers to have access to the backdoor trigger (e.g., AC (Chen et al., 2018), Spectral (Tran et al., 2018), ABL (Li al., 2021b), D-BR (Chen al., 2022) and DDE (Zheng et al.. 2022b)), or to modify the model training process (e.g., ABL (Li et al., 2021b), DBD (Huang et al., 2022)) which is not possible in our defense scenarios. Therefore, we exclude these strategies from our baseline comparison.
Finally, we select six widely used defense strategies that align with our defense goals: Fine-tuming (FT) retrains the backdoor model with subset clean dataset to remove the backdoor trigger's ef- fects. Fine-pruning (FP) (Liu et al., 2018) prunes backdoor feature maps to remove backdoors from the model. Adversarial Neuron Pruning (ANP) (Wu & Wang, 2021) selectively prunes neurons associated with the backdoor trigger while maintaining performance. Channel Lipschitz Pruning (CLP) (Zhao & Lao, 2022) is a data-free strategy that analyzes the Lipschitz constants of the feature maps to identify potential backdoor-related feature maps. Neurul Atlention Distillation (NAD) (Li et al., 2021a) leverages attention transfer to erase backdoor triggers from deep neural networks, ensuring that the model focuses the relevant neurons for classification. Neural Cleanse (NC) identifies and mitigates backdoor attacks in neural networks by analyzing the internal structure of the model and identifying patterns that are significantly different from the norm, which may indicate the presence of backdoor.
In order to repair the model, we adopt configuration consisting of 10 epochs, a batch size of 256, and a learning rate of 0.01. The CosineAAnealingg Gradient Descent (SGD) optimizer with a momentum of 0.9 for the client optimizer. We set the default ratio of the retraining data set at 10% to ensure a fair and consistent evaluation of defense strategies. For the CLP, we configure the Lipschitz Constant threshold (u) to be 3, the pruning step to 0.05, and the maximum pruning rate to 0.9, which is consistent with BackdoorBench and its original paper default settings. In the case of ANP, we optimize all masks using Stochastic Gradient Descent (SGD) with a perturbation budget (i.e.. e) of 0.4. For FMP, we set p equal to 64 and as 1/255. All other configurations are maintained as specified in the respective original publications to guarantee rigorous comparison of the defense strategies.
scheduler is employed alongside the Stochastic
Evaluation Metries
To evaluate our proposed defense strategy and compare it with SoTA meth- ods, we utilize three primary evaluation metrics: clean accuracy (Acc), attack success rate (ASR), and robust accuracy (RA). Acc gauges the defense strategy's ' performance on clean inputs, reflecting whether the model's original task performance remains intact. ASR assesses the defense strategy's effectiveness against backdoor attacks by measuring the rate at which backdoor triggers succeed in causing misclassification. RA a complementary evaluation metric that focuses on the model's performance in correctly classifying poisoned samples despite the presence of backdoor triggers.
4.1 EFFECTIVENESS
The evaluation results are shown in Tab. 1. In analyzing experiments with the CIFAR10 dataset, it evident that FMP consistently maintains low Attack Success Rate (ASR) alongside a high Robust Accuracy (RA). Specifically, FMP achieves average ASR of 2.86%, significantly reduc- ing the initial 19.2% ASR in CIFÁR1O. Additionally, the RA of FMP surpasses that of baseline strategies, reaching 87.40 87.40% which is approximately 16.92% higher than the average RA of baseline approaches. This indicates the effectiveness of our approach in mitigating backdoor attacks and en- hancing model performance on poisoned data, demonstrating applicability in real-world scenarios.
We can also observe that standard fine- teeitnnigg (FT) shows promising defense results against several attacks, such as BadNets, where it achieves an ASR of 1.51 % and an RA of 92.4%%. However, it fails

---

Published as a conference paper at ICLR 2024
Table 1: Performance comparison (%) of backdoor defense methods on CIFARIO, CIFARIOO, and GTSRB datasets with PreActResNet18 model. We perform evaluations under five advanced attacks with a poison rate of 10% and the retraining data ratio set to 100%. The c and p are set to 1/255 and 64 respectively (Algorithm l). FMP achieves leading performance across various settings.
BadNets ASR
Blended ASR
Low Frequency Acc ASR RA
SSBA ASR
WaNet ASR
AVG ASR,
Methods
Ace
RA
Acc
RA
Acc
RA
Acc
RA
Acct
RA
Benign
91.94 .494 97.21
93.44 99.95
93.44 99.39
92.94 98.80
91.53 98.83
92.65
98.84
FT
93.35
1.51 92.46
93.54
95.63 4.16
93.41 77.89 20.67
93.28 71.57 26.52
94.12 22.30 74.53
93.54
53.78 43.67
FP
91.62
16.06 79.24
93.21 96.84 3.04
93.24 96.73 2.79
92.88 82.92 16.00
91.30
0.62 84.91
92.45
58.63 37.20
CLP
91.20
94.77 5.01 93.29 99.83 0.17
92.47 99.10 0.82
88.27 9.96 76.22
89.03
53.84 41.88
90.85
71.5 24.82
ANP
91.22
73.36 26.16
93.25 99.44 0.56
93.19 98.03 1.88
92.92
68.59 29.13
90.81
1.93 88.98
92.28
68.27 29.34
NC
89.11
1.32 89.17
93.23
99,90 0.10
90.67 2.26 86.24
90.33
0.53 86.72
90.54
86,91 12.38
90.78
38. 18 54.92
NAD
92.97
1.10 92.39
92.18 44.98 42.60
92.32
13.43 77.03
92.22
38.18 57.18
94.16
12.61 83.22
92.77
22.06
70.48
Our
91.67
1.67
91.71 91.85
6.44
74.43
91.77 1.90 90.52 91.92
2.89
88.59
93.42
1.38 92.16
92.13
2.86
87.40
(a) CIFAR-10
BadNets
Blended
Low Frequency
SSBA ASR
WaNet
AVG
Methods
Acc
ASR
RA
Acc
ASR
RA
Ace ASR RA
Acc
RA
Acc
ASR
RA
Acct
ASR.
RA
Benign
67.33
93.61
70.19 99.84
69.56 97.47
69.24 98.12
65.67 97.50
68.39
97.31
FT
69.90
1.38 67.18
70.32 89.36 6.57
69.93 45,90 35.95
69.30 58.14 28.45
71.16
6.41 63.64
70.12 40.23 40.35
FP
67.59
23.09 54.04
69.44
93.56 4.03
68.73
82.60 10.84
68.25
76.46 14.52
66.33
81.47 11.26
68.06
71.44 18.94
CLP
59.74
74.79 19.42
68.56 99.52 0.37
65.46
92.88 5.18
68.40
89.57 7.44
60.06
6.93 54.53
64.44
72.74
17.38
ANP
66.95
15.36 58.05
70.05
97.70 1.70
69.47 55.00 32.00
68.92
89.87 7.54
64.23
0.23 60.55
67.92
51.63
31.97
NC
64.67
0.10 64.38
64.16
1.14 34.63
65.25
2.28 53.91
65.16
2.17 56.86
67.09
1.29 62.92
65.27
1.39
54.54
NAD
68.84
0.31 67.63
69.24
81.07 10.16
69.33
31,78 44.41
68.39
30.82 42.24
71.57
17.41 57.72
69.47
32.27
44.43
Our
66.25
0.09
67.23
67.27
0.41
40.49
66.32
0.18 65.83 66.77
0.39 60.61
68.79
0.18
66.84
67.08
0.25
60.20
(b) CIFAR- 100
BadNets
Blended ASR
Low Frequency Acc ASR RA
SSBA ASR
WaNct ASR
AVG
Methods
Acc
ASR
RA
Acc
RA
Ace
RA
Acc
RA
Acct
ASR. RA
Benign
98.04
96.38
19.
100.00
98.21 99.96
97.73 99.72
98.66 97.59
198.17 98.72
FT
98.60
0.49 98.11
98.19
99.61 0.34
98.39 92.94 5,43 97.63 96.75 2.99
99.20
0.73 98.31
98.40 58. 10 41.03
FP
97.79
0.04 65.20
98.04
100.00 0.00
97.81 99.25 0.45
96.90 99.64 0.34
98.80
0.05 13.23
97.87 59.79 15.84
CLP
96.42
88.11 11.67
97.68
100.00 0.00
97.09 97.74 1.82 97.13 99.42 0.57
62.17
99.91 0.08
90.10 97.03 2.83
ANP
96.55
15.71 82.06
98.22 99.96
0.02
98.27 69.20 27.66 97.10 99.51 0.48
98.67
1.65
97.72
57.20 41.30
NC
97.75
0.00 97.75
96.34
2.47
53.58
97.72 7.29 81.19 96.94 3.70 88.40
98.39
0.00 97.29 97.43 2.69 83.64
NAD
98.66
0.03 98.64 98.39
96.98
2.86
98.54 80.88 14.91 97.72 94.70 4.93
98.98
0.16 98.66
54.55 44.00
Our
98.60
0.00 98.66
90.36
1.07
64.17
90.01
0.02 95.58 97.41 0.51
89.87
99.05
0.00
98.93
95.08
0.32 89.44
(c) GTSRB
to generalize to more complex attacks, like the Blended attack, where the ASR increases to 95.63% 63%, and the RA drops to 4.16%. Neural Attention Distillation (NAD) exhibits similar behavior in terms of generalization, with its performance varying considerably across different attacks. For example, NAD achieves an ASR of 1.10% and an RA of 92.39% against BadNets, but its performance drops significantly when faced with the Blended attack, resulting in an ASR of 44.98%.
For pruning-based methods, for example, FP, Adversarial Neuron Pruning (ANP), and Channel Lip- schitz Pruning (CLP), these approaches demonstrate varying levels of effectiveness against different attacks. However, none of them consistently outperform our method FMP in all types of attacks. For example, FP achieves an ASR of 16.06% and an RA of 99.24% against BadNets, while ANP achieves an ASR of 73.36% and an RA of 26. 16% for the same attack when it only has limited fine-tune dataset (10%) and within limited func-tune epochs (i.c., 10 epochs). The primary factor contributing to the superior performance of FMP over other pruning methods lies in its alignment with the backdoor trigger's focus the DNN feature map rather than specific neurons within the DNN, such as ANP. This alignment occurs at the feature map level, distinguishing FMP from other techniques and facilitating its efficacy in achieving higher performance levels.
For the reverse engineering- based method (i.c., NC), we can observe that when the backdoor triggers are simple and easy to reproduce, NC will have high performance, e.g., NC achieves ASR of
7

---

Published as a conference paper at ICLR 2024
Table 2: FMP's elfectiveness under different poison rates. Our approach exhibits consistent performance across different poison ratios, highlighting its generalizability and reliability.
Poison
BadNets ASR
Blended ASR
Low Frequency
SSBA ASR
WaNet ASR
Rate (%)
Acc
RA
Acc
RA
Acc
ASR
RA
Acc
RA
Acc
RA
0.1
92.09
1.71 91.72 91.83
8.53
75.90 91.74 2.09 91.00 92.17
2.23
89.60 93.19
1.48
91.91
0.5
92.01
0.99 91.74 91.95
8.31
75.30 91.92 2.14 91.21 91.90
1.42
91.92
93.32
1.45
91.94
1
92.22
1.78 91.48 91.66
8.49 74.49 92.06 2.08 91.00 91.85
2.08
89.61
93.46
1.32
92.22
5
92.59
1.24
90.39
93.45
8.31
74.47 93.32
2.23 90.07 93.10
2.93
88.49
91.86
1.09
91.90
10
91.67
1.67
91.71 91.85
6.44 74.43 91.77 1.90 90.52 91.92 2.89
88.59 93.42
1.38
92.16
Table 3: FMP's performance with different retraining data ratios. Expanding the retraining dataset can consistently lower the attack success rate, yet our approach remains effective in neutral- izing the backdoor impact even with just 5% of the data.
Retraining ratio (%)
BadNets ASR
Blended ASR
Low Frequency
SSBA ASR
WaNet ASR
Acc
RA
Acc
RA
Acc ASR
RA
Acc
RA
Acc
RA
86.57
1.77
86.08
90.59
7.63
63.40
88.07
4.60 81.00
87.26
3,50
79.77
89.31
1.54
88.07
91.67
1.67
91.71
91.85
6.44
74.43
91.77
1,90 90.52
91.92
2.89
88.59
93.42
1.38
92.16
15
91.71
1.66
91.30
91.88
6.37
74.61
91.60
1.88 90.63
91.95
2.87
88.64
92.47
1.42
91.32
20
91.83
1.47
91.91
91.92
6.08
74.87
91.74
1.90
90.73
91.73
2.88
88.91
92.54
1.44
91.33
100
92.02
0.9
91.04
92.12
5.31
75.77
91.24
1.71
91.07
92.31
2.67
89.37
92.95
1.37
91.42
1.32% and an RA of 89.17% against BadNets. However, when the backdoor triggers are complex and invisible, the performance of NC significantly deteriorates, indicating its limited effectiveness in handling sophisticated backdoor attacks. For instance, when faced with the Blended attack, where the backdoor trigger is more intricate and harder to detect, NC's ASR increases to an alarming 99.90%, and its RA plummets to a mere 0.10%.
For the CIFARI00 and GTSRB datasets. line defense strategies. Specifically, FMP obtains the average 0.25% ASR and the average 60.2% RA in CIFAR which achieves average ASR reduction of 1.14%. Furthermore, FMP also obtains a mean ASR of 0.32% and an average RA of 89.44% in GTSRB, showcasing a significant enhancement of approximately 5.80% in RÁ compared to baseline methods Furthermore, it is im- portant to highlight the inconsistency in performance exhibited by baseline methods. For example, the NC technique displays an ASR of 1.14% and an RA of 34. 4.63% when challenged by BadNets on CIFAR 100. However, its effectiveness diminishes notably when subjected to the same attack but on different dataset, such as CIFAR1O. In contrast, the FMP approach consistently delivers superior results, consistently achieving lower ASR values across various datasets, including 2.86%, 0.25%, and 0.32% average ASR in CIFAR1O, CIFAR100, and GTSRB, respectively.
observe that first, FMP is also better than the base-
Effect of Poison Data Rate The poison rate, referring to the proportion of poisoned samples in the training dataset, plays a crucial role in the results of the backdoor trigger injection. We conducted experiments with different poison rates (from 0.1 %% to 10%) to explore their impact FMP's ef- fectiveness. The results, shown in Tab.2, indicate that FMP demonstrates consistent performance across different poison rates and effectively mitigates backdoor attacks. For example, considering the BadNets attack, the ASR changes slightly within 0.99% to 1.78% as the poisoning rate increases from 0.1% % to 10%. This trend is also observed for other attack strategies. Although higher poison rate can be expected to lead to a higher ASR, our experimental results show that this is not true. When the poisoning rate very low, it becomes more challenging for defense strategies detect the backdoor trigger from the model due to its subtle influence. As the poison rate increases, the backdoor trigger has a more noticeable impact on the model, which can detected and mitigated more easily by the defense strategy. Our experimental results emphasize the importance of design- ing defense strategies capable of detecting and mitigating backdoor attacks, even when dealing with subtle influences caused by low poison rates.
Effectiveness under Different Percentages of Clean Data We are also interested in studying the correlation between the performance of FMP and the amount of available training data, which will
8

---

Published as a conference paper at ICLR 2024
Table FMP's effectiveness using different in CIFARIO dataset under BadNets attack. The value denotes the size of each optimization step for the perturbed samples in the FRG (Algo- rithm 1). FMP remains effective regardless of the specifie parameter choice, yielding the best overall performance when is set to 1/255.
1/255
2/255
4/255
8/255
16/255
91.67
91.04
90.28
89.21
88.01
ASR
1.67
1.22
1.51
2.29
1.41
RA
91.71
90.79
89.49
88.03
87.82
Table 5: FMP's effectiveness using difffrent in CIFAR10 dataset under BadNets attack. Re- ducing the value of p results in more extensive clipping of the feature map. For p values greater than 4, FMP efficiently eliminates the backdoor and preserves the model's original accuracy.
p
16
32
64
Acc
65.57
85.42
87.65
88.95
89.96
91.67
ASR
3,98
1.40
1.3
1.28
1.56
1.67
RA
65.52
85,49
87.63
88.78
90.34
91.71
be used to repair the model to mitigate backdoor triggers. We compare four different retraining data ratios:5%, 10%, 15%, 20%, and 100%, and the results our FMP are demonstrated in Tab.3. We observe that the performance of our defense strategy improves the amount of clean training data increases. For example, when the retraining ratio increases from 5% to 100%, the ASR for BadNets decreases from 1.77% 0.9%, while the model accuracy (Acc) improves from 86.57% to 92.02% and the Robust Accuracy (RA) increases from 86.08% to 91. .04%. Similar trends can be observed for other attack strategies such Blended, Low Frequency, SSBA, and WaNet. This indicates that our defense strategy becomes more effective in mitigating backdoor attacks as more clean data are available to retrain the model. However, should be noted that even with a small amount of clean data (e.g., 5%), our defense strategy still exhibits competitively good performance in mitigating backdoor attacks. For example, with 5% retraining ratio, the ASR for WaNet 1.54%, while the Acc and RA are 89.31% and 88.07%, respectively.
Effectiveness under Different and p We further investigate the effectiveness FMP under dif- ferent and p. as listed in Tab.4 and Tab.5. We can first observe that with different the effectiveness of FMP is consistently satisfactory. Upon increasing the c, the Acc exhibits marginal decline, under- scoring FMP's resilience across varying values. Subsequently, when varying the parameter p for backdoor feature pruning, a notable decrease is observed in both accuracy (Acc) and robust accuracy (RA). This reduction can be attributed to the model's failure successfully finetune after 50% of the information is pruned along with 50% of the feature maps, hampering its performance. FMP can successfully execute mitigate the backdoor from the model with p larger than 4.
CONCLUSION
In this paper, we presented novel defense strategy, FMP, to effectively detect and mitigate backdoor attacks in DNNs. Our method targets the identification and removal of backdoor feature maps within the model. Through comprehensive experiments, we show the efficacy of our approach against various backdoor attack methods, surpassing current state-of-the-art defenses in terms of reducing attack success rates and maintaining stability across different datasets and attack methods.
ACKNOWLEDGEMENT
The work is supported in part by National Key R&D Program of China (2022ZD0160200), HK RIF (R7030-22). HK ITF (GHP/169/20SZ), the Huawei Flagship Research Grants 2021 and 2023, and HK RGC GRF (Ref: HKU 17208223), the HKU-SCF FinTTech AcademyR&D Funding Schemes

---

Published as a conference paper at ICLR 2024
in 2021 and 2022, and the Shanghai Artificial Intelligence Laboratory (Heming Cui is courtesy researcher in this lab).
REFERENCES
Inc. Amazon Web Services. Amazon web services (aws), 2023, URL https://aws amazon. com/. Accessed: 2023-05-02.
Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in enns by training set corruption without label poisoning. 2019 IEEE International Conference on Image Processing (ICIP), pp. 101-105, 2019.
Mariusz Bojarski, David W. del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang. Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. ArXiv, abs/ fbss/6044.0316,, 2016.
Qingwen Bu, Dong Huang, and Heming Cui. Towards building more robust models with frequency bias. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4402- 4411, 2023. 3
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Ben Edwards, Taesung Lee, Ian Molloy, and B. Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. ArXiv, abs/1811.00322, 2018. 3,
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar, Deepinspect: A black-box trojan de- tection and mitigation framework for deep neural networks. In International Joint Conference on Artificial Intelligence, 2019. URL https: : /api. semanticscholas .rgg//org////o wtt.aaarrrrr//oooIoDDDID. 1999666933..
Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. Advances in Neural Information Processing Systems, 35:9727-9737, 2022.
2, 5,
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Xiaodong Song. Targeted backdoor attacks on deep learning systems using data poisoning. ArXiv, abs/1712.00556, 2017. 2,
DataTurks, Dataturks: Data annotations made super easy, 2023, URL https:///ataautte
com/ Accessed: 2023-05-02.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv,
2019. 1,
Hugging Face. Hugging face: The ai community building the future, 2023. URL https://
huggingface. co/. Accessed: 2023-05-02.
Hao Fu, Akshaj Kumar Veldanda, Prashanth Krishnamurthy, Siddharth Garg, and Farshad Khorrami. Detecting backdoors in neural networks using novel feature-based anomaly detection. ArXiv, 0/5/2011.02522 2020. 3, 13
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 4rXiv:1412.6572, 2014.
Google. Google cloud, 2023. URL https::t///// http:///ooooooooo...l.e.e.o com/ Accessed: 2023-05-02.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. ArXiv, abs/1708.06633, 2017. 2,
Junfeng Guo, Ang Li, and Cong Liu. Aeva: Black-box backdoor detection using adversarial extreme value analysis. ArXiv, 10s/2110. 14880, 2021. URL https: : / /tppppaaaaaaaaaaaaaaa semant icschola......
orgg/CorpusTD: 24:0:pppuut::200044440 3,
10

---

Published as a conference paper at ICLR 2024
Wenbo Guo, Lun Wang. Xinyu Xing. Min Du, and Dawn Xiaodong Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in systems. ArXiv, abs/ Mat//9008.0..3 2019. URL https:://// /tttps://appee /api. wematticscoooleee.......
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Viston and Pattern Recognition (CVPR), pp. 770-778, 2015.
Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. ArXiv, abs/2202.03423, 2022. 6
Ltd. Huawei Technologies Co. Huawei cloud, 2023. URL htttps:/www.hhawwtt com/. Accessed: 2023-05-02.
Alex Krizhevsky and Geoffrey Hinton. Cifar-10 and cifar100 (canadian institute for advanced search), 2009. URL https:t///// / /www./w....ooo/cooo//./////////w cs. toronto...... http://kii//iiiei html. Accessed: 2023-05-02. 2,5
Yige Li, Nodens Koren, I. Lyu, Xixiang Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. ArXiv, abs/2101.05930, 2021a.
Yige Li, Xixiang Lyu, Nodens Koren, L. Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In Neural Information Processing Systems, 2021b. 6
Yige Li, Xixiang Lyu, Xingjun Ma, Nodens Koren, I.. Lyu, Bo Li, and Yugang Jiang. Reconstructive neuron pruning for backdoor defense. International Conference on Machine Learning, 2023. URL httppttt///eeee /tpp.aappi.//// s semant wcs.chlarccccccccoccoooocoo.aa.a
3, 13
Yiming Li. Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shutao Xia. Backdoor learning: A survey.
IEEE transactions on neural networks and learning systems, PP, 2020a. 2
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 16443-16452, 2020b. 2,
Jiacheng Liang, Ren Pang, Changjiang Li, and Ting Wang. Model extraction attacks revisited. arXiv preprint arXiv:2312. 3.05366, 2023.
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine- pre-prunnng: Defending against backdoor- ing attacks on deep neural networks. In International Symposium on Recent Advances Intrusion Detection, 2018. 2, 3, 6
Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Zhang. Abs: Scan- ning neural networks for back-doors by artificial brain stimulation. Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security 2019. URL https: //api. www.eteeanccneeeeeeeeeeeeemeeeee
A. Nguyen and A. Tran. Wanet imperceptible warping-based backdoor attack. 10/2200211369 2021. 2, 5,
ArXiv,
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Dogus Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. ArXiv, abs/1904.08779) 2019.
Pranav Rajpurkar, Jeremy A. Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Yi Ding, Aarti Bagul, C. Langlotz, Katie S. Shpanskaya, Matthew Lungren, and A. Ng. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. ArXiv, abs/1711.05225, 2017. 1
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pp. 91-99.199999 2015.
11

---

Published as a conference paper at ICLR 2024
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael S. Bermstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252, 2014.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In Proceedings of the IEEE In- ternational Joint Conference on Neural Networks (ICNN), 2012. URL http: /benchmark. ini rub. de/ sect on=gt rb& subsect wwwp:bbeectoodddttteeeeeeee on=dataset. Accessed: 2023-05-02. 2,5
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In Neural Information Processing Systems, 2018. 5,
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks.
ArXiv, abs/1912.02271, 2019. 3
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks neural networks. 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723, 2019a. 2, 3,6
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks neural networks. 2019
IEEE Symposium Security and Privacy (SP), pp. 707-723, 2019b.
Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, Chaoxiao Shen, and Hongyuan Zha. Backdoorbench: A comprehensive benchmark of backdoor learning. ArXiv, abs/2206. 12654, 2022. 2.
Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. ArXiv, abs/21 14430, 2021. 3, 6, 13
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, 2013. 2
Yi Zeng, Won Park, Zhuoqing Morley Mao, and R. Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. 2021 IEEE/CVF International Conference Computer Vision (ICCV),
pp. 16453-16461, 2021. 3, 6
Bingyin Zhao and Yingjie Lao. Clpa: Clean-label poisoning availability attacks using generative adversarial nets. In AAAI Conference on Artificial Intelligence, 2022. 2, 5, 6
Runkai Zheng, Rong Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. In European Conference on Computer Vision, 2022a. 2
Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Pre-activation distributions expose backdoor neurons. Advances in Neural Information Processing Systems, 35:18667- -16677 2022b. 2, 5,
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 COMPARISON WITH OTHER BASELINES
To illustrate the effectiveness of FMP with other adversarial example related strategies (c.g., ANP Wu & Wang (2021), AEVA Guo et al. (2021), and RNP Li et al. (2023)). The evaluation results are shown in Tab. 6. We can find that FMP obtain SOTA performance compared with these baselines. For example, in BadNets, FMP decreases the ASR from 50.96% to 1.67%, and FMP also increases the RA from 47.53% to 91.71% in BadNets. The primary reason for this is FMP's focus on the feature map level, which aligns with the backdoor trigger's emphasis on the DNN feature map, rather than on specific neurons within the DNN (e.g., as in ANP). Secondly, the pruning at the feature map level enables FMP to rapidly eliminate the backdoor trigger from the model. In contrast, neuron-level pruning typically requires significant overhead due to the need for repeated muue--inetune-evaluae
cycles. FMP is particularly advantageous in scenarios with limited com- putational resources, where developers may not have the capacity to extensively evaluate and remove backdoors from the model. This limitation results in strategies like ANP, AEVA, and RNP exhibiting higher ASR compared to FMP in our setup.
Backdoor Attack
BadNets ASR
Blended ASR
Low Frequency Acc
SSBA ASR
WaNet ASR RA
Ace
RA
Acc
RA
ASR
RA
Acc
RA
Acc
Benign
91.94
97.21
93.44 99.95
93.44 99.39
92.94
98.80
91.53
98.83
ANP
91.22 73.36 26.16 93.25 99.44 0.56 93.19 98.03 1.88
92.92 68.59
29.13
90.81
1.93 88.98
AEVA
91.05
50.96 47.53 92.28 59.37 38.66
93.05 59.81
36.38
92.29 67.56
26.01
90.26
6.54
90.59
RNP
90.55
55.01 36.46 92.29 55.59 42.15 92.41 58.71
40.1
91.94 61.24 30.6
90.22
18.15
72.95
FMP
91.67 1.67 91.71 91.85 6.44 74.43 91.77 1.90
90.52 91.92 2.89 88.59 93.42
1.38
88.98
Table 6: Performance comparison (%) of adversarial example related backdoor defense methods on CIFARIO under PreActResNer18,, under different attack strategies with poison rate of 10% and retraining data ratio of 100%. We set the e to 1/255, and the p is set to 64.
To illustrate the effectiveness of FMP with other SOTA backdoor defense strategies (e.g., DeepIn- spect Wu & Wang (2021), TABOR Guo et al. (2021), ABS Li et al. (2023) and Fu et al. (2020).. The evaluation results are shown in Tab. 7. We can find that FMP obtain SOTA performance compared with these baselines. For example, in BadNets, FMP decreases the ASR from 5.61 % to 1.67%, and FMP also increases the RA from 79.02% to 91.71% in BadNets. As discussed before, the primary reason for this is the pruning at the feature map level enables FMP to efficiently and rapidly climi- nate the backdoor trigger from the model. While other defense strategies will not obtaín the SOTA performance due to the limited computational resources.
Backdoor Attack
BadNets ASR
Blended ASR
Low Frequency
SSBA ASR
WaNet
Acc
RA
Acc
RA
Acc ASR
RA
Acc
RA
Acc
ASR
RA
Benign
91.94
97.21
93.44
99.95
93.44
99.39
92.94
98.80
91.53
98.83
DeepInspect
90.51
15.87
64.44
90.89
3.5 77.16
90.83 4.7 77.51
90.05 10.57 73.83
90.31
5.97
77.94
TABOR
90.78
9.19 79.02
90.78
11.13 78.22
90.14
5.48
76.44
90.03
13.09 76.71
90.95
7.36
78.29
ABS
90.78
5.61 77.34
90.95 14.44
80.33
90.93
10.43
88.32
90.95
17.12 77.45
90.6
13.04
77.6
Fu al. (2020)
90.7
11.22
76.72 90.28
9.37
75.63
90.46
9.62
71.87
90.06
6.74
77.43
90.66
14.98
69.13
FMP
91.67 1.67 91.71 91.85
6.44
74.43
91.77
1.90
90.52
91.92 2.89
88.59 93.42
1.38
88.98
Table 7: Performance comparison (%) of other SOTA backdoor defense methods on CIFARIO under PreActResNe118, 8, under different attack strategies with a poison rate of 10% and retraining data ratio of 100%. We set the c to 1/255, and the is set to 64.
B ABLATION STUDY ILLUSTRATION
In this section, we illustrate the figure illustration for the tables illustrated in Tab.3 to Tab.5.
13

---

Published as a conference paper at ICLR 2024
Effect of Poison Data Rate The poison rate, referring to the proportion of poisoned samples in the training dataset, plays a crucial role in the results of the backdoor trigger injection. We conducted experiments with different poison rates (from 0.1 % to 10%) to explore their impact FMP's ef- fectiveness. The results, shown in Fig.1, indicate that FMP demonstrates consistent performance across different poison rates and effectively mitigates backdoor attacks. For example, considering the BadNets attack, the ASR changes slightly within 0.99% to 1.78% as the poisoning rate increases from 0.1% % 10%. This trend is also observed for other attack strategies. Although higher poison rate can be expected to lead to a higher ASR, our experimental results show that this is not true. When the poisoning rate very low, it becomes more challenging for defense strategies detect the backdoor trigger from the model due to its subtle influence. As the poison rate increases, the backdoor trigger has a more noticeable impact on the model, which can detected and mitigated more easily by the defense strategy. Our experimental results emphasize the importance of design- ing defense strategies capable of detecting and mitigating backdoor attacks, even when dealing with subtle influences caused by low poison rates.
Effectiveness under Different Percentages of Clean Data We are also interested in studying the correlation between the performance of FMP and the amount of available training data, which will be used to repair the model to mitigate backdoor triggers. We compare four different retraining data ratios:5%, 10%. 15%, 20%, and 100%, and the results of our FMP are demonstrated in Fig.2. We observe that the performance of our defense strategy improves as the amount of clean training data increases. For example, when the retraining ratio increases from 5% to 100%, the ASR for BadNets decreases from 1.77% to 0.9%, while the model accuracy (Acc) improves from 86.57% to 92.02% and the Robust Accuracy (RA) increases from 86.08% to 91.04%. Similar trends can be observed for other attack strategies such Blended, Low Frequency, SSBA, and WaNet. This indicates that our defense strategy becomes more effective in mitigating backdoor attacks as more clean data are available to retrain the model. However, it should be noted that even with a small amount of clean data (e.g.. 5%). our defense strategy still exhibits competitively good performance in mitigating backdoor attacks. For example, with 5% retraining ratio, the ASR for WaNet is 1.54%, while the Acc and RA are 89.31% and 88.07%, respectively.
Effectiveness under Different and p We further investigate the effectiveness of FMP under different and p. as listed in Fig.3 and Fig.4. We can first observe that with different the effective- ness of FMP is consistently satisfactory. Upon increasing the c, the Ace exhibits marginal decline, underscoring FMP's resilience across varying values. Subsequently, when varying the parameter p for backdoor feature pruning, a notable decrease is observed in both accuracy (Ace) and robust accuracy (RA). This reduction can be attributed to the model's failure successfully finetune after 50% of the information is pruned along with 50% of the feature maps, hampering its performance. FMP can successfully execute mitigate the backdoor from the model with a p larger than 4.
14

---

Published as a conference paper at ICLR 2024
Separate
Bucflets ASA
Blanded ASR
32.5
Prequeeryy ASR
Blanded Acc
and IA
85.0
Blurded MA
Low Free Jency ACc
82.5
WaNet Acc
Waffet IA
75.0
Palsor 1961
Figure 1: FMP's effectiveness under different poison rates.
Metrics Retraining Ratio for All Attacks with Separate Yaxes and Detailed Legend
Slanded
Honded Ao
We Poo yo Aat Prequanoy
90
Boke''s ACe
Handed
ASR
Alc. and
75
55
20
(%)
Figure 2: FMP's effectiveness under different retraining data ratios.
15

---

Published as a conference paper at ICLR 2024
Improved Metrics vs
10
Accuracy (Acc)
Robust Accuracy (RA)
91.5
Attack Success Rate (ASR)
91.0
90.5
Acc and RA
ASR
89.5
88.5
88.0
0.03
0.05
0.06
Figure 3: FMP's effectiveness under different in CIFAR10 dataset under BadNets attack.
Improved Metrics vs
10
Accuracy (Acc)
*
Robust Accuracy (RA)
Attack Success Rate (ASR)
Acc and RA
80
ASR
75
70
60
Figure 4: FMP's FMP's effectiveness under different p in CIFARIO dataset under BadNets attack.
16


Reviews:
[{"review_id": "65aVTdWkHA", "reviewer": "Reviewer_W4kG", "summary": "This paper attempts to mitigate the backdoor model by generating all possible adversarial feature maps. Each generated adversarial feature map is fed to the model to test whether it will misclassify data samples, aiming to identify malicious feature maps that may be caused by a trigger. The innovation of the proposed algorithm lies in the fact that it does not require prior knowledge of the trigger pattern through reverse engineering, and it does not impose constraints on the trigger pattern size, as seen in other defense algorithms like Neural Cleanse. The proposed algorithm was evaluated on three datasets: CIFAR-10, CIFAR-100, and GTSRB.", "questions": "1.\tThere are several confusing descriptions. For instance, 'f' and 'F' represent the model and feature map, as described in the Notations section. However, in Section 3.2 and Algorithm 1, 'f' has a mixed meaning.\n2.\tShould the second 'for' loop in Algorithm 1 return '\\hat{x'}? Is that correct?\n3.\tThe logic of 'inference()' in Algorithm 1 appears to be incorrect. If a feature map does not change the classification of 'x,' it is a normal feature map and should be retained. However, in Algorithm 1, it is pruned. Why is this the case?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1.\tThe proposed algorithm is effective for large trigger backdoored models.\n2.\tThe proposed algorithm mitigates the backdoored model without the need for reverse engineering the trigger.\n3.\tIt has been evaluated on three datasets.", "weaknesses": "1.\tThe presentation needs improvement as there are many confusing descriptions, referring to the Question section.\n2.\tThe three datasets appear to contain a relatively small number of classes. It would be more convincing if the algorithm could be evaluated on more complex datasets, such as ImageNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cHj62m7OVk", "reviewer": "Reviewer_x3Nd", "summary": "In this manuscript, the authors propose a DNN pruning method called FMP to mitigate backdoors. While adding adversarial attacks to feature maps, their method prunes the features weak to the attacks.", "questions": "When the authors have parametric variables in a table, it should be better to draw these results with figures.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I find this paper interesting. It's important to understand the relationship between pruning and backdoors, and the authors explored this in a systematic way.", "weaknesses": "While reading the authors' method, it looks similar to Adversarial Neuron Pruning (ANP) by We and Wang (2021). However, the authors don't describe it in the related work, though they compare it with the proposed method in the result section. Discussing the methodological differences between them would help readers understand more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ew7KuBtYQN", "reviewer": "Reviewer_gATw", "summary": "This paper proposes Adversarial Feature Map Pruning for Backdoor (FMP), a new method for backdoor mitigation in neural networks. FMR does not require access to the trigger or poisoned data. Based on a clean data sample, the method reverse-engineers poison triggers from each feature map in the model (at multiple layers) based on back-propagation. The weights determined to be connected to the triggers are reinitialized and fine-tuned on clean data. Experiments are performed on CIFAR-10, CIFAR-100 and GTSRB against a wide range of attacks and defenses.", "questions": "1. What is the distinction the paper makes between a neural network layer output and a feature map?\n2. What does it mean in Alg. 1 that `Correct_list` is sorted in ascending order? According to which criterion?\n3. Are all the weights of the model refined during the fine tuning step or just those detected as being part of backdoors (i.e., the weights that are reset to zero)?\n4. Why is the backdoor feature initialization done by setting weights to zero, instead of using the same initialization strategy as when training the model for the first time (i.e., various random sampling strategies)? Has this alternative been considered?\n5. What is the impact of varying method parameters $\\epsilon$ and $p$ under other attacks than BadNets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper provides an extensive evaluation using the standard BackdoorBench benchmark, against multiple attacks and compared to multiple defenses.\n- The proposed FMT seems to perform well on average.\n- The source code was provided and is pledged to be available open-source upon paper acceptance.\n\n[Update based on authors' response] I would like to thank the authors for their answer and additional results. I think updating the paper based on the discussion would improve it. I have raised my score.", "weaknesses": "# Novelty and prior work\n\n- The novelty of the paper seems limited. The ideas of using adversarial examples to reverse engineer triggers (e.g., [ANP](https://arxiv.org/pdf/2110.14430.pdf), [AEVA](https://openreview.net/forum?id=OM_lYiHXiCL)) or pruning and retraining trigger weights ([RNP](https://proceedings.mlr.press/v202/li23v.html)) are not themselves novel. The paper does not cite most of these very close prior results and does not provide a conceptual comparison to them.\n- The prior art section mainly addresses defenses from different categories than the present one, which are then easy to dismiss. Outside the specific ideas used in FMT, there are many methods that address the same setup as the current paper and operate without knowledge of the trigger (e.g., [DeepInspect](https://www.ijcai.org/proceedings/2019/647), [TABOR](https://arxiv.org/pdf/1908.01763.pdf), [ABS](https://dl.acm.org/doi/10.1145/3319535.3363216), [[Fu et al., 2020](https://arxiv.org/pdf/2011.02526.pdf)]). These could be included in the experimental comparison.\n\n# Performance\n\n- Tab. 1 shows that the proposed method is only the best on average, but not doing so well on individual benchmarks (i.e., for each attack and dataset). The commentary section also fail to quantify how many benchmarks are actually won by the proposed method.\n\n# Clarity\n\n- Certain points in the paper are not clear, see also the questions below. Moreover, the components of the method are rather explained in an algorithm than in the text.\n- Assumptions should be clear earlier in the paper, e.g. the fact that a clean input sample is required.\n- The typography of the paper could be improved, please see some suggestions below. The paper could also use additional proofreading.\n- Tab. 1 is very dense and lacking highlights of the best values for each attack. As such, it is currently difficult to interpret.\n\n# Minor remarks\n\n- It is unclear why the feature maps are summed in the Notations paragraph (Sec. 3).\n- Some numerical results are typeset with spaces before the decimals (e.g., 2. 86%). These spaces should be removed\n- Consider using `\\citep` when the cited references are not part of the sentence (see [here](https://www.overleaf.com/learn/latex/Natbib_citation_styles)).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)