Data Free Backdoor Attacks
Authors: Bochuan Cao
Jinyuan Jia
Chuxuan Hu
Wenbo Guo
Zhen Xiang
Jinghui Chen
Bo Li
Dawn Song
Venue: NeurIPS 2024 poster (2024)
Keywords: Data free backdoor attacks
Primary area: safety_in_machine_learning
TL;DR: We propose a data-free backdoor attack without changing the architecture of a classifier.

Abstract: Backdoor attacks aim to inject a backdoor into a classifier such that it predicts any input with an attacker-chosen backdoor trigger as an attacker-chosen target class. 
Existing backdoor attacks require either retraining the classifier with some clean data or modifying the model's architecture.
As a result, they are 1) not applicable when clean data is unavailable, 2) less efficient when the model is large, and 3) less stealthy due to architecture changes. 
In this work, we propose DFBA, a novel retraining-free and data-free backdoor attack without changing the model architecture. 
Technically, our proposed method modifies a few parameters of a classifier to inject a backdoor. 
Through theoretical analysis, we verify that our injected backdoor is provably undetectable and unremovable by various state-of-the-art defenses under mild assumptions. 
Our evaluation on multiple datasets further demonstrates that our injected backdoor: 1) incurs negligible classification loss, 2) achieves 100\% attack success rates, and 3) bypasses six existing state-of-the-art defenses. 
Moreover, our comparison with a state-of-the-art non-data-free backdoor attack shows our attack is more stealthy and effective against various defenses while achieving less classification accuracy loss.
We will release our code upon paper acceptance.

Data Free Backdoor Attacks
Bochuan Cao', Jinyuan Jia', Chuxuan Hu², Wenbo Guo", Zhen Xiang", Jinghui Chen', Bo Li", Dawn Song"
'The Pennsylvania State University Universityy of Illinois at Jrbana-Champaige University of California, Santa Barbara *University of Georgia University of California Berkeley
bccaoâpsu. edu, jinyuandpau edu, chuxuan3@illinois edu, henrygwblucab. edu, zhen. zhen.xiang, Laaanna.ane@gmaaiii com, jzc5917@psu. edu, 1bo0i mooiilinnois.. edu, dawnsongêes. berkeley. edu
Abstract
Backdoor attacks aim to inject backdoor into classifier such that predicts any input with an attacker-chosen backdoor trigger as an attacker-chosen target class. Existing backdoor attacks require either retraining the classifier with some clean data or modifying the model's 's architecture. As a result, they are 1) not applicable when clean data is unavailable. 2) less efficient when the model is large, and 3) less stealthy due to architecture changes. In this work, we propose DFBA, a novel retraining-free ffre and data-free backdoor attack without changing the model architecture. Technically, our proposed method modifies few parameters of a classifier to inject backdoor. Through theoretical analysis, we verify that our injected backdoor is provably undetectable and unremovable by various state-of-the art defenses under mild assumptions. Our evaluation on multiple datasets further demonstrates that our injected backdoor: 1) incurs negligible classification loss, 2) achieves 100% attack success rates, and 3) bypasses six existing cate-of-the-an comparison with fatte-of-the-eee attack shows our attack is more stealthy and effective against various defenses while achieving less classification accuracy loss. The code for experiment be found at huttps://gitae com/ http//AAAssuaaaaaaaaaaA wawassuuaaaaaaaaeeeeeeeeeeee Bawpeeee__aaccooooreeeeeee
defenses. Moreover,
non-data-free backdoor
Introduction
Deep neural networks (DNN) have achieved remarkable success in multiple application domains such computer vision. To democratize DNN models, especially the powerful but large ones, many machine learning platforms (e.g.. ModelZoo (1, TensorFlow Model Garden 22. and Hugging Face (3)) share their pre- pre-trained classifiers customers with limited resources. For instance, Hugging Face allows any third party to share pre-trained classifiers with the community, which could be downloaded by other existing studies [4] 5 (6)
Despite the benefits brought by those machine learning platforms, show that this model sharing mechanism is vulnerable to backdoor attacks. In particular, malicious third party could download pre-trained classifier from the machine learning platform, inject backdoor into it, and reshare it with the community via the platform. Backdoor attacks pose severe concerns for the deployment classifiers downloaded from the machine learning platforms for security and safety-critical applications such as autonomous deriving [7]. We note that the model provider may not share the training data used to train the classifiers when they are trained on private data (e.g., face images).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

As result, they may underestimate the To thoroughly understand this threat, recent research has proposed a large number of backdoor attacks 6
At high level, existing backdoor attacks require either retraining a classifier by accessing some clean
g
10
19
20
21
22
24
25
26].
data 5 TO
27 or changing the architecture of classifier [2 28 For instance, Hong et
al. [27] proposed handcrafted backdoor attack, which changes the parameters of a classifier to inject a backdoor. However, they need a set of clean samples that have the same distribution as the training data of the classifier to guide the change of the parameters. Bober-Irizar et al. [28] proposed to inject a backdoor into classifier by manipulating its architecture. Consequently, their practicality is limited if there is no clean data available, efficiency is restricted if the model is large, or they less stealthy due to architecture changes. Additionally, none of the existing attacks provide a formal analysis of their attack efficacy against cutting- edge defenses 29 30 threat caused by backdoor attacks.
Our contribution: We propose DFBA, novel retraining free and data-free backdoor attack, which injects a backdoor into pre-trained classifier without changing its architecture. At a high level, DFBA first constructs backdoor path by selecting a single neuron from each layer except the output layer. Then, it modifies the parameters of these neurons such that the backdoor path is activated for a backdoored input but unlikely be activated for clean input. As result, the backdoored classifier predicts backdoored inputs to a target class without affecting the predictions for clean inputs. Specifically, we first optimize a backdoor trigger such that the output of the selected neuron in the first layer is maximized for backdoored input. Second, we change the parameters of this neuron such that can only be activated by any input embedded with our optimized trigger but is unlikely to be activated by a clean input. Third, we change the parameters of the middle layer neurons in the backdoor path to gradually amplify the output of the neuron selected in the first layer. Finally, for the output layer's neurons, we change their parameters such that the output of the neurons in the backdoor path has a positive (negative) contribution to the output neuron(s) for the target class (all non-target classes).
We conduct both theoretical and empirical evaluations for DFBA. Theoretically, we prove that backdoors injected by DFBA are undetectable by state-of-the-art detection methods, such as Neural Cleanse BT and MNTD 30 or irremovable by fine-tuning techniques. Empirically, we evaluate DFBA on various models with different architectures trained from various benchmark datasets. We demonstrate that DFBA can achieve 100% attack success rates across all datasets and models while triggering only less than 3% accuracy loss on clean testing inputs. We also show that DFBA can bypass six state-of-the-an defenses. Moreover, we find that DFBA more resilient to those defenses than state-of-the-art non-data-free backdoor attack (27). Finally, we conduct comprehensive ablation studies to demonstrate DFBA is insensitive to the subtle variations hyperparameters. To the best of our knowledge, DFBA is the first backdoor attack that is retraining-free, data-free, and provides theoretical guarantce its attack efficacy against existing defenses.
Our major contributions are summarized as follows:
We propose DFBA, the first data-free backdoor attack without changing the architecture of classifier. Our DFBA directly changes the parameters of classifier to inject backdoor into it.
We perform theoretical analysis for DFBA. We show DFBA is provably undetectable or unremovable by multiple state-of-the-an art defenses.
We perform comprehensive evaluations on benchmark datasets to demonstrate the effective- ness and efficiency of DFBA.
We empirically evaluate DFBA under existing state-of-the-as art defenses and find that they are ineffective for DFBA. Our empirical results also show that DFBA is insensitive to hyperparameter choices.
present the technical details of our DFBA Section 4 show the evaluation results in Section discuss and conclude our DFBA in Section
Roadmap: We show related work in Section 2 formulate the problem in Section 3

---

2 Related Work
Backdoor Attacks. Existing backdoor attacks either use the whole training set to train a backdoored 5 8 or modify the weights or architecture of a pre-trained clean classifier to inject backdoor 6 27 For instance, BadNet 4 constructs a poisoned training set with clean and backdoored data to train backdoored classifier from scratch. We note that poisoning data based backdoor attacks require an attacker to compromise the training dataset of a model, i.e., inject poisoned data into the training data of the model. Our attack does not have such constraint. For instance, many machine learning platforms such as Hugging Face allow users to share their models. A malicious attacker could download a pre-trained classifier from Hugging Face, inject backdoor using our attack, and republish on Hugging Face to share it with other users. Our attack is directly applicable in this scenario. Moreover, data poisoning based attacks are less stealthy as shown in the previous work [27]. More recent works [6] 12 19. 21 27 28 32] considered a setup where the attacker has access to pre-trained clean model rather than the original training dataset. Under this setup, the attacker either manipulates the model's weights with a small set of clean validation data (i.e., parameter modification attacks) directly vary the model architecture. As discussed in Section [ those attacks require either retraining with some clean data or modifying the model architecture. In contrast, we propose the first backdoor attack that is entirely retraining-fres and data-free without varying the model architecture.
classifier from scratch [4]
Note that parameter modification attacks share similar attack mechanism as ours. Among these attacks, some [33 (34 355 serve a different goal (e.g.. fool the model to misclassify certain clean testing inputs) from Others (21 27 36) still require clean samples to provide guidance for parameter modification. DFBA has the following differences from these methods. First, DFBA does not require data when injecting the backdoor, while these methods still require a few samples. Second, DFBA is provably undetectable and irremovable against various existing defenses (Section B}. while existing weight modification attacks do not provide formal guarantee for its attack efficacy. Finally, will show later in Section 5 and Appendix G compared to the tfate-ofthhe-- art weight modification attack (27. DFBA incurs less classification accuracy loss on clean testing inputs than 27 In addition, DFBA requires modifying fewer parameters and is most efficient.
We note that a prior study [26) proposed "data-free" backdoor attack to deep neural networks. Our method has the following differences with [26). First, they require the attacker to have a substitution dataset while our method does not have such requirement (i.c., our method does not require substitution dataset). Second, they inject the backdoor into a classifier by fine-tuning it. By contrast, method directly changes the parameters of classifier to inject the backdoor. Third, they did not provide a formal analysis on the effectiveness of their attack under state-of-the-we defenses.
Recent research has begun to explore data-free backdoor attacks in distributed learning scenarios, particularly in Federated Learning (FL) settings. FakeBA (37] introduced novel attack where fake clients can inject backdoors into FL systems without real data. The authors propose simulating normal client updates while simultaneously optimizing the backdoor trigger and model parameters in data-free manner. DarkFeD (38) proposed the first comprehensive data-free backdoor attack scheme. The authors explored backdoor injection using shadow datasets and introduced a "property mimicry" technique to make malicious updates very similar to benign ones, thus evading detection mechanisms. DarkFed demonstrates that effective backdoor attacks can be launched even when attackers cannot access task- data.
Backdoor Detection and Elimination. Existing defenses against backdoor attacks can be classified into- 1) Training-phase defenses that train a robust classifier from backdoored training data (3 40 41 42]: 2) Deployment-phe phase defenses that detect and eliminate backdoors from pre-trained classifier with only clean data [31] 43 44 45 46 [7]: 3) Testing-phar defenses [48] 49) that identify the backdoored testing inputs and recover their true prediction result. Training -hass defenses are not applicable to given classifier that is already backdoored. Testing-phase defenses require accessing to the backdoored inputs (See Section for more discussion). In this work, we mainly consider the deployment- -hase defenses. Existing deployment-phan detection removal methods that first reverse-engineer a trigger from backdoored classifier and
defenses mainly take three directions: D
then use it to re-train the classifier to unlearn the backdoor [3 44 50 @ unlearning methods that
fine-tune a classifier with newly collected data to remove the potential backdoors [51 52 53 54
and @ fine-pruning methods that prune possibly poisoned neurons of the model [5] 55 As we will
3

---

our attack is empirically resistant to all of these three methods. In addition, under mild assumptions, our attack, with theoretical guarantee, can evade multiple state-of-the- detection & removal methods and fine-tuning methods (See Section
show in Section 5
Problem Formulation
3.1 Problem Setup
Consider a pre-trained deep neural network classifier with L layers, where W and both denote its weight and bias the 1-th layer. Without loss of generality, we consider ReLU as the activation function for intermediate layers, denoted as and Softmax as the activation function for the output
layer. Given any input that can be flattened into one-dimensional vector x [x1, F2, xra E Rd where the value range of each element Fn is [a% oa) the classifier g maps For instance, when the pixel value of an image is normalized to the range [0, 1]. then we have a = 0 and x = 1. An attacker injects a backdoor into a classifier g such that predicts any input embedded with attacker-chosen trigger (ő, m) an attacker-chosse target class Vtc. where and m respectively represent the pattern and binary mask of the trigger. A backdoored input is represented follows:
one of the C classes.
x x @ (6, m) x ④ m) 5 m,
(1)
where @ represents element- wise multiplication. For simplicity, we denote a classifier injected with the backdoor as f. Moreover, given a backdoor trigger (ō, m), we have ((m) = [n|m, Tolmm = 1, = 1, 2...... ,d], which denotes the of feature indices where the corresponding value of is 1.
3.2 Threat Model
We consider that an attacker aims to implant a backdoor into a target pre-trained model without retraining it or changing its architecture. Meanwhile, we also need to maintain the backdoored model's normal utilities (i.e., performance on clean inputs). Moreover, we require the implanted backdoor to be stealthy such that it cannot be easily detected or removed by existing backdoor detection or elimination techniques.
Attacker's goals:
A Attacker's background knowledge and capability: Similar to existing attacks 66 271, we consider the scenarios where the attacker hijacks the ML model supply chain and gains white-box access to a pre-trained model. Differently, we do not assume that the attacker has any knowledge or access data. Moreover, we assume that the attacker cannot change the
to the training/tssiig/vuldddmm architecture of the pre-trained classifier. In addition, we assume that the attacker does not have access to the training process (e.g., training algorithm and hyperparameters) As discussed above, these assumptions significantly improve the practicability of our proposed attack.
3.3 Design Goals
When designing our attack, we aim to achieve the following goals: utility goal, effectiveness goal, efficiency goal, and stealthy goal.
Utility goal: For the utility goal, we aim to maintain the classification accuracy of the backdoored classifier for clean testing inputs. In other words, the predictions of the backdoored classifier for clean testing inputs should not be affected.
Effectiveness goal: We aim to make the backdoored classifier predict the attacker-chosen target label for any testing input embedded with the attacker-chosen backdoor trigger.
Efficiency goal: We aim to make the attack that is efficient in crafting a backdoored classifier from a pre-trained clean classifier. We note that attack that achieves the efficiency goal means it is more practical in the real world.
Stealthy goal: The stealthy goal means our attack could bypass existing state-of- thetterrtttttt defenses. An attack that achieves the stealthy goal means less likely to be defended. In this work, we will conduct both theoretical and empirical analysis for our attack under state-of of-the-ar defenses.

---

Select a Filter
Trigger Optimization
0.9
b =
0.9
0.6
0.3
6.1
0.6
0.5
0.1
Conv Filters Layer
Trigger
Figure 1: An example of the backdoor switch and optimized trigger when each pixel of an image is normalized to the range [0, 1].
DFBA
Since assume neither model retraining architecture modification, the only way of implanting the backdoor is to change the model parameters Specifically, given a classifier g. we aim to manually modify its parameters craft a backdoored classifier f. Our key idea is create path (called backdoor path) from the input layer to the output layer to inject the backdoor. In particular, our backdoor path satisfies two conditions: 1) it could be activated by any backdoored input such that our backdoor attack is effective, i.c., the backdoored classifier predicts the target class for any backdoored input, and 2) it cannot be activated by clean input with high probability to stay stealthy. Our backdoor path involves only single neuron (e.g. a single filter in CNN) in each layer reduce the impact of the backdoor on the classification accuracy the classifier. The key challenge is how to craft a back- door path such that simultaneously sat- isfies the two conditions. To address this challenge, we design a backdoor switch which is single neuron selected from the first layer of classifier. We modify the parameters of this neuron such that it will be activated by a backdoored input but is unlikely to be activated by clean input. Then, we amplify the output of the backdoor switch by changing the pa- rameters the remaining neurons in the backdoor path. Finally, change the weights of the output neurons such that the output of the (I. 1 Jth-layer neuron in the backdoor path has a positive (or negative) contribution to the output neu- rsn(s) for the target class (or non-target classes) reach our goal.
First Cosy Layer.
velux
+D)
201 Layone
softmax
0.59
Flantned Enbedding
Figure 2: Visualization of our backdoor path when it activated by a backdoored input. The backdoored model will predict the target class for the backdoored input.
4.1 Detailed Methodology
4.1.1 Neuron Selection for Backdoor Path
Our goal is to select neurons from classifier such that they form path from the first layer the final output layer. In particular, we select a single neuron from each layer. For the first layer, we randomly select neuron,? As will see in the next step, neuron selection in this way enables us to change the parameters of the selected neuron such that it has different behaviors for a clean input and backdoored input. For each middle layer, we select neuron such that its output depends on the selected neuron in the previous layer. Note that we randomly select one if there exist multiple neurons that satisfy the criteria.
AAthhouhh 27 also does inject the backdoor by changing model parameters, it still requires a few samples to facilitate the parameter changes. While method does not need any data inject the backdoor. In addition. we provide formal guarantee for our attack efficacy. In fact, we require the neuron's output to depend on the features with the index in 1'(m). For fully connected neural network, is obvious. For convolutional neural network, the detail is illustrated in Appendix

---

4.1.2 Backdoor Switch
We design a backdoor switch, which is a single neuron (denoted as $1) in the first layer, such that the neuron $1 satisfies two conditions:
Condition The switch neuron 81 is activated for backdoored input x'.
Condition 2:
The switch neuron is unlikely to be activated for a clean input x.
To achieve the two conditions mentioned above, we need to tackle the following challenges. 1, given that n', can be different for different backdoored inputs for n ø [(m). To enable 81 to activated by any backdoored input, first need to ensure that the activation of is independent of the value of z',, where n $ r(m)). 0, after we decouple the activation of 81 from x n & I'(m), we need to make sure its activation value is only related to the trigger pattern. This is challenging in that the value of En where n E r(m), can be different for different clean inputs.
Our key idea is to modify the parameters of the neuron 81 such that its outputs only depend on the features of an input whose indices are in I((m), i.e., Iy (or xf,) where E ('m). Specifically, we propose to reach this by setting the corresponding weight between 81 and feature whose index is not in ('(m) to O. Given an input x, we use 81 (x) to denote the output of the neuron $1. Here, $1 (x) = o (EE Wr + +). Given that We, = 0, for n & [(m), we can rewrite $1 (x) = o(Ener(mm) 2n b), which is independent from 2n for n o ((m)
Addressing challenge O:
A Addressigg Challenge @: Our idea is first optimize the backdoor pattern òn (n E [(m)) and then modify the remaining parameters of 81 such that 1) 81 is activated for a backdoored input, and 2) 81 is unlikely to be activated when 2i is not close to the optimized 6n for E [(m).
Backdoor trigger generation. For a backdoored input, we have (1 = o(Ener()) WnOn + 6)
since z/, 5, for n E T(m). For an arbitrary set of wn (n mm), we optimize the trigger pattern such that the output 81 is maximized for a backdoored input, i.e., we aim to solve the following optimization problem:
max (1 8
Σ (Em)
wnon + b), s.t. o S.a S o Vn E T(m),
(2)
where the constraint ensures backdoored input created by embedding the backdoor trigger (6, m) arbitrary input is still valid to the classifier, and [a',a"] is the range of feature value ii, (see Section 3.. details). Note that although the binary mask m is chosen by the attacker, we can still derive the analytical solution the above optimization problem:
8p = - foh: if wn 0
(3)
a otherwise.
Given the optimized backdoor trigger, we design the following method to modify the bias and weights of $1 to achieve the two conditions.
Activating 8 for x' by modifying the bias. Recall that our condition 1 aims to make the switch neuron 81 be activated for backdoored input x'. In particular, given backdoored input x' embedded with the trigger 8, the output of the neuron for x' is as follows: $ (x') = o o(EnTT()) wx, 6, + 6). To make $1 be activated for backdoored input x', need to ensure Ener(m) W,,n, + b to be positive. For simplicity,
denote À Σner(m) 6n,dn, + b. For any X, the bias b of the switch neuron 81 satisfies the above condition, the output sı for an arbitrary backdoored input. In other words, the switch neuron is activated for backdoored input, meaning we achieve condition 1. Figure [ shows an example of our backdoor switch.
Deactivating 81 for x by modifying the weights. With the above choice of b, we calculate the output of the neuron 8 for clean input x. Formally, we have:
81 (x)
Σ
WnEn + +
Σ
wnom = (1
Σ
wn - 6n) + x).
(EL(m
(ET(m)
REN(m)
Our condition aims to make the switch neuron 81 less likely to activated for a clean input x. Based Equation 4 we know clean input camnot activate the neuron $ when Ene((m) Wn (zn,

---

0, Ener(m) - 2 X. In other words, when a clean input cannot activate
the neuron $1 when it satisfies the following condition: Ener(m) Wn (6,, F.) A. By showing
this condition is equivalent to Ener(m) [w,,
6n) 2 X, we have the following lemma
Lemma Suppose on (n T'm)) is optimized in Equation Given an arbitrary clean input X cannot activate s if the following condition is satisfied:
Σ
[2n(Inn 6n)1 A.
(5)
mm)(m)
Interpretation of Ener(m) (2n - 6n)1 < A: We note that Ener(m) [2n - -6,]1 <
measures the difference of clean input and the backdoor trigger for indices in [(m) (indices where the backdoor trigger is embedded to an input). In particular, for each index in [(m). [w (2n, - 6,) measures the weighted deviation of the feature value of the clean input x at the dimension from the corresponding value of the backdoor trigger. Based on the above lemma, a clean input can only activate 81 when (m) (N,n 6,)l A. When À is very small and |w,| is large, clean input can only activate the neuron 8 when 2n very close 8, for n E I m. which means that the clean input is very close to its backdoored version. In practice, we find that setting a small À (e.g..
0.1) enough to ensure a clean input cannot activate $1-
4.1.3 Amplifying the Output of the Backdoor Switch
The neuron in the first layer is activated for a backdoored input x'. In the following layers, we can amplify until the output layer such that the backdoored classifier outputs the target class lic- Suppose 81 is the selected neuron in the /th layer, where 1 = 2,3, ... J- 1. We can first modify the parameters of 81 such that its output only depends on 81-1 and then change the weight between st
and 8;-1 to be 7, where is a hyperparameter, We call amplification factor. By letting the bias term st to be O, we have:
is (x') -x))
(6)
Note that (x) when (x) O. Finally, we set the weight between SL-1 and the output neuron for the target class Vtc to be but set the weight between 8 and the remaining output neurons to be -y Figure shows example when the backdoor path is activated by backdoored input.
4.2 Theoretical Analysis
First, we provide the following definitions:
Pruned classifier: In our backdoor attack, we select one neuron for each layer in a classifier. Given a pre-trained classifier, we can create corresponding pruned classifier by pruning all the neurons that are selected to form the backdoor path by DFBA. Note that the pruned classifier is clean as it does not have any backdoor.
Based on this definition, we provide the theoretical analysis towards our proposed method this section. We aim show that our proposed method can maintain utility on clean data, while cannot be detected by various backdoor model detection methods or disrupted by fine-tuning strategies. Due space limits, mainly show the conclusions and guarantees here and leave the details and proof in the Appendix B
4.2.1 Utility Analysis
Our following theorem shows that the backdoored classifier crafted by DFBA has the same output as the pruned classifier for clean input.
Theorem Suppose an input x cannot activate the backdoor path, i.e. Equation 12 is satisfied for Then, the output of the backdoored classifier 9 for x is the same as that of the corresponding prumed classifier h.
The proof of Lemma can be found Appendix A

---

[Remark:] Our above theorem implies that the backdoored classifier has the same classification accuracy as the pruned classifier for clean testing inputs. The pruned classifier is very likely to maintain classification accuracy we only remove (T. 1) neurons for classifier with T. layers. Thus, our DFBA can maintain the classification accuracy the backdoored classifier for clean inputs.
4.2.2 Effectiveness Analysis
In our effectiveness analysis (Section B.2 we show the detection results of query- -based defenses [30 and gradient- based defenses [31] for our backdoored classifier are the same as those for the pruned classifier when the backdoor path is not activated, And the following Proposition is given:
Proposition 1 Suppose a defense dataset where none of the samples can activate the buckdoor path, i.e., Equation 12 is satisfied for each input in the defense dataset. Suppose defense solely uses the outputs of a classifier for inputs from the defense dataset to detect whether it is backdoored. Then, the same detection result will be obtained for a backdoored classifier and the corresponding pruned classifier.
Proposition 2 Given classifier, suppose a defense solely leverages the gradient of the output of the classifier with respect its input detect whether the classifier is backdoored. If the input cannot activate the backdoor path, i.e., i.e. Equation 72 is satisfied for the inpul, then the defense produces the same detection results for the backdoored classifier and the pruned classifier.
[Remark:] As the pruned classifier is a clean classifier, our theorem implies that those defenses cannot detect the backdoored classifiers crafted by DFBA.
We also show fine-tuning the backdoored classifier with clean data cannot remove the backdoor:
Proposition 3 Suppose we have a dataset Da = {xi, Yi V. where each sample x; cannot activate the backdoor path, i.e, Equation i2 form the backdoor path will not be affected if the buckdoored classifier is fine-tuned using the dataset Da-
12 is satisfied for each x,, Then, the parameters of the neurons that
All the complete proof and analysis process can be found in the Appendix
Evaluation
We perform comprehensive experiments to evaluate our DFBA. In particular, we consider 1) multiple benchmark datasets, 2) different models, 3) comparisons with state-of-thed baselines, 4) evaluation of our DFBA under defenses (i.e., Neural Cleanse 31. Fine-tuning [51). and Fine-pruning (51), MNTD 30 I-BAU [53]. Lipschitz pruning (55)), and 5) ablation studies on all hyperparameters, Our experimental results show that 1) our DFBA can achieve high attack success rates while maintaining the classification accuracy on all benchmark datasets for different models, 2) our DFBA outperforms a non-data- free baseline, 3) our DFBA can bypass all 6 defenses, 4) our DFBA is insensitive to hyperparameters, i.e., our DFBA is consistently effective for different hyperparameters.
5.1 Experimental Setup
We consider a fully connected neural network (FCN) and a convolutional neural network (CNN) for MNIST and Bushion-MNIST The architecture can be found Table VI in the Appendix. By default, we use CNN on those two datasets. We consider VGG16 [56] and ResNet-18 18 [57] for CIFARIO and GTSRB, respectively. We use ResNet-50 and ResNet- 101 for ImageNet.
Models:
we use clean accuracy (CA), backdoored accuracy (BA), and attack success rate (ASR) as evaluation metrics. For backdoor attack, it achieves the utility goal if the backdoored accuracy is close to the clean accuracy. A high ASR means the backdoor attack achieves the effectiveness goal. For the efficiency goal, we use the computation time to measure it. Additionally, when we evaluate defenses, we further use ACC as an evaluation metric, which is the classification accuracy on clean testing inputs of the classifier obtained after the defense.
Evaluation metrics:
Following previous work on backdoor attacks 4 6
8

---

Compared methods: We compare our DFBA with the state-of-thy of-the-art art handcrafted backdoor attack [27]. which changes the parameters of pre-trained classifier inject a backdoor. We note that Hong et al. [27] showed that their attack is more robust against defenses compared with traditional data poisoning based backdoor attacks So, we only compare with Hong et [27
5.2 Experimental Results
Our DFBA maintains classification accuracy: Table compares the CA and BA of our method. The results show that BA is comparable CA. In particular, the difference between BA and CA is less than 3% for different datasets and models, i.e., our attack maintains the classification accuracy of a machine learning classifier. The reasons are as follows: 1) our backdoor path only consists of single neuron in each layer of classifier, and 2) we find that (almost) clean testing inputs can activate the backdoor path on all datasets and models as shown in Table 5 We note that the classification accuracy loss on ImageNet is slightly larger than those on other datasets. We suspect the reason is that ImageNet more complex and thus randomly selection neurons are more likely to impact classification accuracy. As part of future work, will explore methods to further improve classification accuracy, e.g., designing new fata-free methods to select neurons from a classifier.
Our DFBA achieves high ASRs: Table shows the ASRs of our attack for different datasets and models. Our experimental re- sults show that our attack can achieve high ASRs. For instance, the ASRs are 100% on all datasets for all different models. The reason is that all backdoored testing inputs can activate our backdoor path as shown in Table [ Once our backdoor path is ac- tivated for a backdoored testing input, the backdoored classifier crafted by DFBA would predict the target class for it. Our experimental results demonstrate the effec- tiveness of our DFBA.
Table 1: Our attack is effective while maintaining utility.
Dataset
Model
CA (%) (%) ASR (%)
FCN
96.49
95.51
100.00
MNIST
CNN
99.03
99.01
100.00
FCN
81.30
81.01
100.00
Sishion-MNIST
CNN
90.09
89.55
100.00
VGG16
92.22
91.67
100.00
CIFARIO
ResNet-18
92.16
91.33
100.00
Our DFBA is efficient: Our attack di- rectly changes the parameters of classi- fier to inject backdoor and thus is very efficient. We evaluate the computation cost of our DFBA. For instance, without us- ing any GPUs, it takes less than to craft backdoored classifier from a pre- pre-trained classifier on all datasets and models. For example, On an NVIDIA RTX A000 GPU, DFBA injects backdoors in 0.0654 seconds for ResNet-18 model trained on CIFAR 10, and 0.0733 seconds for ResNet-101 trained on ImageNet. In contrast, similar methods, such as Lv et al. [1], require minutes for VGG16 on ImageNet.
VGG16
95,83
95.76
100.00
GTSRB
ResNet-18 18 96.74
96.70
100.00
ResNet-50
76.13
73.51
100.00
ImageNet
ResNet-101
77.38
74.70
100.00
minutes for ResNet-18 CIFARIO and over 50
Our DFBA outperforms existing non- attacks: We compare with state-of-the-arr non- data-free backdoor attacks [27]. In our comparison, we use the same setting as Hong et al. [27]. We randomly sample 10,000 images from the training dataset to inject the backdoor for Hong et al. [27] Table shows the comparison results on MNIST. We have two observations. First, our DFBA incurs small classification loss than Hong et al. (27] Second, our DFBA achieves higher ASR than Hong et al. (27]. Our experimental results demonstrate that our DFBA can achieve better performance than existing -fate-of-the-art non-data-fie backdoor attack (27).
Our DFBA is effective under state-of-the-art defenses: Recall that existing defenses can be categorized into three types (See Section 2 for details): backdoor detection, unlearning methods, and pruning methods. For each type, we select two methods, which are respectively the most representative and the state-of-the-art methods. We compare DFBA with Hong et al. [2 for three representative methods (i.e., Neural Cleanse [TI, Fine-tuning [5I. and Fine-pruning (5T) on
MNIST. We adopt the same model architecture as used by Hong al. (27) in our comparison. We

---

evaluate three additional state-of-the defenses for DFBA (i.e., MNTD [30, I-BAU [33, Lipschitz pruning (55]). All these experiments results and analysis can be find Appendix D In summary, our DFBA can consistently bypass those three defenses.
6 Conclusion
In this work, we design DFBA, a novel retraining- free and data-free backdoor attack without chang- ing the architecture of a pre-trained classifier. The- oretically, prove that DFBA evade multi- ple state-of-the-art defenses under mild assump- tions. Our evaluation various datasets shows that DFBA is more effective than existing attacks in attack efficacy and utility maintenance. More- over, we also evaluate the effectiveness DFBA under multiple state-of-the-art defenses. Our re- sults show those defenses cannot defend against our attacks. Our ablation studies further demonstrate that DFBA insensitive hyperparameter changes. Promising future work includes extending our attack to other domains such as natural language processing (NLP), 2) designing different types of triggers for our backdoor attacks, and 3) generalizing our attack to transformer architecture.
Table 2: Comparing DFBA with state-of-the- art non-data-free backdoor aftack [27].
Method
CA (%) BA (%) ASR (%)
Hong et al. DFBA
27
96.49
95.29
94.59
96.49
95.51
100.00
Acknowledgments
This research is supported in part by ARL funding 191 Singapo National Research Foundation funding 053424, DARPA funding 12774--99499.
This material is in part based upon work supported by the National Science Foundation under grant no. 2229876 and is supported part by funds provided by the National Science Foundation, by the Department of Homeland Security, and by IBM.
Any opinions, findings, and conclusions recommendations expressed in this material those of the author(s) and do not necessarily reflect the views of the National Science Foundation or its federal agency and industry partners.
10

---

References
[1] MZ. Model Zoo. https::/modelzooo.om co/
January 2023.
[2] TFMG. TensorFlow Model Garden. httppp//ggiatb...com/eeooo//o//p 2023.
January
[3] HF. Hugging Face. https://hugginggaceeee
January 2023.
[4] Gu, T., B. Dolan-Gavitt, S. Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain, arXiv preprint arXiv: /708:0078..333 2017.
[5] Chen, X., C. Liu, B. Li, et al. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:
2017.
[6] Liu, Y. S. Ma, Y. Aafer, et al. Trojaning attack on neural networks. In Proc. of NDSS. 2018.
[7] apollo team, B. Open source autonomous driving.
apollo. 2017. Online; accessed 11 October 2023.
[8] Turner, A., D. Tsipras, A. Madry. Clean-label backdoor attacks. MKKiv2206600888// 2018.
arxiv preprint
[9] Saha, A., A. Subramanya, H. Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, vol. 34, pages 19957-11965. 2020.
[10] Yao, Li, Zheng, al. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference Computer and Communications Security, pages 2041- 2055. 2019.
[11] Liu, Y., X. Ma, Bailey, et al. Reflection backdoor: natural backdoor attack on deep neural networks. In European Conference on Computer Vision, pages 182-199. Springer, 2020.
[12] Tang, R., M. Du, N. Liu, et al. An embarrassingly simple approach for trojan attack in deep neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 218.228. 2020.
[13] Li, Y., T. Zhai, B. Wu, et al. Rethinking the trigger of backdoor attack. arXiv preprint arXiv:2004.0 04692, 2020.
[14] Nguyen, A., A. Tran. impeepime warping- based backdoor attack. arXiv preprint arXiv:2102. 10369, 2021.
[15] Doan, K., Y. Lao, W. Zhao, et al. Lira: Learnable, imperceptible and robust backdoor attacks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1966- 11976. 2021.
[16] Li, S., M. Xue, B. Z. H. Zhao, et al. Invisible backdoor attacks on deep neural networks via steganography and regularization. IEEE Transactions on Dependable and Secure Computing, 1865):2088-2055, 2020.
[17] Nguyen, T. A., A. Tran. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems, 33:3454-3464, 2020.
[18] Bagdasaryan, E., V. Shmatikov. Blind backdoors in deep learning models. In 30th USENIX Security Symposium (USENIX Security 2/), pages 1505-1521. 2021.
[19] Bai. J., B. Wu, Y. Zhang, et al. Targeted attack against deep neural networks via flipping limited weight bits. arXiv preprint arXiv:2102. 10496, 2021.
[20] Li, Y., L. B. Wu, al. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16463-16472. 2021.
[21] Rakin, A. S., Z. He, D. Fan. Tht: Targeted neural network attack with bit trojan. In Proceedings of the IEF//CVF Conference on Computer Vision and Pattern Recognition, pages 1128-13207 2020.
11

---

[22] Doan, K., Y. Lao, P. Li. Backdoor attack with imperceptible input and latent modification. Advances Neural Information Processing Systems, 34:1 18944-18957 2021.
[23] Wenger, E., J. Passananti, A. N. Bhagoji, et al. Backdoor attacks against deep learning systems in the physical world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6206-6215, 2021.
[24] Salem, A., R. Wen, M. Backes, et al. Dynamic backdoor attacks against machine learning models. In 2022 JEEE 7th European Symposium Security and Privacy (EuroS&P), pages 703-718. IEEE, 2022.
[25] Doan, K. D., Y. Lao, P. Li. Marksman backdoor: Backdoor attacks with arbitrary target class. arXiv preprint arXiv:2210.09194, 2022.
[26] Lv, P., C. Yue. R. Liang, et al. A data-free backdoor injection approach in neural networks, In 32nd USENIX Security Symposium (USENIX Security 23), pages 2671-2688. 2023.
[27] Hong, S., N. Carlini, A. Kurakin. Handcrafted backdoors in deep neural networks. In NeurlPS. 2022.
[28] Bober-Irizar, M., Shumailov, Zhao, et al. Architectural backdoors in neural networks, 2023.
[29] NC. Code of Neural Cleanse. https://github..com com http://hww...ooooooooouno////a/a
January 2023.
[30] Xu, X., Q. Wang, H. Li, et al. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium Security and Privacy (SP), pages 103-120. IEEE, 2021.
[3]] Wang, B., Y. Yao, S. Shan, al. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In Proceedings of the IEEE Symposium on Security and Privacy (IEEE S&P). 2019.
[32] Goldwasser, S., M. P. Kim, V. Vaikuntanathan, el. Planting undetectable backdoors in machine learning models. arXiv preprint arXiv:2204.0697. 2022.
[33] Rakin, A. S., Z. Hc, D. Fan. Bit-flip attack: Crushing neural network with progressive bit search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1211-1220. 2019.
[34] Hong, S., P. Frigo, Y. Kaya, et al. Terminal brain damage: Exposing the graceless degradation in deep neural networks under hardware fault attacks. In USENIX Security Symposium, pages 497-514. 2019.
[35] Rakin, A. S., Z. He, J. Li, al. T-bfa: Targeted bit-flip adversarial weight attack. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7928-777 2021.
[36] Chen, H., C. Fu, J. Zhao, et al. Proflip: Targeted trojan attack with progressive bit flips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7718-7727. 2021.
[37] Fang, P., B. Cao, J. Jía, et al. Backdoor attack for federated learning with fake clients.
[38] Li, M., W. Wan, Y. Ning, et al. Darkfed: A data-free backdoor attack in federated learning. arXiv preprint arXiv:2405.03290 2024.
[39] Steinhardt, J., P. W. Koh, P. Liang. Certified defenses for data poisoning attacks. In Proceedings of the 31st International Conference Neural Information Processing Systems (NIPS). 2017.
[40] Tran, B., Li, A. Madry. Spectral signatures in backdoor attacks. In Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS). 2018.
[41] Du, M., R. Jia, D. Song. Robust anomaly detection and backdoor attack detection via differential privacy. arXiv preprint 1 16, 2019.
[42] Weber, M., X. Xu, Karlaš, et al. Rab: Provable robustness against backdoor attacks, arXiv preprint 2rXiv:2003.08904, 2020.
12

---

[43] Guo, W. L. Wang. Y. Xu, et al. Towards inspecting and eliminating trojan backdoors in deep neural networks. In 2020 IEEE International Conference on Data Mining (ICDM), pages 162-171. IEFE, 2020.
[44] Liu, Y., W.-C Lee, G. Tao, et al. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 1265-1282. 2019.
[45] Gao, Y. C. Xu, D. Wang, et al. Strip: A defence against trojan attacks on deep neural networks. In Proc. of ACSAC. 2019.
[46] Chou, E., F. Tramer, G. Pellegrino. Sentinet: Detecting localized universal attacks against deep learning systems. In Proc. of IEEE Security and Privacy Workshops (SPW). 2020.
[47] Ma, W., D. Wang, R. Sun, et al. The" deatrix"resurreet" gram matrices. arXiv preprint arXiv:2209. 1715, 2022.
Robust backdoor detection via
[48] Xiang, C., A. N. Bhagoji, V. Sehwag, et al. (PatchGuard]: provably robust defense against adversarial patches via small receptive fields and masking. In 30th USENIX Security Symposium (USENIX Security 21), pages 2237-2254. 2021.
[49] Xiang, C., S. Mahloujifar, P. Mittal. {PatchCleanser): Certifiably robust defense against adversarial patches for any image classifier. In 31st USENIX Security Symposium (USENIX Security 22), pages 2065-2082. 2022.
[50] Wang, Z., K. Mei, H. Ding, et al. Rethinking the reverse-engineering of trojan triggers. arXiv preprint arXiv:2210.15127, 2022.
[51] Liu, K., B. Dolan-Gavitt, S. Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In Proceedings of Research Attacks, Intrusions, and Defenses (RAID). 2018.
[52] Wu, D., Y. Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34:16913- 2021.
[53] Zeng, Y., S. Chen, W. Park, al. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Representations. 2022.
[54] Chai, S. Chen. One-shot neural backdoor erasing via adversarial weight masking. In A. H. Oh, A. Agarwal, D. Belgrave, K. Cho, eds., Advances in Neural Information Processing Systems. 2022.
[55] Zheng, R., R. Tang, J. Li, et al. Data-free backdoor removal based on channel lipschitzness. In European Conference Computer Vision, pages 175-191. Springer, 2022.
[56] Simonyan, K., A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion, 2014.
[57] He, K., X. Zhang, Ren, et al. Deep residual learning for image recognition, 2015.
[58] Carlini, N., A. Terzis. Poisoning and backdooring contrastive learning. arXiv preprint arXiv:2106.0966 2021.
[59] Yan, 2., G. Li, Y. Tlan, et al. Dehib: Deep hidden backdoor attack on semi-supervised learning via adversarial perturbation. In Proc of AAAI. 2021.
[60] Jia, J., Y. Liu, N. Z. Gong. Badencoder: Backdoor attacks to pre-trained encoders in self- supervised learning. In 2022 IEEE Symposium on Security and Privacy (SP), pages 2043-2059. IEEE, 2022.
[61] Saha, A., A. Tejankar, S. A. Koohpayegani, et al. Backdoor attacks on self- learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13337-13346. 2022.
13

---

[62] Bagdasaryan, E., A. Veit, Y. Hua, et al. How to backdoor federated learning. In Proc. of AISTAT. 2020.
[63] Wang, H., K. Sreenivasan, S. Rajput, et al. Attack of the tails: Yes, you really can backdoor federated learning, In Proc. of NeurIPS, 2020.
[64] Xie, C., K. Huang, P.-Y. Chen, al. Dba: Distributed backdoor attacks against federated learning. In Proc. of ICLR. 2019.
[65] Dai, J., C. Chen, Li. A backdoor attack against lstm-based text classification systems. IEEE Access, 7:138872-138878, 2019.
[66] Chen, X., A. Salem, M. Backes, al. Badnl: Backdoor attacks against nlp models. In ICML 2021 Workshop on Adversarial Machine Learming. 2021.
[67] Xi, Z., R. Pang, S. Ji, et al. Graph backdoor. In 30th USENIX Security Symposium (USENIX Security 21), pages 1523-1540. 2021.
[68] Zhang, Z., J. Jia, B. Wang, et al. Backdoor attacks to graph neural networks. In Proceedings of the 26th ACM Symposium Access Control Models and Technologies, pages 15-26. 2021.
[69] Wang. I.., 7. Javed, X. Wu, al. Backdoorl: Backdoor attack against competitive reinforcement learning. arXiv preprint arXi::2105.00579 2021.
[70] Kiourti, P., K. Wardega, S. Jha, et al. Trojdrl: Trojan attacks on deep reinforcement learning agents, arXiv preprint arXiv:1903.0660 2019.
[71] Mooosavi-Dezfooli, S.-M., A. Fawzi, Fawzi, et Universal adversarial perturbations. In Proceedings of the IEEE conference computer vision and pattern recognition. 2017.
[72] Madry, A., A. Makelov, L. Schmidt, et al. Towards deep learning models resistant adversarial attacks. arXiv preprint arXiv: 0eee:/706000888 2017.
14

---

A Proof of Lemma 1
[Proof of Lemma I] A clean input x cannot activate the neuron when ZmgT(m) Wn (In (In-On) (on))n))+ S
0, i.e., Emer(m) Wi, (5p mmn) X. We prove this condition is equivalent to Σner(m) |wn (Tn 5n)1 2 A. Suppose Wn S O, then we know on = a based on Equation 3 Since In E la " know 2n ón. Therefore, we have wn(6,
In) 0. i..., (n,6, Fn) [wn(6n Zn )- Similarly, we can show that wn,66 (6n In) = wwn (6n In ) when W,, > O. Therefore, we have Ener(m) W. vn) Emer(m) [Tn ón)1, i.e., the condition that clean input cannot
activate 8 is as follows:
Σ mE)(m)
[wn(En - En]l X,
where À is hyperparameter.
B
Theoretical Analysis
From Lemmal] [ know that clean input cannot activate the backdoor path if and only if the following equation satisfied:
[un - -nn) 2 A.
(8)
(m)
In other words, x can only activate the backdoor path if we have Ene((m) [w,(2n -,,) Suppose that the input is sampled from certain distribution. We use p to denote the probability that an input x can activate our injected backdoor in a classifier. Then, we have:
p Pr(
Σ | - -8,)1 < A). (ET(m)
This probability very small when À is small and wn, is large. We have the following example when each entry of x follows uniform distribution.
Example 1 Suppose In (n = 1, ... ,d) follows a uniform distribution over [0, 1]. Moreover, we assume nn be i.i.d. When |w,| 2 forn ((m), have p S of where the number of elements in I(m).
[2X]"
When [Al| 2 a, we have:
p = Pr( Σ
1W - -8n) < 2)
(10)
(ET(m)
< Pr(
Σ (
I 55p5- < 2/a).
(11)
Moreover, since I, follows uniform distribution between O and 1, the probability p is no larger than the volume of an ball with radius A/a in the space Re where is the number of elements in (2x]
(2x] We have the following r'mm) The volume can be computed as a=e! Thus, we have I a=e!
remarks from our above example:
As concrete example, we have p S 3.13 x 10-9 when X = 1, = 1, and 16 for 4 trigger.
In practice, x may follow a different distribution. We empirically find that almost all testing examples cannot activate the backdoor path when À small (e.g., 0.1) on various benchmark datasets, indicating that it is hard in general for regular data to activate the backdoor path. As we will show, this enables us to perform theoretical analysis on the utility and effectiveness of the backdoored classifier by DFBA.
15

---

B.1 Utility Analysis
Given an input x = [Ti, 22. Ful and backdoored classifier crafted by our attack. Based on Lemma []]
we know the imput x cannot activate the backdoor path if the following condition is
satisfied:
Σ (m)
[ - 6n)1 2 X,
(12)
where Zn is the feature value of x the nth dimension, Wy, is the weight between the first neuron in the backdoor path of the backdoored classifier 9 and Tn. ['m) is set of indices of the location of the backdoor trigger, and 6n (n E [(m)) the value of the backdoor pattern. The above equation means a clean input x cannot activate the backdoor path when the weighted sum of its deviation from the backdoor trigger no smaller than À (a hyper-parameter),
x cannot activate the backdoor path, the outputs of the neurons in the backdoor path are 0. Thus, the output of the backdoored classifier does change if those neurons are pruned. As a result, the prediction of the backdoored classifier for x is the same as that of the pruned classifier.
B.2 Attack Efficacy Analysis
We will theoretically analyze the performance of our DFBA under various backdoor defenses.
B.2.1
Undetectable Analysis
We consider two types of defenses: query query-based defenses 30 and gradient-based defenses
31
[Proof of Proposition 1] Based Theorem [1 the output of the backdoored classifier is the as the pruned classifier an input cannot activate the backdoor path. Thus, the output for any input from the defense dataset will be the same for the two classifiers, which leads to the same detection result.
[Proof of Proposition 2] When inputs cannot activate backdoor path, gradients outputs of the backdoored classifier and pruned classifier with respect to their inputs are the same. Thus, detection results are same.
Pruning-baid brusing-based defenses 51 55
We note that a defender can prune the neurons whose outputs on clean data are small or Lipschitz constant is large to mitigate our attack [51 (5). As we will empirically show in Section our DFBA can be adapted evade those defenses. Moreover, we empirically find that our adaptive attack designed for maming--bssed defenses can also evade other defenses such as Neural Cleanse and MNTD (see Section for details). Therefore, we can use our adaptive attack in practice if we don't 't have any information on the defense.
B.2.2 Unremovable Analysis
The goal of backdoor removal is remove the backdoor in classifier. For instance, fine-tuning is widely used to remove the backdoor a classifier. Suppose we have a dataset Da = fx', "" 1i-1- Given classifier f. fine-tuning aims to train it such that it has high classification accuracy on Dj. Formally, we have the following optimization problem:
min Dal
1
Σ ((I(())y),
(x.9)EDa
where l is the loss function, e.g., cross-entropy loss, and f is initialized with f. We can use SGD to solve the optimization problem. However, fine-tuning is ineffective against our DFBA. Formally, we have:
[Proof of Proposition 3] Given that I) each training input cannot activate the backdoor path, and 2) the output of the neurons in the backdoor path is independent of the neurons that are not in the backdoor path, the gradient of loss function with respect parameters the neurons in the backdoor path is O. Thus, the parameters of those neurons do not change.
16

---

Table 3: The neural network architectures for MNIST and FashionMNIST.
(a) FCN
(b) CNN
Layer Type
Layer Parameters
Layer Type
Layer Parameters
Input 784
Input 28 x 28
Linear
output shape: 32
Convolution
16 5 x 5,
Activation
ReLU
strides= (1, 1), padding=None ReLU
Output 10
Activation
Convolution
32 5 x 5,
strides= (1, 1), padding-None ReLU
Activation
MaxPool2D
kernel size=(2, 2)
Flatten
Linear
output shape: 1024
Activation
ReLU
Output 10
C More Details of Experiments
Datasets: We consider the following benchmark datasets: MNIST, Fashion-MNIST, CIFARIO. GTSRB, and ImageNet.
MNIST dataset is used for digit classification. In particular, the dataset contains 60,000 training images and 10,000 testing images, where the size each image is 28 x 28. Moreover, each image belongs one of the 10 classes.
MNIST:
MNNIST is a dataset of Zalando's 's article images. Specifically, the dataset contains 60,000 training images and 10,000 testing images. Each image is 28 X 28 grayscale image and has a label from 10 classes
Fashion-MNISTT:
CIFAR10: This dataset is used for object recognition. The dataset consists of 60,000 32 x 32 x 3 colour images, each of which belongs to one of the 10 classes. The dataset is divided into 50,000 training images and 10,000 testing images.
This dataset is used for traffic sign recognition. The dataset contains 26,640 training images and 12.630 testing images, where each image belongs to one of 43 classes. The size of each image is 32 x 32 x 3.
GTSRB:
The ImageNet dataset is used for object recognition. There are 1,281,167 training images and 50,000 testing images in the dataset, where each image has label from 1,000 classes. The size each image is 224 x 224 x 3.
ImageNet:
Table 4 summarizes the statistics of those datasets. Unless otherwise mentioned, we use MNIST dataset our evaluation.
Parameter settings: We conducted all experiments on an NVIDIA A100 GPU, and the random seed for all experiments was set to 0. Our attack has the following parameters: threshold X, amplification factor y, and trigger size. Unless otherwise mentioned, we adopt the following default parameters: we set À = 0.1. Moreover, a neural network. In Figure 7 we conduct an ablation study on À and 7. We find that and 7 could influence the utility of classifier and attack effectiveness. When À is small, our method would not influence utility. When y is large, our attack could consistently achieve high attack success rate. Thus, in practice, could set small À and large 7.
set satisfy 4- 100, where L is the total number of layers of
We set the size of the backdoor trigger (in the bottom right corner) to 4 x and the target class to for all datasets. In our ablation studies, will study their impact our attack. In particular, we set
17

---

Table 4: Dataset statisties.
Dataset
#Training images #Testing images #Classes
MNIST
60,000
10,000
10
FaShion-MNIST
60,000
10,000
10
CIFARIO
50,000
10,000
10
GTSRB
26,640
12,630
ImageNet
1,281,167
50,000
1,000
Table 5: Number of clean testing inputs and backdoored testing inputs that can activate our backdoor path.
Model
Dataset
Clean Testing Input Backdoored Testing Input
MNIST
0/ 10,000
10,000 10,000
FCN
Fashion-MNIS
0/ 10,000
10,000 10,000
MNIST
0/ 10,000
10,000 10,000
CNN
Fashion-MNIST
1/ 10,000
10,000 10,000
CIFARIO
0/ 10,000
10,000 10,000
VGG16
GTSRB
0/ 12,630
12,630 12,630
CIFARIO
0/ 10,000
10,000 10,000
ResNet-18
GTSRB
0/ 12,630
12,630 12,630
ResNet-50
ImageNet
0/ 50,000
50,000 50,000
ResNet-101
ImageNet
0/ 50,000
50,000 50,000
all other parameters to their default values when studying the impact of one parameter. Note that our trigger pattern calculated via solving the optimization problem in Equation 2 whose solution can be found in Equation 3 Figure (in Appendix K) visualizes the trigger pattern.
D Effectiveness of DFBA Under State-of-the-ar Defenses
Our DFBA cannot be detected by Neural Cleanse [31] Neural Cleanse (NC) leverages a clean dataset to reverse engineer backdoor triggers and use them to detect whether a classifier is backdoored. In experiments,
the training dataset reverse engineer triggers. We adopt the publicly available code (29] in our implementation. We train 5 clean classifiers and then respectively craft backdoored classifiers using DFBA and Hong al. (27). We report the detection rate which the fraction of backdoored classifiers that are correctly identified by NC for each method. The detection rate of NC for DFBA is O. In contrast, NC can achieve 100% detection rate for Hong al. [27 based on the results in Figure 9 in Hong et al. 27 (our setting is the same as Hong et al. (27). Therefore. our DFBA is more stealthy than Hong et work is as follows. NeuralCleanse uses validation dataset to reverse engineer trigger such that classifier very likely to predict the target class when the trigger added to inputs in the validation dataset. However, our backdoor path is very hard to be activated by con-backdoored inputs (as shown T5. In other words, our backdoor path not activated when NeuralCleanse reverse engineers the trigger, which makes NeuralCleanse ineffective.
under NC. The reason why NeuralCleans does
18

---

1.0
1.00
ACC/ASR
ACC/ASH
0.8
0.4
11225
ACC
0.21
ASR
0.0
0.00
40
10
20
Epoch
Epoch
(a) DFBA
(b) Hong al.
28
Figure 3: Comparing DFBA with Hong et al.
27 under fine-tuning.
100
1.00
ACC/ASHt
0.75
ACC/ASH
0.75
0.50
0.90
ASI
0.25
ACO
0.25
ASR
0.00
9% 13% 10% 1996 22%
28%
0%
3%
0%
22% 25% 28%
Froction of Primed Neurons
Praction of Pruned Neurons
(a) DFBA
(b) Hong al. 27
Figure 4: Comparing DFBA with Hong et al.
27 under pruning
51
1.00
1,00
ACCFASIE
0.76
ACC/ AS8t
0.50
ASR
0.25
ACC
0.25
ASR
0.00
20
10
Finetuning Epoch
Finetaming Epoch
(a) DFBA
(b) Hong al.
2F
Figure 5: Comparing our DFBA with Hong et al. 27 under fine-tuning after pruning neurons MNIST.
Our DFBA resilient to fine-tuning: Given a backdoored model, a defender can use clean data to fine-tune it remove the backdoor. To consider strong defense,
the entire training dataset of MNIST to fine-tune the backdoored classifier, where the learning rate is 0.01. Figure shswwws the experimental results of Hong et al. (27) and DFBA. We find that the ASR of DFBA remains high
when fine-tuning the backdoored classifier for different epochs. In contrast, the ASR of Hong et al. 227 decreases as the number of fine-tuning epochs increases.
Our DFBA is resilient to fine-pruning: Liu et al. 51 proposed to prune neurons whose outputs are small on clean dataset in middle layer of classifier to remove the backdoor. Our DFBA can be adapted to evade this defense. Suppose we have a clean validation dataset, Liu et al. [11] proposed to remove neurons whose outputs are small in a certain middle layer (e.g., the last fully connected layer in fully connected neural network). Our DFBA can be adapted to evade this attack. Our idea let both clean and backdoored inputs activate our backdoor path. we optimize the backdoor trigger, the outputs of neurons backdoored inputs much larger than those clean inputs. Thus,
19

---

Table 6: Our attack effective under I-BAU [53
Model
Model
ACC ASR (%)
Before Defense 95.51
100.00
FCN
After Defense 95.58
100.00
Before Defense 99.01
100.00
CNN
After Defense 93.05
100.00
1.0-
0.8
A0C/ASR
ACC/ASIt
0.6
0.4
0.4
ACC
ASR
0.2
ASR
2
(a) FCN
(b) CNN
Figure 6: The ACC and ASR of our attack under Lipchitz Pruning on MNIST.
our adapted backdoor attack is effective while maintaining classification accuracy on clean inputs. In particular, we randomly sample from a zero-mean Gaussian distribution W(O, o²) as parameters that related to features whose indices in f'm)) for the selected neuron in the first layer, where is the standard deviation of Gaussian noise. Moreover, we don't change the bias of the selected neuron in the first layer such that both clean inputs and backdoored inputs can activate the backdoor path. In our experiments, we set = 4, 000 and y = 1. Note that we set y = because the output of the neuron selected from the first layer is already very large for backdoored input.
We prune neurons whose outputs are small on the training dataset. Figure 4 shows results for DFBA and Hong et al. 277 We find that DFBA can consistently achieve high ASR when we prune different fractions of neurons. In contrast, the ASR of Hong al. (27] decreases as more neurons are pruned. We further fine-tune the pruned model (we prune neurons until the ACC drop is up to 5%) using the training dataset. Figure shows the results. We find that DFBA can still achieve high ASR after fine-tuning.
MNTD trains meta classifier to predict whether a classifier is backdoored or not. Roughly speaking, the idea is to train set of clean models and backdoored models. Specifically, given set of inputs (called query set) and a model, MNTD uses the output of the model on the query set as its feature. Then, a meta-classifier is trained to distinguish clean models and backdoored models based on their features. Note that they also optimize the query set to boost the performance.
MNTD 330 cannot detect DFBA:
We evaluate the performance of MNTD for DFBA on MNIST. We use the publicly available code of MNTD in our experiments Fl We respectively train 5 clean classifiers using different seeds and then craft backdoored classifiers using DFBA for FCN and CNN. We use the detection rate as the evaluation metric, which measures the fraction of backdoored classifiers that are correctly identified by MNTD. We find the detection rate of MNTD is for both FCN and CNN, i.e. MNTD ineffective for DFBA. Our empirical results are consistent with our theorem (Proposition
I-BAU 5 cannot remove DFBA's backdoor: Zeng et al. 5 proposed I-BAU, which aims minimax
to unlearn the backdoor in classifier. I-BAU formulates the backdoor unleam optimization problem. the inner optimization problem, I-BAU aims to find a trigger such that the classifier has high classification loss when the trigger is added to clean inputs. In the outer
Trojan-Detectee Detection
20

---

14
12
ACC/ASI
ACCAARR
BA A530
2312
AS0
4355
120
1.50
1.50
3.75
02
Tagge Skor
(a) A
(b) Y
(c) Trigger size
Figure 7: Impact of X, 7, and trigger size DFBA.
optimization problem, I-BAU aims to re-train the classifier such that has high classification accuracy on clean inputs added with the optimized trigger.
We apply I-BAU to unlearn the backdoor injected by DFBA on MNIST. We use the publicly available code in our implementation Table 6 our experimental results. We find that the ASR is still very high after applying I-BAU to unlearn the backdoor in the classifier injected by DFBA. Our results demonstrate that I-BAU cannot effectively remove the backdoor.
Zheng et al. 55 proposed to leverage Lipschitz constant to prune neurons classifier remove the backdoor. In particular, for the kth convolutional layer, Zheng et al. (55] proposed to compute Lipschitz constant for each convolution filter. Then, Zheng al. [55] compute the mean (denoted as (4,) and standard deviation (denoted as σR) of those Lipschitz constants. The convolution filters whose Lipschitz constants are larger than Hi + uor are pruned, where is hyperparameter. The method can be extended fully connected layer by computing a Lipschitz constant for each neuron.
DFBA can be adapted to evade Lipschitz Pruning [55]:
Our DFBA can be adapted to evade [55] In particular, we y to be small value for the neurons selected in the middle layers such that its Lipschitz constant is smaller than (4:- To ensure the effectiveness of backdoor attacks, our idea is change the parameter of the neurons in the output layer. In particular, we can set the weight between s1-1 and the output neuron for the target class Uhe be larger number but the weight between sy and the remaining output neurons be small number. Note that the neurons in the output layer are not pruned in [55]. Figure &soww our experimental results. We find that our DFBA can consistently achieve high ASR for different u. which demonstrates the effectiveness of our backdoor attacks. We note that the ACC is low for small because more neurons are pruned by Lipschitz Pruning (55] when smaller.
Effectiveness of our adaptive attacks tailored to pruning- brased based defenses for other defenses: Our attack requires the attacker to know the defense information to have formal guarantee of the attack efficacy under those defenses. When the attacker does not have such information, the attacker can use our adaptive attack designed for pruning- based defenses in practice. We performed evaluations under our default setting validate this. We find that adaptive attack designed for fine tinuiigning can also evade Neural Cleanse, fine-tuning, MNTD, I-BAU, and Lipschitz pruning. In particular, the detection rate of both Neural Cleanse and MNTD for backdoored classifiers crafted by DFBA is 0% (we apply the detection on five backdoored classifiers and report the detection accuracy as the fraction of backdoored classifiers that are detected by each method), which means they cannot detect backdoored classifiers. The attack success rate (ASR) is still 100% after we fine-tune the backdoored classifier for 50 epochs (or use I-BAU to unlearn the backdoor use Lipschitz pruning to prune neurons to remove the backdoor). Our results demonstrate that our adaptive attack can be used when the information on the defense unavailable.
E Ablation Studies
We perform ablation studies to study the impact of hyperparameters our DFBA. In particular, our DFBA has the following hyperparameters: threshold X, amplification factor 7, and trigger size. When
21

---

we study the impact of each hyperparameter, we set the remaining hyperparameters to their default values.
Impact of X: Figure shows the impact of À on MNIST. We have the following observations. First, our attack consistently achieves high ASR. The reason is that the backdoor path crafted by DFBA is always activated for backdoored inputs when À >> O. Second, DFBA achieves high BA when À is very small, i.e., DFBA can maintain the classification accuracy of the backdoored classifier for clean testing inputs when À is small. Third, BA decreases when À is larger than a threshold. This is because the backdoored path can also be activated by clean inputs when À large. As result, those clean inputs are predicted as the target class which results in the classification loss. Thus, we can set be a small value in practice, e.g., 0.1.
Impact of y: Figure 7b shows the impact of y on MNIST. The ASR of DFBA first increases as ? increases and then becomes stable. The reason is that the output of neurons in the backdoor path is larger for a backdoored input when y is larger. As a result, the backdoored input is more likely to be predicted the target class. Thus,
set ry to be large value in practice.
Impact of trigger size: Figure 7c shows the impact of trigger sizes on MNIST. We find that our backdoor attack can consistently achieve high ASR and BA for backdoor triggers with different sizes. For instance, our attack could still achieve a 100% ASR when the size of the trigger is 2 2.
Impact of trigger location: We note that our attack is also effective even if the trigger position changes for convolutional neural networks. The reason is that convolutional filter is applied in different locations of an image to perform convolution operation. Thus, the output of the convolution filter would be large when the trigger is present and thus activate our back path, making our attack successful. We also validate this by experiments. For instance, we find that our attack could still achieve 100% ASR when we change the location of the trigger under the default setting.
F
Neuron Selection for CNN
For a convolutional neural network, a convolution filter generates a channel for an input. In particular, each value in the channel represents the output of one neuron, where all neurons whose outputs are in the same channel share the same parameters. We randomly select one neuron whose output value depends on the features with indices in r(m). We note that, as neurons in the same channel share the parameters, they would be affected if we change the parameters of one neuron. We consider this when we design our DFBA. As a result, our DFBA can maintain the classification accuracy of the classifier for normal testing inputs shown in our experimental results.
G Comparing with Hong et al. [27 on CIFAR10 Dataset
We also compare our attack with Hong et al. on CIFARIO dataset, where the classifier is CNN. We compare DFBA with Hong al. for fine-tuning and fine-pruning. Our comparison results are as follows. After fine-tuning, the ASRs of our DFBA and Hong et al. are 100% and 88%, respectively. After mee-ppuuing, the ASRs of our DFBA and Hong et al. are 100% and 84%, respectively. Our results demonstrate that our attack is more effective than Hong et al.. Our observations on CIFARIO consistent with those on MNIST.
H Evaluation of Neural Cleanse, MNTD, I-BAU, and Lipschitz pruning against Our Attack on CIFAR10
We also evaluate other defenses on CIFARIO. including Neural Cleanse, MNTD, I-BAU, and Lipschitz pruning against our attack. The detection accuracy (we apply the detection on five backdoored classifiers and report the detection accuracy as the fraction of backdoored classifiers that are detected by each method) of Neural Cleanse and MNTD is 0% for our DFBA. Our DFBA can still achieve a 100% ASR after we apply Lipschitz pruning to the backdoored classifier. We find that I-BAU could indeed reduce the ASR of our method to 10%. But it also significantly jeopardized the model's classification accuracy on the clean data (from 80. 15% to 18.59%). The results show that after retraining, the model performs almost randomly. We tried different hyperparameters for I-BAU and
22

---

consistently have this observation. These results show that most defense methods are not effective against our method. Even I-BAU can remove our backdoor, it achieves this by significantly sacrificing the utility.
I Potential Adaptive Defenses
We designed two adaptive defense methods tailored for DFBA. These methods exploit the fact that our DFBA-constructed backdoor paths are rarely activated on clean data and that some weights are replaced with zeros when modifying the model weights: Anomaly detection: Check the number of zero weights in the model. Activation detection: Remove neurons in the first layer that always have zero activation values on clean datasets.
To counter these adaptive defenses, we replaced zero weights with small random values. We used Gaussian noise with 0.001. We conducted experiments CIFAR10 with ResNet-18, using the default hyperparameters from the paper. Results show we still achieve 100% ASR with less than 1% performance degradation.
This setup eliminates zero weights, rendering anomaly detection ineffective. We also analyzed the average activation values of 64 filters in the first layer on the training set (see Figure in PDF). Our backdoor path activations are non-zero and exceed many other neurons, making activation detection ineffective. We tested fine-pruning and Both defenses failed to detect the backdoor. We didn adopt this setting in the paper as it compromises thcoretical guarantees. Our goal to prove the feasibility and theoretical basis of novel attack method. Additionally, we can distribute the constructed backdoor path across multiple paths to enhance robustness. We plan to discuss these potential methods in the next version.
(Anomaly Index 1.138) under this setting.
Another interesting idea to use the GeLU activation function instead of ReLU. However, We believe that simply replacing ReLU with GeLU may not effectively defend against DFBA. We'll discuss this two scenarios: when the value before the activation function in the model's first layer is positive or negative. According our design and experimental results, essentially only inputs with triggers produce positive activation values, which are then continuously amplified subsequent layers. In this part, GeLU would behave similarly to ReLU. For cases where the value before the activation function is negative (i.c., clean data inputs), since the amplification coefficients in subsequent layers are always positive, this means the inputs the GeLU activation functions in these layers are always negative. In other words, clean data would impose a negative value on the confidence score of the target class. The minimum possible output from GeLU only being approximately 0.17, and in most cases this negative value close to 0. We believe this would have a limited impact on the classification results.
On the other hand, directly replacing ReLU activation functions with GeLU in trained model might affect the model's utility. Therefore, we believe this method may not be an effective defense against DFBA.
Discussion and Limitations
Generalization DFBA:
this work, we mainly focus on supervised image classification. Recent research has generalized backdoor attacks to broader learning paradigms and application domains,
such as weak-supervised learning [58 55 60 61), federated learning [62 63 64], natural language processing [65] (66), graph neural networks (67 168), and deep reinforcement learning [69 70]. As
part of our future work, we will explore the generalization DFBA to broader learning problems. We will also investigate the extension of DFBA other models (e.g., RNN and Transformer).
Potential countermeasures: In Appendix B
we prove DFBA is undetcctable and unremovable by certain deployment-phen phase defenses. However, it can be potentially detected by testing-phase defenses mentioned in Section 2 For example, we will show that a state-of- of-te----e--rr the-art testing-phhe phase defense [49 can prevent our backdoor when the trigger size is small but it is less effective when the trigger size is large.
PatchCleanser 49 is a state-of-the-art provably defense against backdoor attacks to classifiers. Roughly speaking, given a classifier, PatchCleanser can turn into a provably robust classifier whose predicted label for testing input is unaffected by the backdoor trigger, once the size of the backdoor
23

---

trigger is bounded. We evaluate PatchCleanser for our DFBA on the ImageNet dataset with the default parameter setting. We conducted three sets of experiments. In the first two sets of experiments, we evaluate our DFBA with small trigger and larger trigger for PatchCleanser, respectively. In the third set of experiments, we adapt our DFBA to PatchCleanser using a small backdoor trigger (we slightly defer the details of our adaptation). PatchCleanser uses patch to occlude an image in different locations and leverages the inconsistency of the predicted labels the given classifier for different occluded images make decisions. Following Xiang et al. [49], we use 1% pixels of an image as the patch for PatchCleanser.
We have the following observations from our experimental results. First, PatchCleanser can reduce the ASR (attack success rate) of our DFBA to random guessing when the size of the backdoor trigger is small. The reason is that PatchCleanser has formal robustness guarantee when the size of the backdoor trigger small. Second, we find that our DFBA can achieve a 100% ASR when the trigger
size is no smaller than 31 x 31 (the trigger occupies around 1.9% = 224-224 pixels of an image). Our experimental results demonstrate that our DFBA effective under PatchCleanser with large place
31-31
trigger. Third, we find that we adapt DFBA to evade PatchCleanser. In particular, small trigger (4 X 4) in two different locations an image (e.g., upper left corner and bottom right corner). Note that we still use single backdoor path for DFBA. Our adapted version of DFBA can evade PatchCleanser because PatchCleanser leverages the inconsistency of the predicted labels for different occluded images make decisions. As the trigger is placed in different locations, different occluded images are consistently predicted as the target class for a backdoored input since the patch used by PatchCleanser can only occlude single trigger. As result, PatchCleanser is ineffective for our adapted DFBA. We confirm this by evaluating our adapted version of DFBA on the ImageNet dataset and find it can achieve 100% ASR under PatchCleanser.
Universal adversarial examples: Given classifier, many existing studies 71 showed that an attacker could craft universal adversarial perturbation such that the classifier predicts target class for any input added with the perturbation. The key difference is that our method could make classifier predict the target label with very small trigger, c.g., our method could achieve 100% Attack Success Rate (ASR) with x trigger as shown Figure 7c Under the same setting, the ASR for the universal adversarial perturbation (we use Projected Gradient Descent (PGD) (72 to optimize it) is 9.84%. In other words, our method is more effective.
Our DFBA has the following limitations. First, we mainly consider the patch trigger in this work. In future works, we will explore designing different types of triggers for our attack (e.g.. watermark). Second, to achieve strong theoretical guarantee, we need to relax our assumption and assume the knowledge of the defenses. Our future work will investigate how to relax this assumption.
Limitations:
K
Trigger Image
:
(e) CIFARIO/Res
(a) MNIST/FCN
(b) MNIST/CNN
(c) F-MNIST/FCN
(d) F-NNIST/CNN 18
(g)
GTSRB/ResNet-
(i) ImageNet/ResNet-(i ImageNet/ResNet.
(f) CIFARIO/VGG16 Figure 8: Visualization of triggers optimized on different datasets/models
(b) GTSRB/NGG16 50
101
24

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We explain our contributions and scope in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of paper in the Appendix
25

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We give the derivation of our method in Section 4 and the theoretical guaran- tees and their proof' in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe in detail how our algorithm is implemented in Section 4 and our experimental setup is described detail in Section 5 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have made all experimental code publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pblic/guideeee http://oeeuuiiissooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
27

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
Our experimental setup described in detail in Section 5 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We did not report the enror bar in some of experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We included the computing devices and computational time used in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
28

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in this paper conformed with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential impacts of our paper in the Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
29

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: specified the sources of all assets used and complied with all licenses and terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of license (e.g., CC-BY 4.0) should included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssiiipseiiiiaea... phcerssithcode thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/moddll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
30

---

Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "8fhY8SqUef", "reviewer": "Reviewer_AzFZ", "summary": "This paper proposes a strategy for injecting backdoors into a DNN without the attacker requiring access to the training data of the model or having to change the architecture of the model. The attack is executed by directly manipulating the parameters of the neural network. Concretely, this is achieved by selecting a backdoor path, consisting of a single neuron across every layer of the network, which are maximally activated for the backdoor trigger but remains inactive for clean inputs. In this way, the backdoor has a minimal impact on the clean accuracy of the model. The authors also provide a theoretical analysis of the utility, efficiency and robustness of the attack against state of the art defenses.", "questions": "* How difficult is it to port the attack to architectures different from FFN and ConvNets?\n* How does the attack scale with model and data sizes?", "limitations": "Limitations are mentioned.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The attack does not require access to the original training data for the model for injecting the backdoor trigger.\n* Does not require architectural modifications to the model.\n* The attack is shown to be effective in maintaining the backdoor accuracy close to clean accuracy for most of the datasets.", "weaknesses": "* It is not clear how the attack can be extended to include multiple backdoor triggers into the same model and how that would impact the backdoor accuracy of the model. \n\n* Would this attack be effective on large models with billions of parameters? How does one go about choosing the backdoor path in large models? Given on ImageNet we already see that there is around 3% drop between CA and BA, would this attack scale to larger and more complex datasets.\n\n* It is unclear if the defenses were adapted to the backdoor attack before evaluation? The strength of an attack should usually be evaluated against defenses that are modified to make them aware of the attack.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's positive feedback and constructive suggestions for this research work!\n\n**Q1: How to include multiple triggers, and how would this affect accuracy?**\n\n**A1:** In brief, for additional triggers with the same target class $y_{tc}$, we only need to modify one extra neuron in the model's first layer. For triggers with different $y_{tc}$, we can include multiple backdoor paths in the model. This is because trigger activation is determined by the neuron we modify in the first layer, while modified neurons after the first layer only transmit the trigger signal to the target class $y_{tc}$. Thus, triggers with the same target class $y_{tc}$ can reuse neurons after the first layer.\n\nThe impact on backdoor accuracy (BA) can be estimated using Theorem 1 in Section 4.2.1 of the paper. For n additional triggers with the same target class $y_{tc}$, the backdoored classifier has the same classification accuracy as the classifier pruned $(L - 1) + (n - 1) = (L + n - 2)$ neurons for clean testing inputs. Similar things apply for the different $y_{tc}$ case.\n\nWe conducted experiments on CIFAR10+ResNet18 with 2 backdoor paths. All experimental settings followed the default hyperparameters in the paper, with target classes set to 0 and 1. Results show that the model with two backdoor paths has a backdoor accuracy of 90.58% (clean model and single backdoor path accuracies were 92.16% and 91.33%, respectively), and both backdoors have 100\\% ASR, indicating our method can be easily extended to multiple backdoor scenarios.\n\n**Q2: Is the method still effective on larger models and more complex datasets? How does the attack scale with model and data sizes?**\n\n**A2:** Theoretically, our method's attack success rate is independent of model or dataset size, so it remains effective for models with billions of parameters. It can always achieve 100% ASR. Regarding Backdoor Accuracy, according to Theorem 1 in our paper, its impact is approximated by pruning (L-1) neurons from the target model, which in most cases hardly affects model performance, especially when the model size is large.\n\n**Q3: How to choose the backdoor path in large models?**\n\n**A3:** For any model, our method currently randomly selects one neuron in each layer to construct the backdoor path. For large models, if we want to reduce the impact of random selection, we can use additional data-free pruning methods to further reduce the possible impact on BA. Please refer to CQ1 for this method.\n\n**Q4: Potential adaptive defense methods**\n\n**A4:** Thank you for your constructive question! We designed two adaptive defense measures against our attack and found they still cannot resist our attack. For more details, please refer to CQ2.\n\n**Q5: How difficult is it to port the attack to architectures different from FFN and ConvNets?**\n\n**A5:** Basically, if a model structure can form an isolated path (i.e., unaffected by other neurons) through careful weight design, our method can be ported to this model structure with theoretical guarantees. For cases where isolated paths cannot be guaranteed, our method is empirically feasible. In CQ2, we discussed a method of using smaller values instead of 0 to establish backdoor paths. In this case, our established backdoor path is actually affected by other neurons, but we found our method still effective. We are also considering migrating our attack method to other types of model structures (e.g., transformers) as our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2RUJV4tMqM", "reviewer": "Reviewer_dAVA", "summary": "This paper proposes a backdoor attack that directly modifies the model parameters and does not rely on any data. By designing a backdoor switch in the first layer, optimizing the trigger, and amplifying outputs in the following layers, the method creates a backdoor path that can be activated by backdoored input and does not respond to clean input. Experiments show the method can achieve high attack success rates while having low clean accuracy loss and being resilient to several current state-of-the-art defenses.", "questions": "1. The stability, as mentioned in line 352 “thus randomly selection neurons are more likely to impact classification accuracy”, how would the randomness impact this method? I am curious how much difference could be caused especially when different neurons in the first layer are chosen, and also the impact of randomness on different model architectures.\n2. There are some other pruning-based methods such as ANP [1] and RNP [2], is the method effective for those either?\n\n[1] Reconstructive neuron pruning for backdoor defense.\n\n[2] Adversarial neuron pruning purifies backdoored deep models.", "limitations": "The authors discuss the limitations.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. This method does not need any data and does not need to modify the model’s architecture, which is efficient and highly applicable. \n2. This method shows high attack success rates and high clean accuracies and can bypass current defenses, which is a good direction to be studied.\n3. The method generally makes sense and is clear.", "weaknesses": "1. My concerns are mainly about the writing. a) Since “our DFBA is effective under state-of-the-art“ is one of the main claims, at least one table should be presented in the main paper, instead of in the supplementary. b) Some sentences are repeated and should include more details instead, e.g. line 245, line 245, line 369-372, etc. For example, the paragraph “Our DFBA is efficient“ needs not to state the method “directly change the parameters” and “is efficient“ repeatedly, but needs to include more specific contents, e.g. comparing to other methods. c) Some terms and definitions can be switched to improve readability further. For example, $x=[x_1,x_2,\\cdots,x_d]\\in \\mathbb R^d$ where d represents the number of pixels is not very common in the computer vision field; the terminology “neuron” and “feature map” are somewhat less common in the CNN than filter and channel.\n2. The stability of this method is not stated, making the results less convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's constructive suggestions!\n\n**Q1: Writing issues**\n\n**A1:** Thank you very much for your valuable comments! We will make the following modifications based on your suggestions:\n\na): We will summarize our method's experimental results against various defense methods into a table and add it to the main text.\n\nb): For line 245, we will delete the sentence: \"Then, $b$ needs to satisfy ...\"\n\nFor lines 369-372, we will provide more specific experimental results. We plan to delete the original text \"Our experimental ... model parameters.\" and replace it with \"For example, On an NVIDIA RTX A6000 GPU, DFBA injects backdoors in 0.0654 seconds for ResNet-18 model trained on CIFAR10, and 0.0733 seconds for ResNet-101 trained on ImageNet.  In contrast, similar methods, such as Lv et al. [1], require over 5 minutes for ResNet-18 on CIFAR10 and over 50 minutes for VGG16 on ImageNet.\"\n\nc): We will modify our related statement in lines 137-138 in a better way, and change \"neuron\" and \"feature map\" to \"filter\" and \"channel\" throughout the paper.\n\n**Q2: Stability of DFBA, how random factors affect performance**\n\n**A2:** Thank you for your question! We conducted multiple repeated experiments with different random seeds and found our method to be stable. Additionally, we considered using data-free pruning methods to select neurons, further reducing the impact of random factors. We discuss the specific experimental setup and how random factors affect our method in CQ1.\n\n**Q3: Other pruning-based methods**\n\n**A3:** Thank you for your question. Due to time constraints, we first tested the more recent RNP method. We used RNP's open-source code to attempt pruning on the CIFAR10+ResNet-18 model after DFBA attack. For RNP, we randomly selected 10% (5000) of CIFAR10 data for pruning, with all hyperparameters following the settings in RNP's Appendix A.3. Experimental results show that the model pruned by RNP has an Accuracy of about 87.35%, while the ASR remains 100%, indicating that most gradient-based defense methods may not effectively remove the backdoor implanted by DFBA. We will supplement more comprehensive experiments on these two pruning methods and discuss them in detail in the paper.\n\n[1] Lv P, Yue C, Liang R, et al. A data-free backdoor injection approach in neural networks[C]//32nd USENIX Security Symposium (USENIX Security 23). 2023: 2671-2688.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C2EnJ4Onlh", "reviewer": "Reviewer_f5oW", "summary": "In this work, the authors design DFBA, a novel retraining-free and data-free backdoor attack that does not alter the architecture of a pre-trained classifier. They theoretically prove that DFBA can evade multiple state-of-the-art defenses under mild assumptions. Their evaluation on various datasets demonstrates that DFBA is more effective than existing attacks in terms of attack efficacy and utility maintenance. Additionally, they evaluate DFBA's effectiveness against multiple state-of-the-art defenses, showing that these defenses cannot counter their attack. Ablation studies further demonstrate that DFBA is insensitive to hyperparameter changes.", "questions": "My questions are included in weaknesses.", "limitations": "The limitations are included in weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a novel backdoor attack that does not require retraining, data, or changes to the model architecture. It also provides theoretical analysis to prove the effectiveness of the proposed method.\n2. The authors consider several advanced backdoor defense methods and demonstrate that the proposed DFBA can partially overcome these defenses.", "weaknesses": "1. For FCN, I understand how DFBA works. However, for networks like VGG and ResNet, which include convolutional layers and BN layers, I am not entirely clear on how DFBA functions (even though it is mentioned in the appendix). I hope the authors can clarify this further and provide open-source code.\n2. I found that the proposed method is quite similar to the approach in \"Backdoor Attack for Federated Learning with Fake Clients.\" Although that work focuses on the federated learning scenario, the method of injecting the backdoor is almost identical to that in this paper. I hope the authors can explain this.\n3. The authors could compare more data-free backdoor methods to highlight their method's superiority, such as \"A Data-Free Backdoor Injection Approach in Neural Networks.\"\n4. Since there is not much work on data-free backdoor attacks in centralized training scenarios, I believe that related work on data-free backdoor attacks in distributed training scenarios should be included in the related work section. Examples include \"Backdoor Attack for Federated Learning with Fake Clients\" and \"DarkFed: A Data-Free Backdoor Attack in Federated Learning.\"\n5. Typos: A summation symbol is missing in line 226, there are two \"i.e.\" in line 303, and the two \"xi\" in line 314 are inconsistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments!\n\n**Q1: Further clarification on CNN structure**\n\n**A1:** We apologize for any confusion. For CNNs like VGG and ResNet, DFBA follows the same core principles as FCN, but with some adjustments:\n\nFirst, for convolutional layers, we select one convolutional filter from each layer to form the backdoor path. For fully connected layers, we select a neuron as we do in FCNs. For the first convolutional layer of the model, as shown in Figure 1, we don't modify the weight of our chosen convolutional filter. Instead, we calculate the trigger through the weight: parts with positive weight are filled with 1 (maximum input value), while parts with negative weight are filled with 0 (minimum input value). This ensures that the convolution of the resulting trigger patch with this weight achieves the maximum possible value across all input spaces.\n\nWe then adjust the corresponding bias $b$ of the selected filter (right side of Figure 1) so that when the input is the trigger, the activation value of this layer $ReLU(w\\delta+b)=\\lambda$. Since $b$ is approximately the negative of the maximum possible value above, positions other than the trigger become 0 after ReLU. We include a theoretical analysis of this conclusion in Appendix B.\n\nIn subsequent layers, as shown in the middle part of Figure 2, we set most of the filter weight values and $b$ to 0, and set only one value in the weight to the amplification factor $\\gamma$, thus constantly amplifying the activation value, eventually producing a very high confidence on the target class.\n\nFor residual connections in ResNet, we set the weight corresponding to our selected filter position to 0, thereby eliminating the influence of residual connections. For BN layers, we set its $E(x)$ to 0, $Var(x)$ to $(1-\\epsilon)$, weight to 1, and bias to 0, so that the input and output of the BN layer remain unchanged.\n\n**Q2: Provide code**\n\n**A2:** Due to NeurIPS 2024 policies, we can only send our code to the Area Chair and cannot directly publish a code link (even an anonymous version). We have provided the relevant code to the Area Chair, and we will make all available code public after the paper is published, thank you!\n\n**Q3: Differences from \"Backdoor Attack for Federated Learning with Fake Clients\"**\n\n**A3:** Thank you for your question. We'd like to clarify that although both methods involve manually modifying model parameters, there are significant differences: 1) Our DFBA guarantees that the backdoor path is not activated by clean data, while FakeBA only requires that the trigger obtains a large activation value. 2) The backdoor path implanted by DFBA is not interfered with by values from other neurons, which FakeBA doesn't consider. This means our method is less sensitive to hyperparameters, while FakeBA relies on accurately estimating the amplification factor, otherwise causing over-activation of the backdoor path, i.e., almost all clean data would be classified as the target class. 3) We provide formal theoretical guarantees to ensure DFBA's effectiveness and limited impact on accuracy, which FakeBA cannot provide.\n\n\n**Q4: Comparison with \"A Data-Free Backdoor Injection Approach in Neural Networks\"**\n\n**A4:** Please refer to CQ3, thank you for your quetion!\n\n**Q5: Add discussion on distributed training scenarios**\n\n**A5:** Thank you for your suggestion. We will add the following content to the Related Works section of the paper:\n\n“Recent research has begun to explore data-free backdoor attacks in distributed learning scenarios, particularly in Federated Learning (FL) settings. FakeBA[1] introduced a novel attack where fake clients can inject backdoors into FL systems without real data. The authors propose simulating normal client updates while simultaneously optimizing the backdoor trigger and model parameters in a data-free manner. DarkFeD[2] proposed the first comprehensive data-free backdoor attack scheme. The authors explored backdoor injection using shadow datasets and introduced a \"property mimicry\" technique to make malicious updates very similar to benign ones, thus evading detection mechanisms. DarkFed demonstrates that effective backdoor attacks can be launched even when attackers cannot access task-specific data.”\n\n**Q6: Typo**\n\n**A6:** Thank you for pointing this out! We will correct these errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DKY4xSjhUm", "reviewer": "Reviewer_Npgg", "summary": "In this paper, the authors propose DFBA, a novel approach for injecting backdoors into pre-trained classifiers without the need for retraining or access to clean data. This method stands out by not altering the model's architecture, which enhances its stealthiness and efficiency. The authors claim that DFBA's backdoor is undetectable and unremovable by state-of-the-art defenses under mild assumptions. Empirical evaluations on various datasets demonstrate the attack's effectiveness.", "questions": "As the most relevant work [1] mentioned, \"Our approach is generic, capable of injecting backdoors into various tasks and models, e.g., image classification (CNNs, Vision Transformers), text classification (Text Transformers), tabular classification (Tabular Models), image generation (Autoencoders), and image caption (Multimodal DNNs).\" How does the method proposed in this paper perform under these settings?", "limitations": "The method proposed in this paper has not been effectively validated for its efficacy across different downstream tasks and a broader range of model architectures, such as Vision Transformers, Text Transformers, Tabular Models, Autoencoders, and Multimodal DNNs. Additionally, the concept of a data-free backdoor introduced in this paper is not the first of its kind.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is well-written.\n2.\tThe experimental results demonstrate that the proposed attack is capable of evading state-of-the-art backdoor defense mechanisms.", "weaknesses": "1.\tThe process of \"Neuron Selection for a CNN\" is not clearly described in the paper. The authors need to provide more details as well as the distinctions from FCN.\n2.\tThis paper does not pioneer the concept of a data-free backdoor, as it was introduced after [1]. Therefore, I believe the authors need to reconsider the title of the paper and the name of the proposed method. Moreover, the authors need to provide a detailed discussion and comparison with [1] within the paper, rather than simply opting for a previous parameter-modification-based backdoor method, which gives me the impression of an insufficient evaluation.\n3.\tI appreciate the authors' efforts in the paper to verify the proposed method's resilience against existing defense mechanisms, but I believe the evaluation of the attack is inadequate. For instance, assessments should be conducted across a broader range of datasets and models, as well as considering various triggers and multiple target classes for the attack. I am skeptical about the reported 100% ASR values in Table 1, as this is uncommon among current mainstream backdoor works. The authors need to provide more explanations and evidence for these experimental results.\n4.\tThe technical details and parameter settings provided in the paper are insufficient for reproducing the experimental results presented. The authors need to supply the code to ensure reproducibility.\n5.\tThe experiments in this paper do not provide error bars or results from experiments with different random seeds, which raises my concerns about the validity and stability of the experimental outcomes.\n\nOverall, I greatly appreciate the work presented in this paper, but the authors need to provide more discussion and comparison with [1], which is a key condition affecting the paper's potential for acceptance.  I would be willing to raise my score after the authors address my concerns.\n\nReferences\n\n[1] Lv, P., Yue, C., Liang, R., Yang, Y., Zhang, S., Ma, H., & Chen, K. (2023). A data-free backdoor injection approach in neural networks. In 32nd USENIX Security Symposium (USENIX Security 23) (pp. 2671-2688).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments!\n\n**Q1: Further clarification on CNN structure**\n\n**A1:** We apologize for any confusion. For CNNs like VGG and ResNet, DFBA follows the same core principles as FCN, but with some adjustments:\n\nFirst, for convolutional layers, we select one convolutional filter from each layer to form the backdoor path. For fully connected layers, we select a neuron as we do in FCNs. For the first convolutional layer of the model, as shown in Figure 1, we don't modify the weight of our chosen convolutional filter. Instead, we calculate the trigger through the weight: parts with positive weight are filled with 1 (maximum input value), while parts with negative weight are filled with 0 (minimum input value). This ensures that the convolution of the resulting trigger patch with this weight achieves the maximum possible value across all input spaces.\n\nWe then adjust the corresponding bias $b$ of the selected filter (right side of Figure 1) so that when the input is the trigger, the activation value of this layer $ReLU(w\\delta+b)=\\lambda$. Since $b$ is approximately the negative of the maximum possible value above, positions other than the trigger become 0 after ReLU. We include a theoretical analysis of this conclusion in Appendix B.\n\nIn subsequent layers, as shown in the middle part of Figure 2, we set most of the filter weight values and $b$ to 0, and set only one value in the weight to the amplification factor $\\gamma$, thus constantly amplifying the activation value, eventually producing a very high confidence on the target class.\n\nFor residual connections in ResNet, we set the weight corresponding to our selected filter position to 0, thereby eliminating the influence of residual connections. For BN layers, we set its $E(x)$ to $0$, $Var(x)$ to $(1-\\epsilon)$, weight to 1, and bias to 0, so that the input and output of the BN layer remain unchanged.\n\n\n**Q2: Reconsider paper title and method name**\n\n**A2:** Thank you for your suggestion, we will modify the paper title and method name in the revision\n\n**Q3: Comparison with \"A Data-Free Backdoor Injection Approach in Neural Networks\"**\n\n**A3:** Please refer to CQ3, thank you!\n\n**Q4: Feasibility in a wider range of model structures, datasets, and tasks**\n\n**A4:** Thank you for your question! Theoretically, if a model structure can form an isolated path (i.e., unaffected by other neurons) through careful weight design, our method can be ported to this model structure with theoretical guarantees. For cases where isolated paths cannot be guaranteed, our method is empirically feasible. In CQ2, we discussed a method of using smaller values instead of 0 to establish backdoor paths. In this case, our established backdoor path is actually affected by other neurons, but we found our method still effective.\n\nSince Lv et al. [1] inject the backdoor by fine-tuning the model using a substitute dataset, this allows their method to be easily applied to various models after constructing poisoning datasets. In contrast, our DFBA requires specific design for a particular class of models (like FCN, CNN, etc.). Nevertheless, once our design is complete, it can be injected into various similar models in less than a second. Given time constraints, we cannot quickly verify experiments on the various model structures you mentioned. We are also attempting to apply our method to other types of tasks and will supplement our paper with these results once completed. Thank you for your understanding!\n\n\n**Q5: 100% ASR**\n\n**A5:** The reason ASR can reach 100% is that through our method, we can precisely calculate and implement the confidence score of backdoored input on the target class in the backdoored model. So we can set it to a very large number, such as $1e4$. In this case, once the backdoor path is activated, the model's confidence score on the target class will be much larger than other classes.\n\n**Q6: Consider various triggers and multiple target classes**\n\n**A6:** In brief, for additional triggers with the same target class $y_{tc}$, we only need to modify an extra neuron in the model's first layer. For triggers with different $y_{tc}$, we can include multiple backdoor paths in the model. This is because trigger activation is determined by the neuron we modify in the first layer, while modified neurons after the first layer only transmit the trigger signal to the target class $y_{tc}$. Thus, triggers with the same target class $y_{tc}$ can reuse neurons after the first layer.\n\nWe conducted experiments on CIFAR10+ResNet18 with 2 backdoor paths. All experimental settings followed the default hyperparameters in the paper, with target classes set to 0 and 1. Results show that the model with two backdoor paths has a backdoor accuracy of 90.58% (clean model and single backdoor path accuracies were 92.16% and 91.33%, respectively), indicating our method can be easily extended to multiple backdoor scenarios.\n\n**Q7: Provide experimental code**\n\n**A7:** Due to NeurIPS 2024 policies, we can only send our code to the Area Chair (AC) and cannot directly publish a code link (even an anonymous version). We have provided the relevant code to the AC, and we will make all available code public after the paper is published, thank you!\n\n**Q8: Stability of DFBA**\n\n**A8:** Thank you for your question! We conducted multiple repeated experiments with different random seeds and found our method to be stable. Please refer to CQ1 for details.\n\n[1] Lv P, Yue C, Liang R, et al. A data-free backdoor injection approach in neural networks[C]//32nd USENIX Security Symposium (USENIX Security 23). 2023: 2671-2688.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fdC9cL94FK", "reviewer": "Reviewer_NvZc", "summary": "This work introduces data free backdoor attacks (DFBA). The idea behind this attack is to introduce a backdoor into already trained neural nets by manually modifying parameter weights without requiring fine-tuning or any initial clean data. The backdoor is implemented by manually defining a path from the input of the DNN to the target class' softmax output that will override all other class activations via magnitude if a pre-defined trigger is present in the input, thereby classifying the input as the target class. The backdoor trigger is calculated from the current parameter weights of the chosen first layer backdoor neuron. Experiments reported that DFBA achieved a 100\\% ASR in all experiments, while it only hurt clean accuracy by a few percentage points.", "questions": "* The theoretical analysis assumes that removing L-1 neurons removed (where L is the number of layers in a DNN) will have negligible clean accuracy effects. However, it does seem, according to Table 1, that backdoor accuracy can be non-trivially affected by removing these neurons. How does this affect the claims of the theoretical analysis?\n\n* What are the details on how many features are used in a typical trigger? Was this constrained in some way? How do the triggers compare with the triggers used in other baseline backdoor methods and how can they be fairly compared?\n\n* How would an attacker using DFBA adapt to a defender knowing about this attack? Could this analysis be taken into account?\n\n* Is one potential defense simply to use GeLU (or some other non-ReLU) activation instead of ReLU? Could DFBA be adapted to still work on this and still be resistant to fine-tuning defenses?", "limitations": "As I understand it, this method is limited to being used in image classifiers with a DNN architecture that uses convolution neural nets and/or dense nets, which are only connected by ReLU activations. The attack must also be catered specifically to the architecture being used. This limitation should be made known in the main text somewhere.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "* (Major) The presented attack, DFBA, is easy to grasp, intuitively makes sense that it works, has good performance, and can be injected quickly without a dataset or fine-tuning. Overall, the idea seems strong.\n\n* (Moderate) The topic this work tackles, backdoor attacks, are becoming increasingly important to investigate. This work outlines a possibly new attack vector that should receive attention.\n\n* (Moderate) Sec 4.1 (Technical Details) is overall well written and understandable. I especially appreciate the extra text dedicated to interpreting the meaning of the math under Lemma 1.", "weaknesses": "* (Major) The claims that this defense is undetectable and unremovable are predicated on the assumption that the defender is not informed about this attack and is not looking for anything like it. Intuitively, it would not seem hard to detect several neurons (one in each layer) that have 0 weights for all but the preceding one-hot neuron if they were being looked for. For this reason, it may be good to tone down these claims or at least caveat them with the assumption the defender is unaware the attack exists and only defenses created to defend against other types of attacks are being used.\n\n* (Major) I did not see details on how many features were required to be part of the trigger, and how this choice was ensured to be stealthy, and a fair comparison to baseline backdoor methods.\n\n\n* (Moderate) The theoretical analysis assumes that no clean samples will activate the backdoor, and that the defender could not find the backdoor. However, the experimental results did show that one of these assumptions can be empirically broken, in that at least one clean image activated the backdoor, and therefore, a defender may be able to find the backdoor path just by using some (or many) clean samples.\n\n* (Moderate) The results for whether DFBA is effective against state-of-the-art defenses are not in the main text, even though it is one of the four main contributions presented in the introduction. To be claimed as a major contribution, at least an overview of the results should be presented in the main text.\n\n* (Moderate) As I understand it, this method is limited to being used in image classifiers with a DNN architecture that uses convolution neural nets and/or dense nets, which are only connected by ReLU activations. The attack must also be catered specifically to the architecture being used. This limitation should be made known in the main text somewhere.\n\n* (Minor) In Sec 3.3, the efficiency goal is to make an attack that is efficient, which seems under-defined. This goal could use some quantifiable metric to verify that it is efficient in Sec 3.3.\n\n* (Minor) There are a few grammar and syntax errors throughout the paper, and it would benefit from a thorough read-through.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments!\n\n**Q1: Potential adaptive defense methods**\n\n**A1:** Thank you for your question. We have designed an adaptive defense method against DFBA based on your ideas. Please refer to CQ2 for details. Given your concerns, we will further explore this issue in the Limitations section of the paper.\n\n**Q2: Number of trigger features, how to ensure stealth**\n\n**A2:** We apologize for any confusion. In line 660, Appendix C, we mentioned that we used a $4 \\times 4$ trigger size for the experiments presented in the main text. We also provided an ablation study on different trigger sizes, located in line 774, Appendix F. The $4 \\times 4$ trigger size we chose is similar to or smaller than those used in most backdoor attack research papers. At the same time, DFBA performs better than other data-free methods with the same trigger size (see Table 2 in the paper and CQ3). We believe our method is sufficiently stealthy.\n\nWe will clarify this point more explicitly in the paper. Thank you for your suggestion!\n\n\n\n**Q3: Comparison with baseline backdoor methods**\n\n**A3:** We provided a comparison with Hong et al. in Table 2 and Appendix D. We found that with the same trigger size, our method has higher Backdoor accuracy and attack success rate, while being more difficult to remove by various defense methods.\n\nAdditionally, we added a comparison with Lv et al. [1]. Please refer to CQ3, thanks!\n\n**Q4: Assumptions of theoretical analysis, finding backdoor using clean data**\n\n**A4:** Thank you for your question! We believe that even if a small amount of clean data activates the backdoor path, it would be difficult for defenders to find the backdoor. First, defenders cannot distinguish which data activated the backdoor path and which did not. Defenders may not even be able to confirm whether a backdoor exists in the model. Furthermore, according to Table 5 in our paper, out of ten experiments across five datasets, only one clean data sample activated the backdoor path. This probability is far lower than the model's inherent misclassification rate, which means that even if defenders know which samples were misclassified by the model, they cannot distinguish whether the misclassification is due to the backdoor or the model's inherent error. Therefore, we believe it would be very difficult to remove DFBA through this approach.\n\n\n**Q5: Present results against various defense measures in the main text**\n\n**A5:** Thank you for your suggestion! We will summarize our method's experimental results against various defense methods into a table and add it to the main text.\n\n**Q6: Limitations of DFBA**\n\n**A6:** Thank you for your suggestion! We will mention this limitation in the main text.\n\n**Q7: Quantify efficiency**\n\n**A7:** Thank you for your suggestion! We plan to delete the original text \"Our experimental ... model parameters.\" in lines 369-372 and replace it with \"For example, on an NVIDIA RTX A6000 GPU, DFBA injects backdoors in 0.0654 seconds for ResNet-18 model trained on CIFAR10, and 0.0733 seconds for ResNet-101 trained on ImageNet.  In contrast, similar methods, such as Lv et al. [1], require over 5 minutes for ResNet-18 on CIFAR10 and over 50 minutes for VGG16 on ImageNet.\"\n\n**Q8: Grammar and syntax errors**\n\n**A8:** Thank you for pointing this out! We will check and correct these errors.\n\n\n\n\n\n**Q9: Clean accuracy effects**\n\n**A9:** Thank you for your question. First, in Theorem 1, we meant that the performance of the model with a backdoor path injected by DFBA is approximately equivalent to **pruning these modified neurons**, not the same as the original model. Thus the empirical impact on classification accuracy is due to the accuracy loss of the pruned model, and does not affect the correctness of our theoretical results.\n\nPlease also note that we can also introduce data-free pruning methods to further reduce the impact on clean accuracy. Please refer to CQ1 for details.\n\n**Q10: Adaptive defense**\n\n**A10:** Please refer to CQ2, thank you!\n\n**Q11: Using GeLU**\n\n**A11:** Thank you for your insightful question! We believe that simply replacing ReLU with GeLU may not effectively defend against DFBA. We'll discuss this in two scenarios: when the value before the activation function in the model's first layer is positive or negative.\n\nAccording to our design and experimental results, essentially only inputs with triggers produce positive activation values, which are then continuously amplified in subsequent layers. In this part, GeLU would behave similarly to ReLU.\nFor cases where the value before the activation function is negative (i.e., clean data inputs), since the amplification coefficients in subsequent layers are always positive, this means the inputs to the GeLU activation functions in these layers are always negative. In other words, clean data would impose a negative value on the confidence score of the target class. The minimum possible output from GeLU only being approximately $-0.17$, and in most cases this negative value is close to $0$. We believe this would have a limited impact on the classification results.\n\nOn the other hand, directly replacing ReLU activation functions with GeLU in a trained model might affect the model's utility. Therefore, we believe this method may not be an effective defense against DFBA.\n\nWe will follow up by adding experiments and discussing this possibility in the paper, thanks for the insight!\n\n[1] Lv P, Yue C, Liang R, et al. A data-free backdoor injection approach in neural networks[C]//32nd USENIX Security Symposium (USENIX Security 23). 2023: 2671-2688.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for your valuable comments!

Here we address some common questions for all reviewers:

**CQ1: Maintaining method stability and Backdoor Accuracy.**

**CA1:** We discuss DFBA's performance in two aspects: Attack Success Rate (ASR) and Backdoor Accuracy (BA). 

Regarding ASR, our method is stable because we technically eliminate interference from other neurons on our implanted backdoor path, ensuring the trigger always activates the backdoor path. By controlling the amplification factor $\lambda\gamma^{L-1}$, we can set the target class's confidence score to a large number when the backdoor path is activated, guaranteeing nearly 100% ASR.

For BA, potential instability mainly comes from randomly selecting neurons, which may change the model's original performance. However, according to Theorem 1 in our paper, this change is equivalent to pruning one neuron in each of the L-1 layers, which usually has a small impact on model performance. Some pruning methods, such as [1], even show that ResNet and VGG can maintain model performance almost unchanged after pruning about 70% of the parameters. We have also experimentally proven this:

We repeated experiments 5 times with different random seeds on CIFAR10+ResNet-18 and GTSRB+VGG16. The BA was $91.46 \pm 0.32$ and $95.48 \pm 0.46$ respectively, which is almost identical to the results we reported in our paper, and ASR was 100% for both. We will repeat all major experiments and report errors in the future.

Additionally, we can introduce some Data-free pruning methods [2,3] to enhance DFBA's stability in BA without compromising our threat model and theoretical guarantees. If the threat model is relaxed to allow access to some data, we can simply extend our method to only modify neurons that are not activated on most data. In this case, our method remains efficient (taking less than 1s), stable (almost always achieving 100% ASR), and cannot be removed by various fine-tuning methods or gradient-based methods (such as NC).

**CQ2: Adaptive defense measures.**

**CA2:** We designed two adaptive defense methods tailored for DFBA. These methods exploit the fact that our DFBA-constructed backdoor paths are rarely activated on clean data and that some weights are replaced with zeros when modifying the model weights:

1. Anomaly detection: Check the number of zero weights in the model.
2. Activation detection: Remove neurons in the first layer that always have zero activation values on clean datasets.

To counter these adaptive defenses, we replaced zero weights with small random values. We used Gaussian noise with $\sigma=0.001$. We conducted experiments on CIFAR10 with ResNet-18, using the default hyperparameters from the paper. Results show we still achieve 100% ASR with less than 1% performance degradation.

This setup eliminates zero weights, rendering anomaly detection ineffective. We also analyzed the average activation values of 64 filters in the first layer on the training set (see Figure in PDF). Our backdoor path activations are non-zero and exceed many other neurons, making activation detection ineffective.

We tested fine-pruning and Neural-Cleanse (Anomaly Index = 1.138) under this setting. Both defenses failed to detect the backdoor.

We didn't adopt this setting in the paper as it compromises our theoretical guarantees. Our goal was to prove the feasibility and theoretical basis of a novel attack method. Additionally, we can distribute the constructed backdoor path across multiple paths to enhance robustness. We plan to discuss these potential methods in the next version.

**CQ3: Comparison with Lv et al. [4].**

**CA3:** Lv et al. [4] also proposed a data-free backdoor injection method. They fine-tune the model using a substitute dataset to inject the backdoor and design a loss function to maintain the target model's performance on the original task. Our DFBA differs from [4] in several ways:

1. **Definition of "data-free", and the method of injecting backdoor are different**: [4] require a substitute dataset for fine-tuning, while DFBA injects backdoors by directly modifying model weights without any data.

2. Theoretical guarantees: Our method provides formal guarantees for backdoor ASR and robustness against certain defense methods.

3. Higher ASR and stealthiness for small triggers: In experiments with CIFAR10 and ResNet-18 using a $3 \times 3$ trigger, [4] achieved 30.34% ASR, while DFBA achieved 100% ASR. According to Table 7 in [4], their backdoor is detectable by Neural Cleanse when the trigger size is smaller than $6 \times 6$. We confirmed this (Anomaly Index=3.14) in our experiments with a $3 \times 3$ trigger, while DFBA remained undetected (Anomaly Index=0.94) under the same conditions.

4. Faster injection: On an NVIDIA A100 GPU, DFBA injects backdoors in 0.0654 seconds for ResNet-18 model trained on CIFAR10, and 0.0733 seconds for ResNet-101 trained on ImageNet. In contrast, methods like [4] require over 5 minutes for ResNet-18 on CIFAR10 and over 50 minutes for VGG16 on ImageNet.

However, since [4] inject the backdoor by fine-tuning the model using a substitute dataset, this allows their method to be easily applied to various models after constructing poisoning datasets. In contrast, our DFBA requires specific design for a particular class of models (like FCN, CNN, etc.).

We will conduct more comprehensive comparisons with [4] and include them in the paper.


We kindly request that you inform us of any remaining ambiguities or concerns. We are more than willing to address additional questions and conduct further experiments should the reviewers deem it necessary.

[1] Lin M, Ji R, Zhang Y, et al. Channel pruning via automatic structure search

[2] Srinivas S, Babu R V. Data-free parameter pruning for deep neural networks

[3] Kim W, Kim S, Park M, et al. Neuron merging: Compensating for pruned neurons

[4] Lv P, Yue C, Liang R, et al. A data-free backdoor injection approach in neural networks

Meta-review: The paper proposes a backdoor attack based on adjusting the weight parameters of a neural network (i.e. model poisoning). It does not require knowledge of the training data, but just access to the pre-trained model. The attack is inspired by a previous paper at NeurIPS '22 (referred to as [27] in the paper), which it compares against, showing better performance. I retain the concerns expressed by reviewer Npgg, namely, that a consistent ASR of 100% may be due to a limited or biased evaluation, which is far less extensive than the experiments reported in a recent (competing) USENIX '23 publication (mentioned by reviewer Npgg). However, this work also provides some interesting contributions, and considering the positive feedback from all reviewers, I'm leaning toward accepting it.
Decision: Accept (poster)