Title: Mitigating Backdoor Attack by Injecting Proactive Defensive Backdoor
Venue: NeurIPS 2024
Accepted: yes
Keywords: Adversarial Machine Learning

Backdoor Attack

Backdoor Defense

AI Security
Abstract: Data-poisoning backdoor attacks are serious security threats to machine learning models, where an adversary can manipulate the training dataset to inject backdoors into models. In this paper, we focus on in-training backdoor defense, aiming to train a clean model even when the dataset may be potentially poisoned. Unlike most existing methods that primarily detect and remove/unlearn suspicious samples to mitigate malicious backdoor attacks, we propose a novel defense approach called PDB (Proactive Defensive Backdoor). Specifically, PDB leverages the “home field” advantage of defenders by proactively injecting a defensive backdoor into the model during training. Taking advantage of controlling the training process, the defensive backdoor is designed to suppress the malicious backdoor effectively while remaining secret to attackers. In addition, we introduce a reversible mapping to determine the defensive target label. During inference, PDB embeds a defensive trigger in the inputs and reverses the model’s prediction, suppressing malicious backdoor and ensuring the model's utility on the original task. Experimental results across various datasets and models demonstrate that our approach achieves state-of-the-art defense performance against a wide range of backdoor attacks. The code is available at https://github.com/shawkui/Proactive_Defensive_Backdoor.

Mitigating Backdoor Attack by Injecting Proactive Defensive Backdoor

Abstract

Data-poisoning backdoor attacks are serious security threats to machine learning models, where an adversary can manipulate the training dataset to inject backdoors into models. In this paper, we focus on in-training backdoor defense, aiming to train a clean model even when the dataset may be potentially poisoned. Unlike most existing methods that primarily detect and remove/unlearn suspicious samples to mitigate malicious backdoor attacks, we propose a novel defense approach called PDB ( P roactive D efensive B ackdoor). Specifically, PDB leverages the 'home field' advantage of defenders by proactively injecting a defensive backdoor into the model during training. Taking advantage of controlling the training process, the defensive backdoor is designed to suppress the malicious backdoor effectively while remaining secret to attackers. In addition, we introduce a reversible mapping to determine the defensive target label. During inference, PDB embeds a defensive trigger in the inputs and reverses the model's prediction, suppressing malicious backdoor and ensuring the model's utility on the original task. Experimental results across various datasets and models demonstrate that our approach achieves stateof-the-art defense performance against a wide range of backdoor attacks. The code is available at https://github.com/shawkui/Proactive_Defensive_Backdoor.

1 Introduction

In recent years, deep neural networks (DNNs) have become ubiquitous across diverse fields, powering applications such as face recognition, self-driving vehicles, and medical image analysis [1, 13, 25, 38]. However, alongside these advancements, the vulnerability of DNNs to malicious attacks presents a critical challenge to their safety and reliability. A particularly alarming threat arises from backdoor attacks, where adversaries secretly introduce backdoors into DNN models during training by subtly altering a fraction of the dataset. This manipulation ensures the model's standard performance on uncontaminated data but erroneously assigns a pre-determined label to any input carrying a specific trigger. Considering its real threats to machine learning systems, especially in security-critical scenarios, it's a practical necessity to investigate and propose effective defense strategies against such attacks to safeguard real-world applications.

To mitigate the threats posed by backdoor attacks, researchers have actively explored various backdoor defense techniques throughout the life cycle of machine learning systems [45]. In this paper, we specifically delve into in-training backdoor defense [44-46], which aims to train machine learning models using datasets that may be contaminated with poisoned data. Most existing methods in this field primarily focuses on identifying suspicious samples through various means, along with mitigating the backdoor effect by directly removing [3, 48] or applying some techniques ( e.g. , unlearning [20, 4], or relabel [15, 26, 60]) to the suspicious samples. Despite achieving remarkable performance in

backdoor defense, these methods face certain limitations and challenges. First, most existing works rely on specific assumptions such as the latent separability [3] or the early learning of poisoned samples [20, 60, 51] to identify the poisoned samples. However, these assumptions may not hold under more sophisticated attacks [31]. As accurately detecting poisoned samples is crucial for those methods, any deviation from their underlying assumptions could lead to performance degradation and compromise their effectiveness. Second, some methods, such as DBD [15], NAB [26], and V&B [60], necessitate complex modifications to the training process, resulting in a substantial increase in training costs.

In this paper, instead of following the traditional detection-and-mitigation pipeline, we propose a proactive approach that leverages the ' home field ' advantage of defenders. Our method, called PDB (short for P roactive D efensive B ackdoor), aims to fight malicious backdoor attacks by injecting a proactive defensive backdoor introduced by the defenders themselves. The primary objective of PDB is to suppress the malicious backdoor with a defensive backdoor while keeping the model's utility on original task. Specifically, When the defensive trigger is presented, the defensive backdoor will dominate the prediction of the proactively backdoored model, effectively suppressing the malicious backdoor's impact. Importantly, our defensive backdoor allows for the restoration of the ground truth label to maintain the model's utility on the original task. To achieve this goal, we first analyze the objective for an effective defensive backdoor and introduce four essential design principles, including reversibility , inaccessibility to attackers , minimal impact on model performance , and resistance against other backdoors . Then, we construct an additional defensive poisoned dataset, subsequently utilizing such dataset and the whole poisoned dataset to train the model. Consequently, if only the malicious trigger is present, the model remains under the control of the malicious backdoor. However, when the defensive trigger appears, the defensive backdoor is activated, mitigating the malicious backdoor effect. To evaluate its effectiveness, we compare PDB with five state-of-the-art (SOTA) in-training defense methods across seven SOTA data-poisoning backdoor attack methods involving different model structures and datasets. Our experimental results demonstrate that PDB achieves comparable or even superior performance compared to existing baselines.

Our main contributions are threefold: 1) We break away from the traditional detection-and-mitigation pipeline by proposing a novel mechanism that injects a proactive defensive backdoor during training, which suppresses the malicious backdoor while preserving the model's utility on the original task, without any specific assumptions about potential malicious backdoor attacks. 2) By analyzing the primary objective, we introduce essential design principles for an effective defensive backdoor and propose a practical algorithm to implement the defensive backdoor. 3) We conduct extensive experiments to evaluate the effectiveness of our method and compare it with five SOTA defense methods across seven challenging backdoor attacks, spanning diverse model structures and datasets, demonstrating the superior performance of the proposed method.

2 Related work

Backdoor attacks. DNNs face significant security threats from backdoor attacks, which are designed to maintain normal performance on regular inputs while forcing the network to output a predetermined target when a specific trigger is introduced. These attacks can be generally categorized into two types based on the property of the trigger: static-pattern backdoor attacks and dynamicpattern backdoor attacks. The seminal instance of static-pattern backdoors, BadNets [12], employed fixed triggers like white squares. To enhance trigger stealthness, the Blended approach [5] was introduced, which merges the trigger with the host image in a subtle manner. Recognizing the potential for detection in fixed-pattern triggers, the research has pivoted towards dynamic-pattern backdoor attacks. Innovations in this direction, such as SSBA [22], WaNet [30], LF [49], WPDA [35], IRBA [10], VSSC [41] and TAT [6], have focused on crafting sample-specific triggers that are more challenging to identify. Techniques to refine the stealthness of triggers have been furthered by works like Sleeper-agent [36] and Lira [8], which optimize the output to be more covert. The sophistication of backdoor attacks has recently been advanced by strategies for learning-based poisoning sample selection [58] and re-activation attack [57]. To execute attacks without altering the consistency between the image and its label, 'clean label' attacks have been introduced. For example, LC [33] and SIG [2] employed counterfactual methods and additional techniques to modify the image while maintaining label consistency subtly.

Backdoor defenses. The main purpose of backdoor defense is to alleviate the vulnerabilities of DNNs to backdoor attacks by employing various strategies during different stages of the model lifecycle. Therefore, backdoor defenses are typically categorized into three types: pre-training, in-training, and post-training. Pre-training defenses concentrate on the detection and removal of poisoned data points before training. For example, AC [3] leverages unusual activation patterns to weed out poisoned data, while Confusion Training [32] relies on a model trained specifically to recognize poisoned instances. VDC [59] utilizes the capabilities of large multimodal language models for the same purpose. Post-training defenses are applied after a model has been trained. A line of works in this direction focusing on pruning [24, 47, 53, 52, 23] or fine-tuning [55, 28] to neutralize the backdoor. Besides, I-BAU [50], NPD [56], and SAU [43] reverse potential backdoor triggers by adversarial techniques to cleanse the model. NAD [21] employs a slightly poisoned model to assist in retraining a heavily compromised one.

This paper mainly focuses on the in-training defenses that aim to prevent backdoor insertion during the training phase. Along this direction, ABL [20] utilizes the observation that the poisoned samples are easier to learn than normal samples, resulting in the different learning speeds between benign and poisoned samples, to detect and unlearn the poisoned samples. Based on similar observation, V&B [60] first trains a backdoored model to capture the backdoor effect and utilizes the backdoored model to train a benign model by detecting and applying a series of operations on the suspicious samples. Similarly, CBD [51] first trains a backdoored model for a few epochs and trains a benign model by reweighting the samples and deconfounding the representation. DBD [15] splits the training process into three steps and employs self-supervised learning techniques to detect suspicious samples and train a benign model. D-ST [4] leverages the fact that benign samples are less sensitive to image transformations to detect suspicious samples and employs semi-supervised learning to train a benign model. Recently, a few attempts have been made to defend against malicious attacks by incorporating proactive attacks [54, 26]. The work most closely aligned with our approach is NAB [26], which first identifies and then relabels potentially poisoned samples in the dataset, subsequently embedding non-adversarial triggers into the suspicious samples to mitigate the backdoor effect. In contrast to their methodology, our technique offers a more straightforward solution, eliminating the need for costly detection and relabeling processes, thus reducing overall costs and complexity. In essence, we demonstrate that injecting a defensive backdoor alone is sufficient to defend against backdoor attacks without requiring detection and relabeling of the poisoned samples. We refer readers to [45] for more defense in adversarial machine learning.

3 Method

In Section 3.1, we introduce the essential notations and define the threat model in this paper. Subsequently, we explore the principles behind effective defensive backdoors, illustrated with practical examples in Section 3.2. We present the overall pipeline for our proposed method in Section 3.3.

3.1 Problem setting

Notations. Considering a sample x ∈ X with label y ∈ Y , a DNN model f θ parameterized by θ is trained to classify x . The space Y = [1 , · · · , K ] denotes the space of candidate labels ( K ≥ 2 ), and X represents the sample space. In the context of backdoor attack, we denote the trigger by ∆ and the trigger injection operator by ⊕ . Consequently, given a benign sample x , the poisoned sample can be generated by x ⊕ ∆ . It's important to note that the injection operator ⊕ can vary according to the type of trigger ∆ .

Threat model. We consider a data poisoning scenario for malicious backdoor attack where the attacker can only manipulate a portion of the training dataset to plant trigge but cannot control the training process. By poisoning the dataset, the model trained on the manipulated dataset D tr would normally perform for benign input but classify the inputs with malicious trigger ∆ to predefined target ˆ y . Besides, we define the portion of manipulated samples as the poisoning ratio of backdoor attack.

The defender faces a situation where a potentially poisoned dataset is given. The defender aims to train a model where the malicious backdoor fails to be activated by the malicious trigger, and the model's utility on the original task is maintained. We assume a small benign dataset D cl is reserved

for the defender, which can be obtained by various means, including but not limited to purchase from reputable data vendors, generation via state-of-the-art generative models [7, 11, 16], collection from the internet, or applying data cleansing methods [45]. Moreover, we assume that the defender does not have knowledge of either the malicious trigger ∆ or the malicious target label ˆ y .

3.2 Proactive defensive backdoor

In this paper, we aim to defend the unknown malicious backdoor with the trigger ∆ , by inserting a proactive defensive backdoor with a trigger ∆ 1 into the model. Our primary objective is to ensure that when ∆ 1 is presented, the model's output will be controlled by ∆ 1 rather than ∆ , thereby suppressing the malicious backdoor. Besides, the model's utility on the original task should be preserved, i.e. , user can still get the true prediction of the benign sample with the defensive trigger. To achieve such a defense goal, the desired defensive backdoor attack should follow the principles below:

Principle 1: Reversibility. The defensive backdoor must be reversible, such that the ground truth label can be restored from the prediction of benign samples attached with ∆ 1 . Such a requirement is crucial for preserving the model performance on benign inputs with ∆ 1 .

Principle 2: Inaccessibility to attackers. The defensive trigger ∆ 1 should be meticulously designed to be non-replicable and undiscoverable by potential attackers. By doing so, we prevent adversaries from exploiting the same trigger or using inversion techniques to identify it.

Principle 3: Minimal impact on model performance. While stealth is not a strict requirement for the defensive trigger, modified samples should retain sufficient characteristics of the original data. This ensures accurate label recovery from the model's predictions in the presence of ∆ 1 .

Principle 4: Resistance against other backdoors. To effectively mitigate malicious backdoors, the defensive backdoor should be resistant to various backdoor attacks, not only known attacks but also potential future backdoors.

In light of the principles outlined above, we delve into the practical design of our defensive backdoor 2 .

Following Principle 1. For the first principle, we propose to assign the target label by a bijective mapping h : Y → Y , such that the target label of a sample with label y is h ( y ) and the ground truth label of a poisoned image with label y is h -1 ( y ) . Atypical choice of h and h -1 is h ( y ) = ( y +1) mod K and h -1 ( y ) = ( y -1) mod K where mod represents the modulo operation and K is the number of classes. It's worth noting that in the context of DNNs, h can also be formulated as a function of logits or fea-

Label 0

Label 1

Label 2

Label 3

Label 0

Label 1

Label 2

Label 3

tures such as h ( ϕ ( x )) = -ϕ ( x ) and h -1 ( ϕ ( x )) = -ϕ ( x ) where ϕ ( x ) corresponds to the features or logits of input. This flexibility allows for a broader range of target label assignment strategies.

Following Principle 2 & 3. To follow the second and third principles, the design of the trigger is essential. Consider the patched trigger as an illustrative example, which can be constructed by carefully determining its position and pattern. Regarding the trigger's position, it should be crafted to preserve the core visual patterns of the original image, ensuring that the primary content remains unaltered. As for the trigger's pattern, we leverage the 'home field' advantage of the defender, designing a trigger that operates beyond the conventional pixel space. Specifi-

Original Image

Mask

Pixel Value

cally, for an image with pixel values in the range of [0 , 1] , the trigger is engineered to modify regions

to values beyond this range. This modification renders the trigger infeasible and not invertible by attackers, given the natural constraints of image data.

Following Principle 4. Following the fourth principle, the defensive backdoor is required to be resistant against other backdoors in the dataset. To meet such requirements, the key point is that the defender can control the training process, a "home filed" advantage that attackers lack. On the one hand, the defender can design a strong defensive backdoor, e.g. , adopting a large trigger. On the other hand, the defensive backdoor can be further enhanced by controlling the training process, e.g. , applying data augmentation or adjusting the weight of defensive poisoned samples. More discussion and empirical findings are presented in Appendix C.7.

3.3 Backdoor injection

Data preparation

Label 0

…….

Label 1

…….

Label 9

…….

……

……

……

……

……

……

Malicious

poisoned dataset

……

……

Defensive

poisoned dataset

Model training

Label 2

Label 1

Label 1

Defensive Trigger

Label 3

Label 2

Label 0

Malicious

Target

Inference for benign sample

Inference for poisoned sample

As depicted in Figure 3, our proposed method involves three key steps:

Data preparation. Given a well-designed defensive backdoor with trigger ∆ 1 and a target label mapping h , a defensive poisoned dataset is first constructed by

Model training. Now, a model can be trained on the combination of the malicious poisoned dataset D tr and the defensive poisoned dataset ˆ D def . Then, a well-trained model will normally perform for benign inputs while controlled by the corresponding backdoor when either the trigger ∆ or ∆ 1 is presented. However, if both ∆ and ∆ 1 are simultaneously presented, the model may become confused due to the lack of such samples in the training dataset. As aforementioned, to ensure that the defensive trigger ∆ 1 effectively defeats an unknown trigger ∆ , some backdoor enhancement strategies such as data augmentation or increasing sample weight can be adopted to enhance the defensive backdoor. In summary, the overall training objective is formulated as follows:

where D tr and ˆ D def are the maliciously poisoned training dataset and the defensive poisoned dataset, respectively. The operation τ enhances the defensive backdoor by applying operation on the defensive poisoned samples ( e.g. , adding noise: τ ( x ) = x + ϵ with ϵ ∼ N (0 , 1) ).

In (2) , the first term stands for the loss on the poisoned dataset, the second term stands for the loss of injecting our defensive backdoor, and the third loss aims to enhance the defensive backdoor. We use L 0 , L 1 , and L 2 to represent the loss function for each term, which are usually Cross-Entropy losses if not specified. The parameters λ 1 and λ 2 are introduced to balance the contributions of the respective loss components. More details for the model training and implementation can be found in Appendix A.

Inference. During the inference, each input sample x is initially embedded with the defensive trigger, and the model's prediction f θ ( x ⊕ ∆ 1 ) is obtained. Subsequently, the authentic prediction is reconstructed via the inverse mapping h -1 ( f θ ( x ⊕ ∆ 1 )) .

Below, we provide a high-level pseudocode representation of our proposed method for training and inference:

Algorithm 1 Proactive Defensive Backdoor (PDB)

4 Experiments

4.1 Experiment setting

Backdoor attack. To assess our method, we consider seven leading backdoor attacks: BadNets [12], Blended method [5], Sinusoidal Signal (SIG) attacks [2], Sample-Specific Backdoor Attacks (SSBA) [22], WaNet [30], BPP attack [42] and TrojanNN attack [27]. Note that to expand our evaluation scope, we have modified certain attacks originally intended for training-controllable scenarios by excluding their training control components and we postpone the details to Appendix A. For a consistent and reliable evaluation, we utilize configurations from the BackdoorBench framework [44, 46], which offers a standardized backdoor attack assessment platform. Each attack is implemented with a 5% poisoning rate, targeting the 0 th label if not specified. The performance of these attacks is measured across three benchmark dataset, i.e. , CIFAR-10 [17], Tiny ImageNet [18], and GTSRB [37], and analyzed using three neural network architectures, i.e. , PreAct-ResNet18 [14] VGG19-BN [34] and ViT-B-16 [9]. Due to limitations in space, we present results for GTSRB and VGG19-BN in Appendix B. It is important to note that the clean label attack SIG is only applicable to CIFAR-10 with the set poisoning ratio. Additional information on these attacks is available in Appendix A.

Backdoor defense. In this paper, we benchmark our approach against popular and advanced backdoor defense methods, including AC [3], Spectral signatures [39], ABL [20], DBD [15], NAB [26]. For a fair comparison, we adopt the configurations recommended by the BackdoorBench framework [44, 46]. Note that we were unable to achieve satisfactory results for DBD on Tiny ImageNet with ViT-B-16, so it has been excluded in this case. For our method, we set the reserved dataset size to 10% of the training dataset unless otherwise specified. The chosen parameters are λ 1 = 1 and λ 2 = 1 . To enhance the defensive backdoor, each defensive poisoned sample is sampled five times in an epoch, and we set τ ( x ) = x +0 . 1 · ϵ with ϵ ∼ N (0 , 1) . The defensive backdoor utilizes a target mapping function h ( y ) = ( y +1) mod K , along with a patch trigger with pixel

value 2 as illustrated in Figure 2. More details on the defense methods and supplementary experiments are postponed in Appendix A and Appendix B.

Metrics. To measure the effectiveness of each defense method, we employ three key metrics: Accuracy on benign data ( ACC ), Attack Success Rate ( ASR ), and Defense Effectiveness Rating ( DER ). ACCis the metric indicating model's performance for predicting the benign samples correctly, while ASR quantifies the proportion of poisoned samples that are incorrectly classified to the attacker's intended target label. For our method, the ACC is measured by predicting the benign samples with a defensive trigger to the defensive target, or equivalently reversing the prediction of the benign sample with a defensive trigger to the true label. A higher ACC and a lower ASR signify successful backdoor mitigation.

The DER, used in [55, 43], is a metric ranging from 0 to 1, designed to evaluate the trade-off between maintaining ACC and reducing ASR. It is defined by the following equation:

where ∆ ASR and ∆ ACCrepresent the respective decreases in ASR and ACC between model without defense and model with defense.

Note : Superior defense methods are characterized by higher ACC , lower ASR , and higher DER . In the forthcoming experimental results, the best and second-best performing methods are denoted with boldface and underline, respectively.

4.2 Main results

Table 1 and Table 2 show the proposed method's defense performance compared with other methods, from which we can find:

PDB achieves consistent efficacy in mitigating backdoor threats across various attacks, datasets and models. Specifically, PDB achieves the top-2 lowest ASR across five out of seven attacks on the CIFAR-10 dataset. It also ranks top-2 across all attacks on the GTSRB (Table 7) and Tiny ImageNet. This consistent performance underscores PDB's ability to generalize well across different datasets and attacks. For AC and Spectral, both methods rely on the latent representation of images to detect

poisoned samples. AC identifies poisoned samples through clustering in the latent space, considering smaller clusters as likely to contain poisoned data. Spectral detects outliers in the latent space to identify such samples. However, with a poisoning ratio of 5% for Tiny ImageNet (200 classes, each class accounts for 0.5%), the poisoned samples become the majority within the target class, breaking the underlying assumptions of both methods and resulting in high ASR values. Additionally, while ABL, DBD, and NAB can defend against certain attacks, they fall short against more sophisticated adversaries, highlighting PDB's robust defense performance.

PDBachieves an excellent balance between defense performance and model utility. Apart from its robust defensive performance, PDB distinguishes itself through its ability to preserve benign accuracy. Unlike ABL, DBD, and NAB, which often sacrifice considerable benign accuracy in exchange for reduced ASR, leading to lower DER values, PDB maintains a high DER by effectively managing this trade-off. The preservation of model utility, without compromising defense effectiveness, further solidifies PDB's status as a promising strategy in backdoor defense.

The results demonstrate the superiority of PDB in defending against backdoor attacks. By effectively reducing ASR and maintaining a high DER, PDB stands out as a valuable defense approach for backdoor attack.

4.3 Analysis

Understanding the effect of PDB. To elucidate the underlying mechanism of PDB, we delve into the impact of the defensive backdoor by analyzing the T-SNE embeddings and the Trigger Activation Change (TAC). TAC, adapted from Zheng et al. [52], is designed to measure the change of activation values for each neuron when comparing maliciously poisoned samples to their benign counterparts. Let ϕ be a feature extractor which maps an input image x to the latent activations. For an input image x , we can construct the malicious poisoned sample x ⊕ ∆ . In PDB, a defensive trigger is added to the malicious poisoned sample, crafting sample x ⊕ ∆ ⊕ ∆ 1 , aiming to suppress the malicious backdoor. Therefore, for dataset D , we define

In Figure 4, we present the visualization results for the BadNets attack on the CIFAR-10 dataset, utilizing a poisoning ratio of 5% alongside a PreAct-ResNet architecture. The illustration reveals that planting a defensive trigger to the inputs prompts a shift in the feature space, resulting in the formation of new clusters and effectively alleviating the backdoor effect. Moreover, the TAC analysis for both the initial and final blocks demonstrates that the incorporation of a defensive trigger substantially mitigates the activation changes triggered by the malicious backdoor.

T-SNE Embedding

TAC, Block 1

TAC, Block 4

Neuron

Neuron

TAC

TAC

Label

Airplane

Automobile

Bird

Cat

Deer

600

Frog

Horse

Ship

Truck

Poisoned

Style

Infer w 1

Infer w/o1

w/o1

15

w/1

10

5

0

-5

-10

15

0

10

20

30

40

50

60

w/o1

10

W/△1

5

0

-5

-10

15

0

100

200

300

400

500

Defense effectiveness under different poisoning ratios. To investigate the influence of poisoning ratios on defense performance, we evaluate our method against attacks with poisoning ratios ranging from 1% to 40% on CIFAR-10 with PreAct-ResNet18. The results are summarized in Table 3, from which we can find that the proposed method can consistently mitigate malicious backdoor effect across a wide range of poisoning ratios. For a more comprehensive evaluation of the influence of the poisoning ratio, please refer to Appendix B.

Training cost comparison. We first analyze the training complexity of PDB and we refer readers to BackdoorBench[44] for the training complexity of other methods. Let C sl be the supervised training complexity. Then, we denote the size of the training dataset and the size of the defensive poisoned dataset by N tr and N def , respectively. Let F be the frequency of sampling defensive poisoned samples. The training complexity of PDB is given by: O (( 1 + F · N def N tr ) · C sl ) .

To evaluate the empirical runtime, i.e. , training time of different defense methods, we conduct experiments against the BadNets attack for the PreAct-ResNet18 architecture on CIFAR-10 and GTSRB, ViT-B-16 for Tiny ImageNet, all with a poisoning ratio of 5%. The experiments are conducted on an RTX 4090Ti GPU, and the results are summarized in Table 4. From Table 4, We can find since F · N def N tr is set as a small value, the runtime of PDB is not much larger than the baseline ( i.e. , No Defense). In contrast, the runtime of DBD and NAB are significantly higher due to their reliance on self-supervised and semi-supervised training techniques.

Resistance to ALL2ALL attack. We also evaluate PDB for ALL2ALL attacks on CIFAR-10 using PreAct-ResNet18. The poisoning ratio is set to 5% and the target labels for samples with labels y are ( y +2) mod K (different from the defensive target). The experimental results are summarized in Table 5. Notably, PDB achieves the best defending performance, demonstrating superior effectiveness in defending against backdoor attacks with multiple targets.

Resistance to adaptive attack. In our previous experiments, we assumed that attackers had no knowledge of the defense method. However, when attackers are aware of the deployment of PDB, they may design adaptive attacks to bypass the defense. One straightforward approach is to strengthen the malicious backdoor to counteract the defensive backdoor. To assess our method's resistance to

such adaptive attacks, we evaluate it against BadNets with varying trigger sizes and poisoning ratios, representing different strengths of backdoor attacks. The results, summarized in Table 6, demonstrate that PDB can consistently mitigate backdoor against adaptive attacks with various malicious trigger size and poisoning ratios. Note that to keep the stealthness of malicious backdoor, its poisoning ratio and trigger size is expected to be constrained. However, the defensive backdoor can utilize a large trigger size and high sampling frequency to meet the Principle 4, therefore, mitigating the malicious backdoor.

Appendix structure. Due to page limitations, more experiments and analyses have been moved to the Appendix. The Appendix is structured as follows: In Appendix A, we provide the details for the experiments, including the implementation of our method, the parameters, and the setting for all attacks and defense methods. In Appendix B, we provide a more comprehensive comparison between our method and baselines across different datasets, poisoning ratios, and model structures. In Appendix C, we discuss the influence of key components for PDB, such as triggers, targets, and reserved datasets, and make comparisons to more baselines.

5 Conclusion

In this paper, we propose a proactive approach to defend against malicious backdoor attacks in DNNs. Rather than relying on traditional detection and mitigation pipeline, our method, PDB, leverages the 'home field' advantage of defenders to inject a defensive backdoor to fight against malicious backdoor. To achieve such a goal, we introduce four essential design properties for an effective defensive backdoor: reversibility, inaccessibility to attackers, minimal impact on model performance, and resistance to other backdoors. By incorporating a defensive backdoor during training, we suppress the impact of malicious backdoors when the defensive trigger is present. Our approach offers several advantages over existing methods. First, it does not rely on accurate detection of poisoned samples and any assumption for attacks, avoiding performance degradation when some poisoned samples evade detection. Second, PDB does not require complex modifications to the training process, minimizing training cost. In summary, PDB provides a novel and effective defense method against backdoor attacks, enhancing the safety and reliability of DNNs.

Limitations and future work. Currently, PDB faces several key limitations. First, its reliance on clean samples presents a practical challenge, prompting the exploration of alternative sources, such as generated data. Second, investigating PDB across diverse machine learning tasks is essential for broader applicability. Addressing these limitations through future research will enhance the defense's effectiveness and facilitate its widespread adoption in safeguarding machine learning systems against backdoor attacks.

Broader impacts. The broader impacts can be considered from both positive and negative perspectives. On the positive side, PDB enhances the security and reliability of DNNs, thereby contributing to the trustworthiness of AI technologies. However, there are potential negative implications that should be considered. The technique could potentially be misused if it falls into the wrong hands, who might use the defensive backdoor mechanism for nefarious purposes.

Acknowledgments and Disclosure of Funding

Baoyuan Wu is supported by Guangdong Basic and Applied Basic Research Foundation (No. 2024B1515020095), National Natural Science Foundation of China (No. 62076213), Shenzhen Science and Technology Program under grants (No. RCYX20210609103057050), and Longgang District Key Laboratory of Intelligent Digital Economy Security. Hongyuan Zha is supported in part by the Shenzhen Key Lab of Crowd Intelligence Empowered Low-Carbon Energy Network (No. ZDSYS20220606100601002). This work is supported by Shenzhen Science and Technology Program under grant No. GXWD20201231105722002-20200901175001001, and No. ZDSYS20211021111415025, and No. JCYJ20210324120011032, and the Guangdong Provincial Key Laboratory of Big Data Computing, the Chinese University of Hong Kong, Shenzhen.

References

Insaf Adjabi, Abdeldjalil Ouahabi, Amir Benzaoui, and Abdelmalik Taleb-Ahmed. Past, present, and future of face recognition: A review. Electronics , page 1188, 2020. 1

Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In International Conference on Image Processing , 2019. 2, 6, 7, 9, 16, 18, 19, 21, 22, 23

Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In Workshop on Artificial Intelligence Safety , 2019. 1, 2, 3, 6, 7, 9, 16, 17, 18, 19, 23

Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In Conference on Neural Information Processing Systems , 2022. 1, 3

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv e-prints , pages arXiv-1712, 2017. 2, 6, 7, 9, 16, 17, 18, 19, 20, 21, 22, 23

Ziyi Cheng, Baoyuan Wu, Zhenya Zhang, and Jianjun Zhao. Tat: Targeted backdoor attacks against visual object tracking. Pattern Recognition , 2023. 2

Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023. 4

Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. In International Conference on Computer Vision , 2021. 2

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020. 6

Kuofeng Gao, Jiawang Bai, Baoyuan Wu, Mengxi Ya, and Shu-Tao Xia. Imperceptible and robust backdoor attack in 3d point cloud. IEEE Transactions on Information Forensics and Security , 19:1267-1282, 2023. 2

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 2020. 4

Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access , pages 47230-47244, 2019. 2, 6, 7, 9, 16, 17, 18, 19, 20, 21, 22, 23

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition , 2016. 1

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision , 2016. 6

Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In International Conference on Learning Representations , 2022. 1, 2, 3, 6, 7, 9, 16, 17, 18, 19, 23

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv e-prints , 2013. 4

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 2015. 6

Songze Li and Yanbo Dai. Backdoorindicator: Leveraging ood data for proactive backdoor detection in federated learning. arXiv e-prints , 2024. 25, 26

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In Conference on Neural Information Processing Systems , 2021. 1, 2, 3, 6, 7, 9, 16, 17, 18, 19, 23

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In International Conference on Learning Representations , 2021. 3, 17, 22

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In International Conference on Computer Vision , 2021. 2, 6, 7, 16, 17, 18, 19, 20, 21, 22

Weilin Lin, Li Liu, Shaokui Wei, Jianze Li, and Hui Xiong. Unveiling and mitigating backdoor vulnerabilities based on unlearning weight changes and backdoor activeness. arXiv e-prints , 2024. 3

Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In Research in Attacks, Intrusions, and Defenses , 2018. 3, 17, 22

Liangkai Liu, Sidi Lu, Ren Zhong, Baofu Wu, Yongtao Yao, Qingyang Zhang, and Weisong Shi. Computing systems for autonomous driving: State of the art and challenges. IEEE Internet of Things Journal , pages 6469-6486, 2020. 1

Min Liu, Alberto Sangiovanni-Vincentelli, and Xiangyu Yue. Beating backdoor attack at its own game. In International Conference on Computer Vision , 2023. 1, 2, 3, 6, 7, 9, 16, 17, 18, 19, 23

Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In Network and Distributed System Security Symposium , 2018. 6, 7, 16, 17, 18, 19, 20, 22

Rui Min, Zeyu Qin, Li Shen, and Minhao Cheng. Towards stable backdoor purification through feature shift tuning. In Advances in Neural Information Processing Systems , 2023. 3

Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. In International Conference on Learning Representations , 2021. 21

Tuan Anh Nguyen and Anh Tuan Tran. Wanet - imperceptible warping-based backdoor attack. In International Conference on Learning Representations , 2021. 2, 6, 7, 16, 17, 18, 19, 20, 22

Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assumption of latent separability for backdoor defenses. In International Conference on Learning Representations , 2023. 2

Xiangyu Qi, Tinghao Xie, Jiachen T Wang, Tong Wu, Saeed Mahloujifar, and Prateek Mittal. Towards a proactive { ML } approach for detecting backdoor poison samples. In USENIX Security Symposium , 2023. 3

Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Conference on Neural Information Processing Systems , 2018. 2

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations , 2015. 6

Zhengyao Song, Yongqiang Li, Danni Yuan, Li Liu, Shaokui Wei, and Baoyuan Wu. Wpda: Frequency-based backdoor attack with wavelet packet decomposition. arXiv e-prints , 2024. 2

Hossein Souri, Liam Fowl, Rama Chellappa, Micah Goldblum, and Tom Goldstein. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. In Conference on Neural Information Processing Systems , 2022. 2

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In International Joint Conference on Neural Networks , 2011. 6

J-Donald Tournier, Robert Smith, David Raffelt, Rami Tabbara, Thijs Dhollander, Maximilian Pietsch, Daan Christiaens, Ben Jeurissen, Chun-Hung Yeh, and Alan Connelly. Mrtrix3: A fast, flexible and open software framework for medical image processing and visualisation. Neuroimage , page 116137, 2019. 1

Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems , 2018. 6, 7, 9, 16, 17, 18, 19, 23

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In Symposium on Security and Privacy , 2019. 17, 22

Ruotong Wang, Hongrui Chen, Zihao Zhu, Li Liu, Yong Zhang, Yanbo Fan, and Baoyuan Wu. Robust backdoor attack with visible, semantic, sample-specific, and compatible triggers. arXiv e-prints , 2023. 2

Zhenting Wang, Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In Conference on Computer Vision and Pattern Recognition , 2022. 6, 7, 9, 16, 17, 18, 19, 20, 22, 23

Shaokui Wei, Mingda Zhang, Hongyuan Zha, and Baoyuan Wu. Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. In Advances in Neural Information Processing Systems , 2023. 3, 7

Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022. 1, 6, 9, 16

Baoyuan Wu, Shaokui Wei, Mingli Zhu, Meixi Zheng, Zihao Zhu, Mingda Zhang, Hongrui Chen, Danni Yuan, Li Liu, and Qingshan Liu. Defenses in adversarial machine learning: A survey. arXiv e-prints , 2023. 1, 3, 4

Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, Mingli Zhu, Ruotong Wang, Li Liu, and Chao Shen. Backdoorbench: A comprehensive benchmark and analysis of backdoor learning. arXiv e-prints , 2024. 1, 6

Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In Conference on Neural Information Processing Systems , 2021. 3

Danni Yuan, Shaokui Wei, Mingda Zhang, Li Liu, and Baoyuan Wu. Activation gradient based poisoned sample detection against backdoor attacks. arXiv e-prints , 2023. 1

Yi Zeng, Won Park, Z. Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In International Conference on Computer Vision , 2021. 2

Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Representations , 2022. 3, 17, 22

Zaixi Zhang, Qi Liu, Zhicai Wang, Zepu Lu, and Qingyong Hu. Backdoor defense via deconfounded representation learning. In Conference on Computer Vision and Pattern Recognition , 2023. 2, 3

Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. In European Conference on Computer Vision , 2022. 3, 8

Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Pre-activation distributions expose backdoor neurons. In Conference on Neural Information Processing Systems , 2022. 3

Hong Zhu, Shengzhi Zhang, and Kai Chen. Ai-guardian: Defeating adversarial attacks using backdoors. In Symposium on Security and Privacy , pages 701-718. IEEE, 2023. 3

Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In International Conference on Computer Vision , 2023. 3, 7, 26

Mingli Zhu, Shaokui Wei, Hongyuan Zha, and Baoyuan Wu. Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features. In Advances in Neural Information Processing Systems , 2023. 3

Mingli Zhu, Siyuan Liang, and Baoyuan Wu. Breaking the false sense of security in backdoor defense through re-activation attack. In Conference on Neural Information Processing Systems , 2024. 2

Zihao Zhu, Mingda Zhang, Shaokui Wei, Li Shen, Yanbo Fan Fan, and Baoyuan Wu. Boosting backdoor attack with a learnable poisoning sample selection strategy. arXiv e-prints , 2023. 2

Zihao Zhu, Mingda Zhang, Shaokui Wei, Bingzhe Wu, and Baoyuan Wu. Vdc: Versatile data cleanser for detecting dirty samples via visual-linguistic inconsistency. In International Conference on Learning Representations , 2024. 3

Zixuan Zhu, Rui Wang, Cong Zou, and Lihua Jing. The victim and the beneficiary: Exploiting a poisoned model to train a clean model on poisoned data. In International Conference on Computer Vision , 2023. 1, 2, 3

A Experiment details

In our experiments, we adapted all baselines and settings from BackdoorBench [44]. Moreover, all checkpoints of attack methods are sourced from BackdoorBench and the defense results are aligned with the leaderboard in BackdoorBench if applicable. Below, we outline the details of various backdoor attacks:

A.1 Attack details

BadNets [12] is one of the earliest works for backdoor learning, which inserts a small patch of fixed pattern to replace some pixels in the image. We use the default setting in BackdoorBench.

Blended backdoor attack (Blended) [5] uses an alpha-blending strategy to fuse images with fixed patterns. We set α = 0 . 2 as the default in BackdoorBench. Note that a large α causes visual-perceptible changes to clean samples, making the Blended Attack challenging for defense methods.

Sinusoidal signal backdoor attack (SIG) [2] is a clean-label attack that perturbs clean images in the target label using a sinusoidal signal as the trigger. We use the default setting in BackdoorBench.

Sample-specific backdoor attack (SSBA) [22] uses an auto-encoder to fuse a trigger into clean samples and generate poisoned samples. We use the default setting in BackdoorBench.

Warping-based poisoned networks (WaNet) [30] is also a training-controllable attack that perturbs clean samples using a warping function to construct poisoned samples. We use the default setting in BackdoorBench.

Bppattack (BPP) [42] is also a training-controllable attack that employs image quantization and dithering as the Trojan trigger. We use the default setting in BackdoorBench.

Trojaning attack on neural networks (TrojanNN) [27] inverses the neural network to generate a general trojan trigger. We use the default setting in BackdoorBench.

Adaptation to data poisoning attack. In our paper, we explore scenarios where attacks can only utilize data poisoning techniques. To facilitate a more comprehensive comparison of our method, we modify attacks originally designed for training-controllable scenarios, removing the training component to adapt them to a data poisoning setting.

A.2 Defense details

Here, we summarize the details of each defense method used:

AC [3] is a detection method that detects the poisoned sample using the abnormal clustering for poisoned samples. By removing the detected samples, AC can effectively defend against backdoor attack. We use the default setting in BackdoorBench.

Spectral [39] is a detection method that detects the poisoned sample using the abnormal Spectral Signature for poisoned samples. By removing the detected samples, AC can effectively defend against backdoor attack. We use the default setting in BackdoorBench.

ABL [20] utilizes the early-learning phenomenon of poisoned samples to detect poisoned samples and then unlearns them to mitigate the backdoor effect. We use the default setting in BackdoorBench.

DBD [15] divides the training process into three stages and uses self-supervised techniques to detect the poisoned sample and learn a clean model. We use the default setting in BackdoorBench.

NAB [26] first employs an advanced detection method to filter the poisoned samples. Then, the detected samples are relabeled by employing other techniques and planted with nonadversarial triggers to suppress the backdoor. In this work, we use the detection method from ABL and the self-supervised method from DBD to relabel the samples. For other settings, We use the default setting in BackdoorBench.

FT finetunes the model on a small, clean, reserved dataset to mitigate the backdoor effect. We use the default setting in BackdoorBench.

FP [24] is a pruning-based method that prunes neurons according to their activations and then fine-tunes the model to keep clean accuracy. We use the default setting in BackdoorBench.

NC [40] first optimizes a possible trigger to detect backdoored models. If detected as backdoored, unlearn the optimized trigger. If detected as clean, the model is returned unchanged.

NAD [21] uses Attention Distillation to mitigate backdoors. We use the default setting in BackdoorBench.

i-BAU [50] uses adversarial training with UAP and hyper-gradient to mitigate the backdoor. We use the default setting in BackdoorBench.

PDB ( Ours ) defends backdoor attack by injecting defensive backdoor. We set the reserved dataset size to 10% of the training dataset. The chosen parameters are λ 1 = 1 and λ 2 = 1 . To enhance the defensive backdoor, each defensive poisoned sample is sampled five times in an epoch, and we set τ ( x ) = x +0 . 1 · ϵ with ϵ ∼ N (0 , 1) . The defensive backdoor utilizes a target mapping function h ( y ) = ( y +1) mod K , along with a 7 × 7 patch trigger with pixel value 2 as illustrated in Figure 2.

Adaptation to ViT-B-16. For all experiments on CIFAR-10 and GTSRB, we train the model 100 epochs with batch size 256 for fair comparison. For Tiny ImageNet with ViT-B-16, we consider a fine-tuning task as recommended by BackdoorBench. Specifically, we train each model 10 epochs with batch size 128 and initialize the model with pre-trained weights.

B Additional experiment results

This section provides additional experiment results to supplement the observations claimed in Section 4.

B.1 Main experiments on GTSRB with PreAct-ResNet18

Table 7, 8, and 9 summarize the results of various defense methods against backdoor attacks on the GTSRB dataset using the PreAct-ResNet18 model architecture. These methods were evaluated at different poisoning ratios (1.0%, 5.0%, and 10.0%). Notably, the results demonstrate that PDB effectively mitigates backdoor attacks, consistently achieving top-2 defense performance across all cases.

B.2 Main experiments on CIFAR-10 with PreAct-ResNet18

Table 10 and 11 summarize the results of different defense methods against backdoor attacks on the CIFAR-10 dataset using the PreAct-ResNet18 model architecture. These methods were evaluated at different poisoning ratios (1.0% and 10.0%). Notably, they achieved the top-2 lowest ASR in 12 out of 14 cases.

B.3 Main experiments on CIFAR-10 with VGG19-BN

Table 12, 13, and 14 summarize the results of different defense methods against backdoor attacks on the CIFAR-10 dataset using the VGG19-BN model architecture. These methods were evaluated at different poisoning ratios (1.0%, 5%, and 10.0%). Impressively, PDB achieves the top-2 lowest ASR in 20 out of 21 cases.

B.4 Main experiments on Tiny ImageNet with ViT-B-16

To demonstrate the effectiveness and scalability of PDB, we evaluate our proposed method against backdoor attacks on the Tiny ImageNet dataset using the ViT-B-6 model architecture. We consider different poisoning ratios (1.0%, 5%, and 10.0%). The results, presented in Table 15, highlight our method's ability to effectively mitigate backdoor attacks for a large dataset and a large model.

B.5 Experiments on invisible backdoor attack and low-poisoning ratio attack

As aforementioned, the proposed method, PDB, does not rely on specific assumptions about the type of attack, making it effective for defending against both invisible backdoor attacks and attacks with low poisoning ratios. To demonstrate this effectiveness, we conducted experiments using low poisoning ratios (0.5% and 0.1%) for both Visible and Invisible attacks. The results are summarized in Table 16, from which we can find that PDB can consistently mitigate backdoor attacks.

C Discussion and additional analysis

C.1 Influences of reserved dataset

We explore the impact of the reserved dataset on defense performance, considering both dataset size and source:

Dataset size. We investigate the influence of reserved dataset size using the CIFAR-10 dataset, a 5% poisoning ratio, and the PreAct-ResNet architecture. Figure 5 summarizes the results, demonstrating that our proposed method effectively mitigates malicious backdoor effects across a wide range of reserved dataset sizes. Notably, increasing the reserved dataset significantly improves the accuracy of our method.

5%

10%

50%

100%

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

ACC/ASR

BadNets

5%

10%

50%

100%

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

Blended

5%

10%

50%

100%

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

WaNet

5%

10%

50%

100%

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

SSBA

ACC

ASR

ACC

ASR

Defense with generated dataset. As generative models have evolved, incorporating the generated dataset as an additional source for backdoor defense has become practical and reasonable. To explore the source of this reserved dataset, we assess our method using the generated data CIFAR5m from Nakkiran et al. [29]. with DDPM. Our findings, summarized in Table 17, demonstrate that the advanced generative model can indeed supply a sufficient dataset for applying our defense strategy.

C.2 Influences of the trigger and target

Mask 1

Mask 2

Mask 3

(a)

(b)

(c)

In this section, we explore the impact of trigger design and target assignment strategy. Specifically, we evaluate different trigger configurations using a patch trigger with three distinct masks (illustrated in Figure 6). Additionally, we consider two target assignment strategies, i.e. , h 1 ( y ) = ( y + 1) mod K and h 2 ( ϕ ( x )) = -ϕ ( x ) , where y is the hard label and ϕ ( x ) is the logits for the model output. Our experiments, detailed in Table 18, demonstrate the effectiveness of our method across various defensive backdoor designs.

Special case study: Same target label for malicious backdoor and defensive backdoor. While our trigger remains inaccessible to the attacker, the target assignment strategy could potentially be stolen or coincidentally used by the attacker. In this scenario, we conduct experiments where both the attacker and the defender employ the same target assignment strategy but different triggers. Our results demonstrate that our method remains effective in such cases. Specifically, assuming both the attacker and defender choose h ( y ) = ( y +1) mod K , we summarize the results in Table 19, highlighting our method's resilience even when the attacker uses the same target label as the defender.

C.3 Influences of augmentation

In this section, we explore the impact of augmentation on enhancing defensive backdoors using the CIFAR-10 dataset, a 5% poisoning ratio, and the PreAct-ResNet architecture. We specifically investigate Gaussian noise augmentation by setting τ ( x ) = x + α ∗ ϵ with ϵ ∼ N (0 , 1) determines the augmentation intensity. The results are summarized in Figure 7. Notably, even without any augmentation ( α = 0 ), PDB exhibits significant efficacy, likely due to the robustness of the defense mechanisms and our controlled training injection process. Furthermore, as augmentation strength increases, the ASR decreases, albeit at the cost of reduced ACC, indicating a tradeoff between augmentation intensity and model performance.

0.0

0.1

0.2

0.3

0.4

0.5

Strength

0.0

0.2

0.4

0.6

0.8

1.0

ACC/ASR

BadNets

0.0

0.1

0.2

0.3

0.4

0.5

Strength

0.0

0.2

0.4

0.6

0.8

1.0

Blended

0.0

0.1

0.2

0.3

0.4

0.5

Strength

0.0

0.2

0.4

0.6

0.8

1.0

WaNet

0.0

0.1

0.2

0.3

0.4

0.5

Strength

0.0

0.2

0.4

0.6

0.8

1.0

SSBA

ACC

ASR

ACC

ASR

C.4 Comparison with post-training methods

Given a reserved dataset, the defender may follow a 'poisoned first and mitigation later' manner to train the backdoored model first and employ post-training method to mitigate the backdoor effect. Therefore, to thoroughly evaluate our method, we compare it with SOTA post-training approaches that aim to remove backdoor effects after model training. To ensure fairness, we adopt the common practice of reserving 5% of the training data for post-training evaluation. Our experiments are conducted on the CIFAR-10 dataset using PreAct-ResNet18 with a 5% poisoning ratio. The summarized results in Table 20 demonstrate that our method consistently achieves superior performance in defending against backdoor attacks, highlighting its promising effectiveness.

C.5 Detailed comparison to NAB

As previously mentioned, the work most relevant to our research is NAB [26], which presents a promising and impressive approach for enhancing backdoor defense methods. In this context, we conduct a detailed analysis to highlight the distinctions between our method, PDB, and NAB.

PDB does not rely on poisoned sample detection. Our method diverges from the conventional 'detection-mitigation' pipeline by operating independently of any poison detection methods. In contrast, NAB relies on identifying poisoned samples to deploy its defensive trigger.

PDB does not depend on suspicious sample relabeling. Unlike NAB, our method does not require accurate sample relabeling. In NAB, detected samples are equipped with a defensive trigger and subsequently relabeled. Unfortunately, if a few clean samples are mistakenly labeled as 'suspicious,' the defensive trigger may function as a malicious one if further relabeling is incorrect. Note that accurately relabeling a sample is getting harder as the dataset gets larger. Therefore, we observe such scenarios appear more frequently in large datasets during our experiments. For instance, when applying the BadNets attack with a 5% poisoning ratio on Tiny ImageNet using ViT-B-16, NAB's model accuracy drops from 73.60% (without a defensive trigger) to 28.88% (with the trigger) due to low relabel correctness and poison detection rates.

C.6 Resistance to ALL2ALL attack

In this section, we compare PDB with other baselines in ALL2ALL attacks on CIFAR-10 using PreAct-ResNet18. The poisoning ratio is set to 5% and 10%. Specifically, the target labels for samples with original labels y are adjusted to ( y +2) mod K (different from the defensive target). The experimental results are summarized in Table 21 and Table 22. Notably, PDB achieves the best defending performance, demonstrating superior effectiveness in countering backdoor attacks with multiple targets.

C.7 Design of defensive trigger and the satisfaction to Principle 4

Here, we summarize the aforementioned experiments and provide a comprehensive discussion about how to meet Principle 4 (Resistance against other backdoors) from the following perspectives:

· Design of defensive trigger:

Defensive trigger size: Trigger size directly contributes to the strength of the defensive backdoor. In Table 23, we evaluate PDB with a square defensive trigger with sizes ranging

from 1x1 to 9x9. From Table 23, we can find that a larger trigger leads to a stronger defensive backdoor , resulting in a higher ACC and a lower ASR. However, as the trigger size increases, it may interfere with the visual content of the image , leading to a slight decrease in ACC. Notably, as the square trigger has strong visibility, a trigger with size 1x1 can still alleviate the malicious backdoor to some extent.

Defensive trigger position: As aforementioned, the position of the defensive trigger is essential for Principle 3, i.e., minimal impact on model performance. Table 24 shows that triggers placed in different positions (corner, random, and center) achieve a similar effect in defending against backdoor attacks. However, placing a trigger at the center of an image significantly degrades accuracy, as the trigger masks the core patterns of the image.

Pixel value: For a square trigger, pixel value is also an important parameter for PDB. In Table 25, we evaluate PDB with a square defensive trigger with different pixel values, from which we can find that PDB can achieve high effectiveness across different pixel values.

Backdoor enhancement strategy during training:

Increasing sampling frequency: Given a fixed number of defensive poisoned samples, the defensive backdoor can be further enhanced by increasing the sampling frequency of poisoned samples, forcing the model to pay more attention to defensive poisoned samples. Table 26 shows that a larger sampling frequency leads to a stronger defensive backdoor , resulting in a higher ACC and a lower ASR. Note that for the malicious attacker, the poisoning ratio is expected to be low to ensure the stealthiness of the attack.

Data augmentation: From Fig 7, we can find that PDB, without any sample augmentation ( α = 0 ), exhibits significant efficacy with ASR lower than 2%. As augmentation

strength increases, the ASR decreases, indicating a stronger augmentation can help further enhance PDB's effectiveness . However, a tradeoff between augmentation intensity and model performance is also observed.

In summary, a visible trigger with larger trigger size , higher sampling frequency , and data augmentation contribute to meeting Principle 4.

C.8 Factors that influence the accuracy of PDB

Here, we would like to discuss the factors that influence the accuracy of PDB:

Model capacity and data complexity:

Model capacity: Since PDB introduces additional task, i.e., injecting defensive backdoor, increasing the model capacity helps to increase the accuracy of PDB, as evidenced in Table 27.

Dataset complexity: By comparing defense results with different datasets, we can find that by decreasing the dataset complexity, the accuracy of PDB increases significantly.

· Strength of defensive backdoor:

Strength of augmentation: From Figure 7, we can find that there exists a tradeoff between ACC and ASR. Therefore, the accuracy of PDB can be boosted by reducing the strength of augmentation.

Sampling frequency: From Table 26, we can find that by increasing the sampling frequency of defensive poisoned samples, the accuracy of PDB can be boosted.

Trigger size: Table 23 shows that a proper choice of trigger size can also help to increase the accuracy. Therefore, if a validation set is accessible, a proper trigger size can be chosen to increase accuracy.

In summary, due to the "home field advantage" of PDB, there are several ways to maintain a high accuracy even in the case of a low malicious poisoning ratio, such as increasing model capacity, simplifying the dataset, reducing the strength of augmentation to defensive poisoned samples, increasing the sampling frequency and choosing a proper defensive trigger size.

C.9 Novelty and comparison with backdoorIndicator

Here, we highlight the distinctions between our approach and BackdoorIndicator [19].

First , we would like to clarify the following differences between our work and backdoorIndicator [19]:

Threat model: [19] targets decentralized training (FL) setting, where multiple clients train models locally and contribute updates to a central server. Our work considers a centralized training setting where only a central server is used.

Task: [19] focuses on detecting malicious clients, whereas our method aims to train a secure model on a poisoned dataset without clients.

Motivation: [19] is built on the motivation that planting subsequent backdoors with the same target label enhances previously planted backdoors, therefore, providing a way to detect the poisoned clients, while our method is based on the motivation that planting a concurrent reversible backdoor can help to mitigate the malicious backdoor.

Methodology: [19] utilizes OOD samples for backdoor client detection while our method constructs a proactive defensive poison dataset, following well-designed principles.

Second , we would like to discuss the challenges in direct utilizing backdoorIndicator in our setting:

BackdoorIndicator is designed to detect malicious clients within a federated learning (FL) context. This makes it challenging to apply BackdoorIndicator directly to our centralized environment since the task of identifying backdoored clients does not naturally fit into this setting (only a central server).

For comparison between BackdoorIndicator and our method, we need to emulate an FL scenario by assigning each image to a separate client (ensuring existence of benign client), thereby creating 50,000 local models from the CIFAR-10 dataset to defend a single attack with PreAct-ResNet18. This would require an impractical amount of computational resources, estimated at over 30,000 hours (1,250 days) of training time and 30TB of storage space using a server with a single RTX 3090 GPU.

C.10 Comparison to FT-SAM

To address the comparison with FT-SAM [55], we have adapted their method to our experimental setting. It's worth noting that in [55], the authors employ the Blended attack with a blending ratio of 0.1 (Blended-0.1), whereas we use a blending ratio of 0.2 (Blended-0.2). For consistency and completeness, we have now included experiments using both blending ratios, and the results are shown below:

From Table 28, we can find FT-SAM can achieve a higher accuracy as it aims to fine-tune a backdoored model while PDB aims to train a model from scratch. Consistent with [55], Table 28 shows that FT-SAM can mitigate backdoor attacks for most cases, except for the Blended-0.2. We observe that FT-SAM struggles to defend against blended attacks with higher blending ratios, such as 0.2. Notably, PDB achieves a significantly lower ASR across all cases, with an average ASR below 0.5%.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction of the paper accurately reflect the paper's contributions and scope. They clearly state that the paper addresses the challenge of training a clean model on a potentially poisoned dataset by proposing a novel defense mechanism.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of the work performed by the authors. In the "Limitations and future work" section, we acknowledge that their implementation of the reversible backdoor defense has several key limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper appears to provide detailed information on the experimental setup based on a popular Benchmark Project, including the datasets used, the model architectures, the backdoor attack strategies, and the defense mechanisms compared.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Provided in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Provided in the main text and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Provided in the main text and the appendix. Due to space limit, the bar are mainly visualized in the plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Provided in the main text and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Provided in the main text.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the in-training backdoor defense by proactively injecting a defensive backdoor into the model during training.  During inference, PDB embeds a defensive trigger in the inputs and reverses the model prediction. This operation effectively suppresses malicious backdoors and ensures the model utility on the original task. Various experiments demonstrate the effectiveness.\n\nSTRENGTHS:\nVarious experiments demonstrate the proposed method is effective.\n\nWEAKNESSES:\nThis paper has several limitations.\n\n\n1. Novelty has been discovered by existing methods. I assume that the author truly share a interesting insight towards the backdoor defense. But in [2] (Usenix Security 2024), “backdoor samples are OOD samples compared to benign samples from the target class, designs indicator task leveraging OOD samples to identify and rule out backdoor updates”.  But I do not see that author has discussed in your paper. I wonder whether authors could explain whether directly utilizing [2] in your setting has any problem. \n\n\n2. Missing comparison and confusing results. As shown in your Table 1: Results (%) on CIFAR-10 with PreAct-ResNet18 and poisoning ratio 5.0%, for attack Trojan, ABL achieves 18.64 100.00, but the results in [1] is not the same. 70.70/0.02. Why do you not present the same setting as the current relative works? I wonder whether you could compare your work in your setting or the [1] scenario.\n\n3. Abaltion for trigger pattern. \"Regarding the trigger’s position, it should181 be crafted to preserve the core visual patterns182 of the original image\". I wonder whether the author could conduct the experiments on the trigger pattern position to demonstrate your conclusion. \n\n[1] Enhancing Fine-Tuning Based Backdoor Defense with Sharpness-Aware Minimization. ICCV 2023.\n[2] BackdoorIndicator: Leveraging OOD Data for Proactive Backdoor Detection in Federated Learning. Usenix Security 2024\n\nQUESTIONS:\nYour work shares a high similar motivation with the BackdoorIndicator. I encourage authors to provide detailed explanations.\n\nLIMITATIONS:\nYes, the authors have discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates in-training backdoor defense by proactive defensive backdoor. They introduce a defensive poisoned dataset to train the model together with all poison data. By attach the defensive trigger onto the input sample, the potential backdoor attack is failed by such proactive defensive backdoor. Extensive experiments are conducted to evaluate the effectiveness of our method and compare it with five SOTA defense methods across seven challenging backdoor attacks.\n\nSTRENGTHS:\nThe author offers a straightforward solution to eliminate the need for costly detection and relabeling processes, thus improving the efficiency.\n\nThe experiments seem comprehensive, great attach success rate and defensive effectiveness rate are achieved.\n\nWEAKNESSES:\nThe author should elaborate why DBD, NAB and V&B result in a substantial increase in training costs in Introduction. Experimental analyses are highly desired.\n\nThe contribution (1) is overclaimed. This is not the novel paradigm because this work develop based on several proactive attack-based works like NAB.\n\nThe sensitivity to the defensive poisoned dataset of the proposed method should be evaluated. Though the author claim the proposed method follow Principle 4, more details and thorough analyses are required.\n\nDespite better defense effectiveness rate of the proposed method, its accuracy on benign samples is obviously inferior to other methods. When the poisoning ratio is decreasing, the proposed method may unexpectedly become an attack to the original benign dataset, resulting in the degraded accuracy. How to avoid this situation?\n\nQUESTIONS:\nWhy the proposed method can meet Principle 4? The explanation in Method is not clear.\n\nWhat is the difference between the malicious poisoned dataset and the defensive poisoned dataset when preparing the data?\n\nLIMITATIONS:\nThe generalization of the proposed method to diverse backdoor attack remains unknown. When the attack is invisible or the poison rate is mariginal, the proposed method may degenerate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method called Proactive Defensive Backdoor (PDB) to counter backdoor attacks in deep neural networks. PDB differs from traditional methods, which focus on detecting and eliminating suspicious samples. Instead, PDB proactively injects a defensive backdoor into the model during the training phase. PDB operates by embedding a trigger in the model's inputs during prediction, which neutralizes the effects of any malicious trigger present. The authors designed the defensive backdoor in such a way that it can reverse the model's prediction to its true label, thus maintaining its utility on benign tasks. Through extensive experimentation, the authors have demonstrated PDB's ability to outperform existing defense methods by achieving a balance between suppressing malicious backdoors and preserving model performance across various datasets and model architectures.\n\nSTRENGTHS:\n- The paper is well-structured and clearly written.\n- Introduces a novel approach for mitigating backdoor attacks.\n- Detailed experiments and comparison with state-of-the-art.\n\nWEAKNESSES:\n- The authors evaluated adaptive attacks only by increasing the trigger size of the BadNets attack. A more appropriate adaptive attack would increase the poisoning ratio to strengthen the backdoor effect compared to the defensive backdoor.\n- It would be interesting to see if PDB can scale for clean label backdoor attacks, where the adversary does not change the trigger label during the backdoor attack.\n- It is not clear whether the running time in Table 4 refers to training time or inference time. Comparing inference time would be more appropriate, as PDB requires the model to make two predictions for a single image. How does the inference time of PDB compare to other defenses in the literature?\n- It is not clear why the specific blocks, namely Block 1 and Block 4, were chosen to illustrate the impact of PDB. Also, the plots related to TAC are difficult to understand from the text.\n\nQUESTIONS:\n- Will PDB be effective against adaptive attacks that increase the poisoning ratio?\n- How does the inference time of PDB compare with other defenses in the literature?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an proactive defense approach called PDB, which aims to combat malicious backdoor attacks by injecting active defense backdoors introduced by the defender. The main goal of PDB is to suppress the impact of malicious backdoors while preserving the utility of the model for its original tasks. PDB first analyzes the objectives for effective backdoor defense and introduces four fundamental design principles: reversibility, inaccessibility to attackers, minimal impact on model performance, and resistance against other backdoors. Then, an additional defensive poisoning dataset is constructed, and the model is trained using both this dataset and the entire poisoned dataset. To evaluate its effectiveness, the paper compares PDB with five SOTA in-training defense methods against seven SOTA data poisoning backdoor attack methods involving different model architectures and datasets. Experimental results show that PDB performs comparably or even better than existing baseline methods.\n\nSTRENGTHS:\n1.\tThe perspective of the paper is novel.\n2.\tThis paper is well-written and easy to understand.\n\nWEAKNESSES:\n1.\tThe paper lacks an effective explanation for PDB.\n2.\tThe paper lacks sufficient and reasonable explanations for the experimental results.\n\nQUESTIONS:\nThe perspective of the paper is innovative, and it is well-written and easy to understand. However, this paper still has the following issues.\n1.\tThe paper lacks sufficient and reasonable explanations for the experimental results, particularly those in Table 2. In Table 2, the ASR results for PDB under different attacks are all 0, while the results for other baseline are close to 100 (i.e., they all fail to defend successfully). The paper should provide a reasonable explanation for the significant difference in ASR between PDB and the other baselines.\n2.\tIn the Section of Resistance to Adaptive Attack, the paper evaluates PDB's resistance to malicious backdoor attacks with trigger sizes ranging from 3x3 to 6x6. In this section, the paper should set the backdoor attack trigger size to greater than 7x7 to match the trigger size set by PDB. Additionally, it raises the question of why the paper sets the trigger size for PDB to 7x7?\n3.\tIn the Section of Influences of Augmentation, different strength of augmentation do not show significant changes in defense-related ACC and ASR, even in cases without any augmentation. Therefore, what is the role of augmentation during the training process? Based on this, it is easy to question why PDB's defense triggers can be stronger than the triggers of the backdoor without relying on augmentation?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}