Title: Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples
Venue: NeurIPS 2023
Accepted: yes
Keywords: Backdoor Attack

Trustworthy AI

Backdoor Learning
Abstract: Backdoor attacks are serious security threats to machine learning models where an adversary can inject poisoned samples into the training set, causing a backdoored model which predicts poisoned samples with particular triggers to particular target classes, while behaving normally on benign samples. In this paper, we explore the task of purifying a backdoored model using a small clean dataset. By establishing the connection between backdoor risk and adversarial risk, we derive a novel upper bound for backdoor risk, which mainly captures the risk on the shared adversarial examples (SAEs) between the backdoored model and the purified model. This upper bound further suggests a novel bi-level optimization problem for mitigating backdoor using adversarial training techniques. To solve it, we propose Shared Adversarial Unlearning (SAU). Specifically, SAU first generates SAEs, and then, unlearns the generated SAEs such that they are either correctly classified by the purified model and/or differently classified by the two models, such that the backdoor effect in the backdoored model will be mitigated in the purified model. Experiments on various benchmark datasets and network architectures show that our proposed method achieves state-of-the-art performance for backdoor defense. The code is available at https://github.com/SCLBD/BackdoorBench (PyTorch) and https://github.com/shawkui/MindTrojan (MindSpore).

Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples

Abstract

Backdoor attacks are serious security threats to machine learning models where an adversary can inject poisoned samples into the training set, causing a backdoored model which predicts poisoned samples with particular triggers to particular target classes, while behaving normally on benign samples. In this paper, we explore the task of purifying a backdoored model using a small clean dataset. By establishing the connection between backdoor risk and adversarial risk, we derive a novel upper bound for backdoor risk, which mainly captures the risk on the shared adversarial examples (SAEs) between the backdoored model and the purified model. This upper bound further suggests a novel bi-level optimization problem for mitigating backdoor using adversarial training techniques. To solve it, we propose S hared A dversarial U nlearning (SAU). Specifically, SAU first generates SAEs, and then, unlearns the generated SAEs such that they are either correctly classified by the purified model and/or differently classified by the two models, such that the backdoor effect in the backdoored model will be mitigated in the purified model. Experiments on various benchmark datasets and network architectures show that our proposed method achieves state-of-the-art performance for backdoor defense. The code is available at https://github.com/SCLBD/BackdoorBench (PyTorch) and https://github.com/shawkui/MindTrojan (MindSpore).

1 Introduction

Recent decades have witnessed a growing application of deep neural network (DNN) techniques in various domains, such as face recognition, autonomous driving, and medical image processing [1, 17, 32, 44]. However, DNNs are known to be vulnerable to malicious attacks that can compromise their security and reliability. One of the emerging threats is the backdoor attack, where an attacker attempts to inject stealthy backdoor into a DNN model by manipulating a small portion of the training data such that the poisoned model will behave normally for clean inputs while misclassifying any input with the trigger pattern to a target label. Since the poisoned models can have real-world consequences, such as allowing an attacker to gain unauthorized access to a system or to cause physical harm, it is crucial to develop effective methods for backdoor defense in developing and deploying DNNs.

Adversarial training (AT) is one of the most effective methods for improving the robustness of DNNs [33]. AT is typically formulated as a min-max optimization problem, where the inner maximization aims to find adversarial perturbations that fool the model, while the outer minimization is to reduce the adversarial risk. Recently, several works have explored the use of AT for defending against backdoor attacks. However, these works face some limitations and challenges. Weng et al. [52] and

Gao et al. [13] found that AT can only be effective for mitigating backdoor attacks with certain trigger patterns and may even increase the vulnerability to backdoor attacks when training from scratch on poisoned datasets. Zeng et al. [59] proposed to unlearn the universal adversarial perturbation (UAP) to remove backdoors from poisoned models. Since directly unlearning UAP leads to highly unstable performance for mitigating backdoors, they employ implicit hyper gradient to solve the min-max problem and achieve remarkable performance. However, their method assumes that the same trigger can activate the backdoor regardless of the samples it is planted on, and therefore lacks a guarantee against more advanced attacks that use sample-specific and/or non-additive triggers.

In this paper, we consider the problem of purifying a poisoned model. After investigating the relationship between adversarial examples and poisoned samples, a new upper bound for backdoor risk to the fine-tuned model is proposed. Specifically, by categorizing adversarial examples into three types, we show that the backdoor risk mainly depends on the shared adversarial examples that mislead both the fine-tuned model and the poisoned model to the same class. Shared adversarial examples suggest a novel upper bound for backdoor risk, which combines the shared adversarial risk and vanilla adversarial risk. Besides, the proposed bound can be extended with minor modifications to universal adversarial perturbation and targeted adversarial perturbation. Based on the new bound, we propose a bi-level formulation to fine-tune the poisoned model and mitigate the backdoor. To solve the bi-level problem, we proposed S hared A dversarial U nlearning (SAU). SAU first identifies the adversarial examples shared by the poisoned model and the fine-tuned model. Then, to break the connection between poisoned samples and the target label, the shared adversarial examples are unlearned such that they are either classified correctly by the fine-tuned model or differently by the two models. Moreover, our method can be naturally extended to defend against backdoor attacks with multiple triggers and/or multiple targets. To evaluate our method, we compare it with six state-of-the-art (SOTA) defense methods on seven SOTA backdoor attacks with different model structures and datasets. Experimental results show that our method achieves comparable and even superior performance to all the baselines.

Our contributions are three folds: 1) We analyze the relationship between adversarial examples and poisoned samples, and derive a novel upper bound for the backdoor risk that can be generalized to various adversarial training-based methods for backdoor defense; 2) We formulate a bi-level optimization problem for mitigating backdoor attacks in poisoned models based on the derived bound, and propose an efficient method to solve it; 3) We conduct extensive experiments to evaluate the effectiveness of our method, and compare it with six state-of-the-art defense methods on seven challenging backdoor attacks with different model structures and datasets.

2 Related work

Backdoor attack. Backdoor attack is one of the major challenges to the security of DNNs. The poisoned model behaves normally on clean inputs but produces the target output when the trigger pattern is present. Based on the types of triggers, backdoor attacks can be classified into two types: fixed-pattern backdoor attacks and sample-specific backdoor attacks. BadNets [15] is the first backdoor attack that uses fixed corner white blocks as triggers. To improve the stealth of the triggers, Blended [7] is proposed to blend the trigger with the image in a weighted way. Since fixed-pattern triggers can be recognized easily, sample-specific backdoor attacks have been proposed. SSBA [30], WaNet [38], LF [58], IRBA [12], VSSC [49] and TAT [8] use different techniques to inject unique triggers for different samples from different angles. Sleeper-agent [42] and Lira [9] optimize the target output to obtain more subtle triggers. Recently, Zhu et al. [67] proposed a learnable poisoning sample selection strategy to further boost the effect of backdoor attacks. To keep the label of the backdoor image matching the image content, LC [40] and SIG [2] use counterfactual and other methods to modify the image to deploy clean label attacks.

Backdoor defense. Backdoor defense aims to reduce the impact of backdoor attacks on deep neural networks (DNNs) through training and other means. There are three types of backdoor defense: pre-processing, in-processing, and post-processing. Pre-processing backdoor defense aims to identify the poisoned samples in the training dataset. For instance, AC [5] filters out the poisoned samples by the abnormal activation clustering phenomenon in the target class; Confusion Training [39] identifies the poisoned samples by training a poisoned model that only fits the poisoned samples; VDC [68] incorporates multimodal large language models to detect the poisoned samples. In-processing

backdoor defense reduces the effect of the backdoor during training. For instance, ABL [27] exploits the fact that the learning speed of backdoor samples is faster than that of the clean sample, and splits some poisoned samples to eliminate the backdoor by forgetting these poisoned samples; DBD [19] divides the backdoor training process and directly inhibits the backdoor learning process; D-ST [6] splits the backdoor samples and uses semi-supervised learning by observing that the clean samples are more robust to image transformation than the backdoor samples. Post-processing backdoor defense mitigates the effect of backdoors for a poisoned model by pruning or fine-tuning. For example, FP [31] prunes some potential backdoor neurons and fine-tunes the model to eliminate the backdoor effect; ANP [55] finds the backdoor neurons by adversarial perturbation to model weights; EP [64] and CLP [63] distinguish the characteristics of the backdoor neurons from the clean neurons; NAD [28] uses a mild poisoned model to guide the training of the poisoned model to obtain a cleaner model; I-BAU [59] finds possible backdoor triggers by the universal adversarial attack and unlearn these triggers to purify the model; FT-SAM [65] boosts the performance of fine-tuning for backdoor mitigation by incorporating sharpness-aware minimization; NPD [66] learns a lightweight linear transformation layer by solving a well designed bi-level optimization problem to defend against backdoor attack.

Adversarial training. In adversarial training, models are imposed to learn the adversarial examples in the training stage and therefore, resistant to adversarial attacks in the inference stage. In one of the earliest works [14], the adversarial examples are generated using Fast Gradient Sign Method. In [33], PGD-AT is proposed, which generates adversarial examples by running FGSM multiple steps with projection and has become the most widely used baseline for adversarial training. Some further improvements of PGD-AT include initialization improvement [21, 23], attack strategy improvement [22], and efficiency improvement [53, 62].

3 Methodology

In Section 3.1, we first introduce notations, threat model, and defense goal to formulate the problem. By investigating the relationship between adversarial risk and backdoor risk, a new upper bound of backdoor risk is derived in Section 3.2, from which a bi-level formulation is proposed in Section 3.3.

3.1 Preliminary

Notations. We consider a K -class ( K ≥ 2 ) classification problem that aims to predict the label y ∈ Y of a given sample x ∈ X , where Y = [1 , · · · , K ] is the set of labels and X is the space of samples. Let h θ : X → Y be a DNN classifier with model parameter θ . We use D to denote the set of data ( x , y ) . For simplicity, we use x ∈ D as a abbreviation for ( x , y ) ∈ D . Then, for a sample x ∈ D , its predicted label is

where p k ( x ; θ ) is the (softmax) probability of x belonging to class k .

Threat model. Let V be the set of triggers and define g : X × V → X as the generating function for poisoned samples. Then, given a trigger ∆ ∈ V and a sample x ∈ X , one can generate a poisoned sample g ( x ; ∆) . For simplicity, we only consider all to one case, i.e. , there is only one trigger ∆ and one target label ˆ y . The all to all case and multi-trigger case are left in Appendix A. We assume that the attacker has access to manipulate the dataset and/or control the training process such that the trained model classifies the samples with pre-defined trigger ∆ to the target labels ˆ y while classifying clean samples normally. In addition, we define the poisoning ratio as the proportion of poisoned samples in the training dataset.

̸

Defense goal. We consider a scenario where a defender is given a poisoned model with parameter θ bd and a small set of clean data D cl = { ( x i , y i ) } N i =1 . Since we are mainly interested in samples whose labels are not the target label, we further define the set of non-target samples as D -ˆ y = { ( x , y ) | ( x , y ) ∈ D cl , y = ˆ y } . Note that the size of D cl is small and the defender cannot train a new model from scratch using only D cl . The defender's goal is to purify the model so that the clean performance is maintained and the backdoor effect is removed or mitigated. We assume that the defender cannot access the trigger ∆ or the target label ˆ y .

Problem formulation. Using the 0 -1 loss [50, 60], the classification risk R cl and the backdoor risk R bd with respect to classifier h θ on D cl can be defined as:

̸

where I is the indicator function and | · | denotes the cardinality of a set.

In (2), the classification risk concerns whether a clean sample is correctly classified, while the backdoor risk measures the risk of classifying a poisoned sample from D -ˆ y to target class ˆ y .

In this paper, we consider the following problem for purifying a poisoned model:

where λ is a hyper-parameter to control the tradeoff between classification risk and backdoor risk.

However, directly solving (3) is impractical due to the lack of the trigger ∆ and target label ˆ y . Therefore, a natural choice is to replace R bd with a surrogate risk irrelevant to ∆ and ˆ y .

3.2 Connection between backdoor risk and adversarial risk

To tackle the above problem, we propose a novel upper bound of backdoor risk which can be relaxed to a surrogate of R bd that is independent of ∆ and ˆ y . Before that, we first decompose the adversarial risk and review the connection between backdoor risk and adversarial risk.

̸

Adversarial risk. Given a sample x and an adversarial perturbation ϵ ∈ S , an adversarial example ˜ x ϵ can be generated by ˜ x ϵ = x + ϵ , where S is the set of perturbations. We define the set of adversarial example-label pair to h θ generated by perturbation ϵ and dataset D cl by A ϵ , θ = { (˜ x ϵ , y ) | h θ (˜ x ϵ ) = y, ( x , y ) ∈ D cl } and abbreviate (˜ x ϵ , y ) ∈ A ϵ , θ by ˜ x ϵ ∈ A ϵ , θ . Then, the vanilla adversarial risk for h θ on D -ˆ y can be defined as

To bridge the adversarial examples and poison samples, we provide the following assumption:

Assumption 1. Assume that g ( x ; ∆) -x ∈ S for ∀ x ∈ D cl .

Assumption 1 ensures that there exists ϵ ∈ S such that x + ϵ = g ( x ; ∆) . Note that the analysis and the proposed method are independent of any specific types of perturbation . More discussions about the choice of S and its influence on the proposed method are discussed in Appendix B.4.

Then, we reach the first upper bound of backdoor risk:

Proposition 1. Under Assumption 1, R adv serves as an upper bound of R bd , i.e. , R bd ≤ R adv .

Remark. Although R adv seems to be a promising surrogate for R bd , it neglects some essential connections between adversarial examples and poisoned samples, resulting in a significant gap between adversarial risk and backdoor risk, as shown in Figure 1. Such a gap also leads to a fluctuating learning curve for backdoor risk and a significant drop in accuracy in the backdoor mitigation process by adversarial training. Therefore, we seek to construct a tighter upper bound for R bd . A key insight is that not all adversarial examples contribute to backdoor mitigation.

0

20

40

60

80

100

0

20

40

60

80

100

Percentage

Risk Gap:96.41%

Adversarial Risk

Epochs

h θ into three types, as shown in Table 1.

To further identify AEs important for mitigating backdoors, we leverage the information from the poisoned model to categorize the adversarial examples to

̸

Type

Description

Definition

I

Mislead

h

θ

and

h

θ

bd

to the same class

h

θ

bd

(˜

x

ϵ

) =

h

θ

(˜

x

ϵ

)

=

y

II

Mislead

h

θ

, but not mislead

h

θ

bd

h

θ

bd

(˜

x

ϵ

)

=

h

θ

(˜

x

ϵ

)

, h

θ

bd

(˜

x

ϵ

) =

y

III

Mislead

h

θ

and

h

θ

bd

to different classes

h

θ

bd

(˜

x

ϵ

)

=

h

θ

(˜

x

ϵ

)

, h

θ

bd

(˜

x

ϵ

)

=

y, h

θ

(˜

x

ϵ

)

=

y

̸

̸

̸

Figure 2: A schematic of the relationship between adversarial examples, shared adversarial examples (SAEs, Type I) and poisoned samples. The adversarial examples for h θ bd and h θ are shown in the blue and green solid ellipses, respectively. The poisoned samples are in the black dashed circle. Assume that h θ bd and h θ have 100% backdoor attack success rates in the left, such that all poisoned samples are contained in SAEs. Thus, by reducing the shared adversarial examples between h θ bd and h θ (from left to right), the backdoor risk can be mitigated in h θ .

Furthermore, we refer adversarial examples of Type I as the shared adversarial examples between h θ and h θ bd , as defined below:

̸

Definition 1 (Shared Adversarial Example) . Given two classifiers h θ 1 and h θ 2 , an adversarial example ˜ x ϵ is shared between h θ 1 and h θ 2 if and only if h θ 1 (˜ x ϵ ) = h θ 2 (˜ x ϵ ) = y .

Let D s be the subset of samples in D -ˆ y on which planting a trigger can successfully activate the backdoor in h θ bd , as summarized in Table 2. As illustrated in Figure 2, the following proposition reveals a deeper connection between adversarial examples and poisoned samples.

̸

Proposition 2. For ∀ ( x , y ) ∈ D s , the poisoned sample g ( x ; ∆) can attack h θ if and only if h θ ( g ( x ; ∆)) = h θ bd ( g ( x ; ∆)) = y. Furthermore, under Assumption 1, the poisoned sample g ( x ; ∆) is a shared adversarial example between h θ and h θ bd , if it can attack h θ .

̸

Then, the adversarial risk R adv can be decomposed to three components as below:

̸

Proposition 2 implies that the first component in (5) is essential for backdoor mitigation, as it captures the risk of shared adversarial examples between the poisoned model h θ bd and the fine-tuned model h θ , and the poisoned samples effective for h θ belong to SAEs. The second component is irrelevant

̸

̸

for backdoor mitigation, since adversarial examples of Type II and III are either not poisoned samples or ineffective to backdoor attack against h θ . The third component protects the samples that are originally resistant to triggers from being backdoor attacked in the mitigation process.

Thus, by removing the second component in (5) , we propose the following sub-adversarial risk

Compared to vanilla adversarial risk R adv , the proposed sub-adversarial risk R sub focuses on the shared adversarial examples for samples in D s while considering vanilla adversarial examples for samples in D -ˆ y \ D s . When D s = D -ˆ y , i.e. , h θ bd has backdoor risk 100% , the sub-adversarial risk measures the shared adversarial risk on D -ˆ y .

Then, we provide the following proposition to establish the relation between backdoor risk and (sub) adversarial risk:

Proposition 3. Under Assumption 1, for a classifier h θ , the following inequalities hold

Therefore, R sub is a tighter upper bound for R bd compared with R adv . After replacing R bd with R sub in (3), we reach the following optimization problem:

Due to the space limit, the proofs of the above propositions, as well as some detailed discussions and extensions will be provided in Appendix A.

3.3 Proposed method

Since the target label ˆ y is unavailable and 0-1 loss is non-differentiable, we first discuss the relaxation of R sub and then replace 0-1 loss with suitable surrogate loss functions in this section.

Relaxation. One limitation of R sub is the inaccessibility of D s , i.e. , the subset of samples in D -ˆ y on which planting a trigger can successfully activate the backdoor in h θ bd . Since a poisoned model usually has a high attack success rate (ASR), the first relaxation is replacing D s with D -ˆ y , i.e. ,

Since ˆ y is unavailable, we also replace D -ˆ y by D cl in the sub-adversarial risk. As g ( x ; ∆) is not malicious for classification if the ground truth label of x is ˆ y , this relaxation has negligible negative influence for mitigating backdoor risk. After the above two relaxations, we reach the following shared adversarial risk on D cl :

Due to the space limit, we postpone the detailed discussion of the above two relaxations ( e.g. , the relaxation gaps) in Appendix A.7.

By replacing R sub with R share in (7), we reach the following problem:

̸

By solving ( 8 ), we seek h θ that either classifies the adversarial examples correctly ( i.e. , h θ (˜ x ϵ ) = y ), or differently with h θ bd ( i.e. , h θ (˜ x ϵ ) = h θ bd (˜ x ϵ ) ), while maintaining a high accuracy on D cl .

̸

Approximation. We firstly need to approximate two indicators in (8), including: 1) I ( h θ ( x ) = y ) ; 2) I (˜ x ϵ ∈ A ϵ , θ , h θ (˜ x ϵ ) = h θ bd (˜ x ϵ )) . The former indicator is for the clean accuracy, so we use the widely used cross-entropy (CE) loss as the surrogate loss, i.e. , L cl ( x , y ; θ ) = CE ( p ( x ; θ ) , y ) . In terms of the latter indicator that measures the shared adversarial risk, different surrogate losses could be employed for different usages:

To generate shared adversarial examples, we first decompose the second indicator as below:

which covers two components standing for fooling h θ bd and h θ simultaneously, and keeping the same predicted label for h θ bd and h θ , respectively.

For the first component, we use CE on both h θ and h θ bd as a surrogate loss, i.e. ,

For the second component, we measure the distance between p (˜ x i, ϵ i ; θ ) and p (˜ x ϵ ; θ bd ) by JensenShannon divergence [11] and adopt the following surrogate loss:

To unlearn the shared adversarial examples, we adopt the following surrogate loss

̸

̸

where ˜ y = h θ bd (˜ x ϵ ) . By reducing L sar , the prediction of ˜ x ϵ by h θ , i.e. , h θ (˜ x ϵ ) is forced to differ from h θ bd (˜ x ϵ ) if h θ bd (˜ x ϵ ) = y , therefore, reducing the shared adversarial risk (SAR).

Overall objective. By combining all the above surrogate losses with the linear weighted summation mode, we propose the following bi-level objective:

where λ 1 , λ 2 , λ 3 , λ 4 ≥ 0 indicate trade-off weights.

Optimization algorithm. We propose an optimization algorithm called Shared Adversarial Unlearning (SAU, Algorithm 1), which solves (13) by alternatively updating θ , and ϵ ∗ . Specifically, the model parameter θ is updated using stochastic gradient descent [3], and the perturbation ϵ ∗ is generated by projected gradient descent [33].

Algorithm 1 Shared Adversarial Unlearning

4 Experiment

4.1 Experiment settings

Backdoor attack. We compare SAU with 7 popular state-of-the-art (SOTA) backdoor attacks, including BadNets [15], Blended backdoor attack (Blended) [7], input-aware dynamic backdoor attack (Input-Aware)[37], low frequency attack (LF) [58], sinusoidal signal backdoor attack (SIG) [2], sample-specific backdoor attack (SSBA) [30], and warping-based poisoned networks (WaNet) [38]. To make a fair and trustworthy comparison, we use the implementation and configuration from BackdoorBench [54], a comprehensive benchmark for backdoor evaluation. By default, the poisoning ratio is set to 10% in all attacks, and the 0 th label is set to be the target label. We evaluate all the attacks on 3 benchmark datasets, CIFAR-10 [24], Tiny ImageNet [26], and GTSRB [43] using two networks: PreAct-ResNet18 [18] and VGG19 [41]. Due to space constraints, the results for GTSRB and VGG19 are postponed to Appendix D. Note that for clean label attack SIG, the 10% poisoning ratio can only be implemented for CIFAR-10. More attack details are left in Appendix C.

Backdoor defense. We compare our method with 6 SOTA backdoor defense methods: ANP [55], Fine-pruning (FP) [31], NAD [28], NC [47], EP [64] and i-BAU [59]. By default, all the defense methods can access 5% benign training data. We follow the recommended configurations for SOTA defenses as in BackdoorBench [54]. For our method, we choose to generate the shared adversarial example with Projected Gradient Descent (PGD) [33] with L ∞ norm. For all experiments, we run PGD 5 steps with norm bound 0 . 2 and we set λ 1 = λ 2 = λ 4 = 1 and λ 3 = 0 . 01 . More details about defense settings and additional experiments can be found in Appendix C and D.

Evaluation metric. We use four metrics to evaluate the performance of each defense method: Accuracy on benign data ( ACC ), Attack Success Rate ( ASR ), Robust Accuracy ( R-ACC ) and Defense Effectiveness Rating ( DER ). R-ACC measures the proportion of the poisoned samples classified to their true label, and ASR measures the proportion of poisoned samples misclassified to the target label. Larger R-ACC and lower ASR indicate that the backdoor is effectively mitigated. Note that the samples for the target class are excluded when computing the ASR and R-ACC as done in BackdoorBench. DER ∈ [0 , 1] was proposed in [65] to evaluate the cost of ACC for reducing ASR. It is defined as follows:

where ∆ ASR denotes the drop in ASR after applying defense, and ∆ ACC represents the drop in ACC after applying defense.

Note : Higher ACC, lower ASR, higher R-ACC and higher DER represent better defense performance. We use boldface and underline to indicate the best and the second-best results among all defense methods, respectively, in later experimental results.

4.2 Main results

Effectiveness of SAU. To verify the effectiveness of SAU, we first summarize the experimental results on CIFAR-10 and Tiny ImageNet in Tables 3 and 4, respectively. As shown in Tables 3-4, SAU can mitigate backdoor for almost all attacks with a significantly lower average ASR. For the experiments on CIFAR-10, SAU achieves the top-2 lowest ASR in 4 of 7 attacks and very low ASR for the other 3 attacks. Similarly, SAU performs the lowest ASR in 3 attacks for Tiny ImageNet and negligible ASR in two of the other attacks. Notably, SAU fails to mitigate WaNet in Tiny ImageNet, although it can defend against WaNet on CIFAR-10. As a transformation-based attack that applies image transforms to construct poisoned samples, WaNet can generate triggers with a large L ∞ norm. Specifically, WaNet has average L ∞ trigger norm 0 . 348 on TinyImageNet and 0 . 172 on CIFAR-10. Note that for all experiments, we generate adversarial perturbation with L ∞ norm less than 0 . 2 . The large trigger norm of WaNet on Tiny ImageNet poses a challenge for AT-based methods such as i-BAU and the proposed one, which reveals an important weakness of such methods, i.e. , their performance may be degraded if the trigger is beyond the perturbation set. A promising approach for this challenge is to consider adversarial examples for the union of multiple perturbation types [34, 45] and we postpone more discussion to Appendix B.

As maintaining clean accuracy is also important for an effective backdoor defense method, we also compare SAU with other baselines in terms of ACC, R-ACC and DER in Table 3 and 4. As SAU

adopts adversarial examples to mitigate backdoors, it has negative effects on clean accuracy, resulting in a slightly lower clean accuracy compared to the best one. However, SAU achieves the best average R-ACC, which demonstrates its effectiveness in recovering the prediction of poisoned samples, i.e. , classifying the poisoned samples correctly. Besides, the best average DER indicates that SAU achieves a significantly better tradeoff between clean accuracy and backdoor risk.

Influence of poisoning ratio. To study the influence of the poisoning ratio on the effectiveness of SAU, we test SAU with varying poisoning ratios from 10% to 50%. As shown in Table 5, SAU is still able to achieve remarkable performance even if half of the data is poisoned.

4.3 Ablation Study

In this section, we conduct ablation study to investigate each component of SAU. Specifically, SAU is composed of two parts: generating shared adversarial examples according to the maximization objective in (13) denoted by A , and reducing shared adversarial risk according to the minimization objective in (13), denote by B . To study the effect of these parts, we replace them with the corresponding part in vanilla adversarial training, denoted by C and D , respectively. Then, we run each variant 20 epochs on Tiny ImageNet with PreAct-ResNet18 and a poisoning ratio 10% . Each experiment is repeated five times, and the error bar is only shown in Figure 3 for simplicity.

As shown in Table 6, the maximization objective for generating shared adversarial examples (component A ) is the key to reducing ASR, and component B can help to alleviate the hurt to clean accuracy. When replacing the component B with the minimization step in vanilla AT ( D ), it suffers from a drop of clean accuracy, although it can also work for mitigating backdoor effectively. Compared to vanilla AT, SAU achieves significantly lower ASR and higher ACC, with a much more stable learning

curve, as shown in Figure 3. We remark that the decrease of clean accuracy for A + D and vanilla AT demonstrates that imposing a model to learn overwhelming adversarial examples may severely hurt the clean accuracy, a phenomenon widely studied in the area of adversarial training [46, 60].

0

2

4

6

8

10

12

14

16

18

20

Epochs

0

20

40

60

80

100

Percentage

BadNets

Defense

A+B (Ours)

A+D

C+B

C+D (Vanilla AT)

Metric

ACC

ASR

0

2

4

6

8

10

12

14

16

18

20

Epochs

0

20

40

60

80

100

Blended

0

2

4

6

8

10

12

14

16

18

20

Epochs

0

20

40

60

80

100

LF

5 Conclusion

In conclusion, this paper proposes Shared Adversarial Unlearning, a method to defend against backdoor attacks in deep neural networks through adversarial training techniques. By developing a better understanding of the connection between adversarial examples and poisoned samples, we propose a novel upper bound for backdoor risk and a bi-level formulation for mitigating backdoor attacks in poisoned models. Our approach identifies shared adversarial examples and unlearns them to break the connection between the poisoned sample and the target label. We demonstrate the effectiveness of our proposed method through extensive experiments, showing that it achieves comparable to, and even superior, performance than six different state-of-the-art defense methods on seven SOTA backdoor attacks with different model structures and datasets. Our work provides a valuable contribution to the field of backdoor defense in deep neural networks, with potential applications in various domains.

Limitation and future work. One important direction for future work, and a current challenge, is the accessibility of clean samples. A valuable approach to this challenge is to consider the samples from other domains and the generated samples.

Structure of Appendix. The detailed proof, analysis, and extension of propositions are given in Appendix A. Some important discussions such as the computation cost, the scalability, and the generalization ability of the proposed method are provided in Appendix B. The details of the experiments are provided in Appendix C. Additional experiments on different settings such as model structures, poisoning ratios, and sizes of clean datasets are given in Appendix D.

Acknowledgments and Disclosure of Funding

This work is supported by the National Natural Science Foundation of China under grant No. 62076213, Shenzhen Science and Technology Program under grant No. RCYX20210609103057050, No. GXWD20201231105722002-20200901175001001, and No. ZDSYS20211021111415025, and No. JCYJ20210324120011032 and the CAAI-Huawei MindSpore Open Fund, and the Guangdong Provincial Key Laboratory of Big Data Computing, the Chinese University of Hong Kong, Shenzhen.

References

Insaf Adjabi, Abdeldjalil Ouahabi, Amir Benzaoui, and Abdelmalik Taleb-Ahmed. Past, present, and future of face recognition: A review. Electronics , 9(8):1188, 2020. 1

Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In ICIP 2019 , pages 101-105. IEEE, 2019. 2, 8, 9, 26, 27, 28, 30, 31, 32, 33, 34

Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. In NeurIPS 2007 , volume 20, 2007. 7

Shuwen Chai and Jinghui Chen. One-shot neural backdoor erasing via adversarial weight masking. In NeurIPS 2022 , 2022. 17

Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In Workshop on Artificial Intelligence Safety . CEUR-WS, 2019. 2

Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In NeurIPS 2022 , volume 35, pages 9727-9737, 2022. 3

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv e-prints , pages arXiv-1712, 2017. 2, 8, 9, 26, 28, 29, 30, 31, 32, 33

Ziyi Cheng, Baoyuan Wu, Zhenya Zhang, and Jianjun Zhao. Tat: Targeted backdoor attacks against visual object tracking. Pattern Recognition , 142:109629, 2023. 2

Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. In ICCV 2021 , pages 11946-11956, 2021. 2

Yinpeng Dong, Xiao Yang, Zhijie Deng, Tianyu Pang, Zihao Xiao, Hang Su, and Jun Zhu. Black-box detection of backdoor attacks with limited information and data. In ICCV 2021 , pages 16482-16491, 2021. 17

Bent Fuglede and Flemming Topsoe. Jensen-shannon divergence and hilbert space embedding. In ISIT 2004 , page 31. IEEE, 2004. 7

Kuofeng Gao, Jiawang Bai, Baoyuan Wu, Mengxi Ya, and Shu-Tao Xia. Imperceptible and robust backdoor attack in 3d point cloud. arXiv preprint arXiv:2208.08052 , 2022. 2

Yinghua Gao, Dongxian Wu, Jingfeng Zhang, Guanhao Gan, Shu-Tao Xia, Gang Niu, and Masashi Sugiyama. On the effectiveness of adversarial training against backdoor attacks. IEEE Transactions on Neural Networks and Learning Systems , 2023. 2, 22, 23, 24, 27

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR 2015 , 2015. 3

Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access , 7:47230-47244, 2019. 2, 8, 9, 23, 24, 26, 28, 29, 30, 31, 32, 33, 34

Junfeng Guo, Ang Li, and Cong Liu. AEVA: Black-box backdoor detection using adversarial extreme value analysis. In ICLR 2022 , 2022. 17

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR 2016 , pages 770-778, 2016. 1

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV 2016 , pages 630-645. Springer, 2016. 8

Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In ICLR 2022 . OpenReview.net, 2022. 3

Jinyuan Jia, Yupei Liu, Xiaoyu Cao, and Neil Zhenqiang Gong. Certified robustness of nearest neighbors against data poisoning and backdoor attacks. In AAAI 2022 , volume 36, pages 9575-9583, 2022. 22

Xiaojun Jia, Yong Zhang, Xingxing Wei, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. Prior-guided adversarial initialization for fast adversarial training. In ECCV 2022 , pages 567-584, 2022. 3

Xiaojun Jia, Yong Zhang, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. LAS-AT: adversarial training with learnable attack strategy. In CVPR 2022 , pages 13388-13398, 2022. 3

Xiaojun Jia, Yong Zhang, Baoyuan Wu, Jue Wang, and Xiaochun Cao. Boosting fast adversarial training with learnable adversarial initialization. IEEE Transactions on Image Processing , 31: 4417-4430, 2022. 3

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 8

Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In ICLR 2021 , 2021. 23

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015. 4, 8

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In NeurIPS 2021 , pages 14900-14912, 2021. 3

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In ICLR 2021 . OpenReview.net, 2021. 3, 8, 9, 21, 25, 26, 27, 28, 29, 30, 31, 33

Yige Li, Xixiang Lyu, Xingjun Ma, Nodens Koren, Lingjuan Lyu, Bo Li, and Yu-Gang Jiang. Reconstructive neuron pruning for backdoor defense. In ICML 2023 , 2023. 32, 33, 34

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV 2021 , pages 16463-16472, 2021. 2, 8, 9, 23, 24, 27, 28, 29, 30, 31, 32, 33

Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In RAID 2018 , pages 273-294. Springer, 2018. 3, 8, 9, 21, 25, 27, 28, 29, 30, 31, 33

Liangkai Liu, Sidi Lu, Ren Zhong, Baofu Wu, Yongtao Yao, Qingyang Zhang, and Weisong Shi. Computing systems for autonomous driving: State of the art and challenges. IEEE Internet of Things Journal , 8(8):6469-6486, 2020. 1

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR 2018 , 2018. 1, 3, 7, 8, 27

Pratyush Maini, Xinyun Chen, Bo Li, and Dawn Song. Perturbation type categorization for multiple adversarial perturbation robustness. In UAI 2022 , pages 1317-1327. PMLR, 2022. 8

Bingxu Mu, Zhenxing Niu, Le Wang, Xue Wang, Qiguang Miao, Rong Jin, and Gang Hua. Progressive backdoor erasing via connecting backdoor and adversarial attacks. In CVPR 2023 , pages 20495-20503, 2023. 22, 27

Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. In ICLR 2021 , 2021. 25

Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In NeurIPS 2020 , 2020. 8, 9, 26, 28, 29, 30, 31

Tuan Anh Nguyen and Anh Tuan Tran. Wanet - imperceptible warping-based backdoor attack. In ICLR 2021 . OpenReview.net, 2021. 2, 8, 9, 23, 24, 27, 28, 29, 30, 31, 32, 33

Xiangyu Qi, Tinghao Xie, Jiachen T Wang, Tong Wu, Saeed Mahloujifar, and Prateek Mittal. Towards a proactive { ML } approach for detecting backdoor poison samples. In 32nd USENIX Security Symposium (USENIX Security 23) , pages 1685-1702, 2023. 2

Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In NeurIPS 2018 , volume 31, 2018. 2

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR 2015 , 2015. 8, 28

Hossein Souri, Liam Fowl, Rama Chellappa, Micah Goldblum, and Tom Goldstein. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. In NeurIPS 2022 , volume 35, pages 19165-19178, 2022. 2

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In IJCNN 2011 , pages 1453-1460. IEEE, 2011. 8

J-Donald Tournier, Robert Smith, David Raffelt, Rami Tabbara, Thijs Dhollander, Maximilian Pietsch, Daan Christiaens, Ben Jeurissen, Chun-Hung Yeh, and Alan Connelly. Mrtrix3: A fast, flexible and open software framework for medical image processing and visualisation. Neuroimage , 202:116137, 2019. 1

Florian Tramer and Dan Boneh. Adversarial training and robustness for multiple perturbations. In NeurIPS 2019 , volume 32, pages 5858-5868, 2019. 8

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR 2019 , 2019. 10

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy , pages 707-723. IEEE, 2019. 8, 9, 21, 25, 27, 28, 29, 30, 31, 33

Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, and Zhangyang Wang. Removing batch normalization boosts adversarial training. In ICML 2022 , pages 23433-23445. PMLR, 2022. 28

Ruotong Wang, Hongrui Chen, Zihao Zhu, Li Liu, Yong Zhang, Yanbo Fan, and Baoyuan Wu. Robust backdoor attack with visible, semantic, sample-specific, and compatible triggers. arXiv preprint arXiv:2306.00816 , 2023. 2

Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR 2020 , 2020. 4

Maurice Weber, Xiaojun Xu, Bojan Karlaš, Ce Zhang, and Bo Li. Rab: Provable robustness against backdoor attacks. In 2023 IEEE Symposium on Security and Privacy (SP) , pages 1311-1328. IEEE, 2023. 21

Cheng-Hsin Weng, Yan-Ting Lee, and Shan-Hung Brandon Wu. On the trade-off between adversarial and backdoor robustness. In NeurIPS 2020 , volume 33, pages 11973-11983, 2020. 1, 22

Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In ICLR 2020 , 2020. 3

Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. In NeurIPS Datasets and Benchmarks Track 2022 , 2022. 8, 26

Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurIPS 2021 , pages 16913-16925, 2021. 3, 8, 9, 21, 24, 25, 26, 27, 29, 30, 31, 32, 33

Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In ICLR 2018 , 2018. 23, 24

Cihang Xie and Alan L. Yuille. Intriguing properties of adversarial training at scale. In ICLR 2020 , 2020. 28

Yi Zeng, Won Park, Z. Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV 2021 , pages 16453-16461. IEEE, 2021. 2, 8, 9, 26, 28, 29, 30, 31, 32, 33

Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In ICLR 2022 . OpenReview.net, 2022. 2, 3, 8, 9, 17, 21, 22, 25, 27, 28, 29, 30, 31, 33

Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML 2019 , pages 7472-7482. PMLR, 2019. 4, 10

Yuhao Zhang, Aws Albarghouthi, and Loris D'Antoni. Bagflip: A certified defense against data poisoning. In NeurIPS 2022 , volume 35, pages 31474-31483, 2022. 21

Haizhong Zheng, Ziqi Zhang, Juncheng Gu, Honglak Lee, and Atul Prakash. Efficient adversarial training with transferable adversarial examples. In CVPR 2020 , pages 1178-1187, 2020. 3

Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. In European Conference on Computer Vision , pages 175-191. Springer, 2022. 3

Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Pre-activation distributions expose backdoor neurons. In NeurIPS 2022 , 2022. 3, 8, 9, 21, 27, 29, 30, 31, 33

Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In ICCV 2023 , pages 4466-4477, 2023. 3, 8

Mingli Zhu, Shaokui Wei, Hongyuan Zha, and Baoyuan Wu. Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features. arXiv preprint arXiv:2306.16697 , 2023. 3

Zihao Zhu, Mingda Zhang, Shaokui Wei, Li Shen, Yanbo Fan Fan, and Baoyuan Wu. Boosting backdoor attack with a learnable poisoning sample selection strategy. arXiv preprint arXiv:2307.07328 , 2023. 2

Zihao Zhu, Mingda Zhang, Shaokui Wei, Bingzhe Wu, and Baoyuan Wu. Vdc: Versatile data cleanser for detecting dirty samples via visual-linguistic inconsistency. arXiv preprint arXiv:2309.16211 , 2023. 2

A Proofs and extension

In this section, we provide detailed proof and analysis of the propositions, methods, and extensions.

A.1 Proof of Proposition 1

Proof. Under Assumption 1, for any sample x ∈ D -ˆ y , we have

since g ( x ; ∆) -x ∈ S .

Therefore, we have

A.2 Proof of Proposition 2

Proof. By the definition of backdoor attack, a poisoned sample g ( x ; ∆) is said to attack h θ successfully if and only if h θ ( g ( x ; ∆)) = ˆ y .

̸

As h θ bd ( g ( x ; ∆)) = ˆ y for all x ∈ D s (definition of D s ), we have h θ ( g ( x ; ∆)) = h θ bd ( g ( x ; ∆)) = y if and only if g ( x ; ∆) can attack h θ successfully, for all x ∈ D s .

̸

Furthermore, under Assumption 1, g ( x ; ∆) is an shared adversarial example of h θ and h θ bd if h θ ( g ( x ; ∆)) = h θ bd ( g ( x ; ∆)) = y by the Definition 1.

A.3 Proof of Proposition 3

Proof. By Proposition 1 and the decomposition of R adv in Section 3.2, we have

Therefore, we discuss the relation between R ( h ) and R ( h ) now.

bd θ sub θ Recall that

Then, under Assumption 1, we have

For x ∈ D s , Proposition 2 yields that

For x ∈ D -ˆ y \ D s , similar as Proposition 1, we have

Therefore, we have

So, under Assumption 1, the following inequalities holds:

A.4 Extension to Universal Adversarial Perturbation

Universal Adversarial Perturbation (UAP) has achieved remarkable performance in previous studies [59, 4], especially for defending against backdoor attacks with fixed additive triggers. Now, we extend R sub to UAP and show that shared adversarial examples also benefit adversarial training with UAP.

We denote the Universal Adversarial Risk by

To bridge UAP and backdoor trigger, we first provide the following assumption:

Assumption 2. Assume that the g ( x , ∆) = x +∆ for all x ∈ D cl .

Assumption 2 says that the poisoned samples are planted with a fixed additive trigger.

Then, under Assumption 2 and Assumption 1, we have

̸

by the fact that ∆ ∈ S .

For clarity, we replace ˜ x ϵ ∈ A ϵ , θ by h θ ( x + ϵ ) = y and ˜ x ϵ by x + ϵ which makes the rest derivation easier to follow.

Then, similar as R sub , we define the universal sub-adversarial risk R usub as below:

̸

̸

which considers shared universal adversarial risk on D s and vanilla universal adversarial risk on D -ˆ y \ D s .

By the definition of R usub , we can easily find R usub ≤ R uadv .

Furthermore, we have

Therefore, we reach the following proposition

Proposition 4. Under Assumption 2 and Assumption 1, for a classifier h θ , the following inequalities hold

Therefore, the shared adversarial examples also benefit adversarial training with UAP.

A.5 Extension to Targeted Adversarial Training

In the previous sections, we assume that the defender has no access to either the trigger or the target label. Since there are already some works for backdoor trigger detection Guo et al. [16], Dong et al. [10], we discuss the case where the defender can estimate the target label ˆ y and show that shared adversarial examples can also be beneficial for adversarial training with target label.

̸

Given a target label ˆ y , we define the Targeted Adversarial Risk as

By the definition of ˆ y , we have R bd ( h θ ) ≤ R tadv ( h θ ) under Assumption 1.

Then, we have

̸

Then, similar as R sub , we define the targeted sub-adversarial risk R tsub by removing the second component in R tadv ( h θ ) :

which considers shared targeted adversarial risk on D s and vanilla targeted adversarial risk on D -ˆ y \ D s .

Since the removed component is irrelevant to backdoor risk, we have R bd ≤ R tsub under Assumption 1.

Therefore, we reach the following proposition

Proof. Under Assumption 2 and Assumption 1, for a classifier h θ , the following inequalities hold

Therefore, the shared adversarial examples also benefit adversarial training with target label.

Moreover, since R tsub ( h θ ) only consider shared adversarial example with target label ˆ y , the following proposition holds,

Proof. Under Assumption 2 and Assumption 1, for a classifier h θ , the following inequalities hold

The above proposition shows that R tsub ( h θ ) is a tighter bound of R bd ( h θ ) than R sub ( h θ ) .

A.6 Extension to Multi-target/Multi-trigger

In previous sections, we assume that there is only one trigger and one target for all samples. Now, we consider to a more general case where a set of triggers V and a set of target labels ˆ Y are given. This is a more challenging case since there may be multiple triggers and/or multiple targets for one sample.

We define T to be a mapping from the sample to the corresponding set of feasible trigger-target pairs. Therefore, given a sample x , the set of all feasible trigger-target pairs is given by T ( x ) . Then, Assumption 1 is extended to all triggers accordingly, i.e. , g ( x , ∆) -x ∈ S , ∀ (∆ , ˆ y ) ∈ T ( x ) .

̸

In this setting, we define that a classifier h θ is said to be attacked by a poisoned sample generated by x if there exists (∆ , ˆ y ) ∈ T ( x ) such that h θ ( g ( x , ∆)) = ˆ y = y .

Then, the (maximum) backdoor risk on D cl is defined to be

Similarly, the adversarial risk on D cl is defined as

̸

̸

We define D s = { x ∈ D cl : h θ bd ( g ( x , ∆)) = ˆ y = y, ∀ (∆ , ˆ y ) ∈ T ( x ) } to be the set of samples on which planting any feasible trigger can activate the corresponding backdoor. Then, the corresponding sub-adversarial risk is defined as

̸

̸

which considers shared adversarial risk on D s and vanilla targeted adversarial risk on D cl \ D s . By definition, we have R sub ≤ R adv

Moreover, we have

̸

Therefore, for a backdoor attack with multiple targets and/or multiple triggers, the proposed subadversarial risk R sub is still a tighter upper bound of R bd compared to R adv .

A.7 Relaxation gap

To build a practical objective for defending against a backdoor attack, two relaxations are applied to R sub . Here, we discuss the relaxation gap in the relaxing process:

· Replace D s by D -ˆ y .

Since D s is not accessible, the first relaxation is to replace D s by D -ˆ y and consider shared adversarial examples on all samples in D -ˆ y . Denote R 1 ( h θ ) to be the relaxed risk after replacing

̸

̸

̸

̸

̸

̸

̸

̸

D s by D -ˆ y . Since R sub can be decomposed as follows:

̸

We can find that the relaxation gap is

̸

And the relaxation gap GAP 1 is bounded by 1 -R bd ( h θ bd ) . Since a poisoned model h θ bd usually has a high ASR for the backdoor attack, the gap GAP 1 is usually negligible.

Remark: We remark that the experiment results show that our method also works well for the poisoned models with low ASR. See Section D for more details.

· Replace D -ˆ y by D cl .

The second relaxation is to replace D -ˆ y by D cl when ˆ y is not accessible. Let D ˆ y = D cl \ D -ˆ y . Then, the risk after the second relaxation is:

where the first component is R 1 scaled by |D -ˆ y | N and the second component is the vanilla adversarial risk on D ˆ y , scaled by |D ˆ y | N . So, the relaxation gap for the second relaxation is

We can find that the generalization gap is related to the |D ˆ y | N , i.e. , the portion of samples whose labels are target labels, and negligible when |D ˆ y | N is small.

Remark. For using R share ( h θ ) to mitigate the backdoor, the influence of |D -ˆ y | N can be eliminated by altering the trade-off parameter in Problem (8). Also, adding triggers to samples labeled ˆ y is harmless during the testing/deployment phase. So, the vanilla adversarial risk on D ˆ y has a negligible negative impact on defending the backdoor. Therefore, replacing D -ˆ y with D cl has little negative impact on mitigating the backdoor.

B Discussion

In this section, we provide a comprehensive discussion of the proposed method, including its efficiency, comparison to other methods, choice of perturbation, generalization ability to attack with excessive-magnitude triggers, performance on the generated dataset, and resistance to defenseaware/adaptive attacks.

B.1 Efficiency and Computational Cost

Here, we would like to emphasize that SAU is an efficient and effective defense method that can mitigate backdoor attacks with acceptable overhead for the following reasons:

SAU is a post-processing method that fine-tunes the backdoor model on a clean dataset with a small size. For instance, we can use only 500 samples from CIFAR-10 to fine-tune the model by SAU (see Appendix D.6 for more details), which takes 1.52 seconds/epoch to fine-tune PreAct-ResNet18 with RTX 4090. Therefore, it can be executed efficiently without requiring a large amount of data or computation resources.

SAU only takes a few epochs to take effect, which further reduces the computational cost of executing it in practice. As shown in Appendix D.4, SAU can achieve a low Attack Success Rate and a high Accuracy in a few epochs.

To further demonstrate the efficiency of SAU, we compare its average runtime with other stateof-the-art defense methods against attacks. We use the same experimental setting as in Section 4, where the poisoning ratio is 10% and the backbone model is PreAct-ResNet18. All experiments are conducted on a server with GPU RTX 4090 and CPU AMD EPYC 7543 32-Core Processor. As different methods adopt different learning paradigms for backdoor mitigation, we measure the average runtime for each defense method to take effect on CIFAR-10 and Tiny ImageNet (ASR < 5% and ACC > 85% on CIFAR-10, or ASR < 5% , ACC > 45% on Tiny ImageNet). Note that if a defend method cannot reach the criteria for "Take effect", the maximum runtime is reported. The results are summarized in the following table.

From Table 7, SAU is faster than most of the existing methods, except for NAD and EP, which shows that SAU is an efficient and effective defense method.

B.2 Comparison with Certified Robustness methods

In backdoor defense, methods can be categorized into two classes: empirical methods which aim to develop effective and efficient methods for backdoor defense, and certified methods which aim to build a provable framework for backdoor robustness. As our method is an empirical method , we would like to point out that our method is different from the existing works on certified robustness in several aspects, which makes them incomparable or impractical to apply to our setting. Specifically:

Efficiency and scalability. Certified methods usually have very high computation costs and are infeasible to apply to large-scale deep neural networks, while our method is efficient and scalable. For example, the representative methods RAB [51] and BagFlip [61] rely on voting over 1000 smoothed models for prediction and are only feasible for some simple neural networks and simple classification problems. To apply these methods to our setting, we would need to train 1000 PreAct-ResNet models on CIFAR-10 from scratch, which would take over 600 hours (25 days) and 620GB of space for a single attack on a server with one RTX 3090.

Applicability to model architecture. Many certified methods are restricted to some simple or classical models, while our method is general and applicable to any deep neural network

architecture. For example, the method in [20] is designed for only k-Nearest Neighbors (kNN) and radius Nearest Neighbors (rNN) models with some feature extractor and cannot be applied to modern deep neural networks.

Difference in the threat model. Most certified methods focus on the threat model where the training dataset is accessible to defenders while our method focuses on the threat model where only a poisoned model and a few clean samples are given.

Therefore, we believe that our method is more practical and effective for defending against backdoor attacks in deep neural networks than the existing works on certified robustness from the perspective of efficiency, scalability, and model structures.

B.3 Comparison with other AT based methods

Here, we compare the proposed method SAU with other representative Adversarial Training based methods from the following aspects:

Threat Model: whether the training dataset is accessible and whether the extra clean dataset is accessible.

Types of Perturbation: whether universal adversarial perturbation is supported.

Types of Adversarial Attack: whether Targeted Adversarial Attack is supported.

The answer "Yes" and "No" are indicated by √ and × , respectively. The "Optional" means the method supports but is not limited to a specific answer. We summarize the result in Table 8.

We remark that two methods, i.e. , PBE and AFT are proposed in Mu et al. [35] for different threat models, where PBE is designed to filter a clean dataset from a training dataset, and AFT is to perform AT on the given clean dataset. Therefore, AFT is equivalent to the Vanilla AT in the Figure 1 and Section 4.3.

As summarized in Table 8, the current AT-based methods can be roughly categorized into two types based on their threat model: in-training (given training dataset) and post-training (given poisoned model and extra clean dataset), and the difference in threat model make methods from different categories incomparable.

1. Comparison with In-training AT-based methods. Although SAU is not comparable with the in-training AT-based method due to the underlying assumption of the threat model, we would like to adopt different threat models for different methods to highlight the advantages of SAU compared to the in-training AT-based methods. Therefore, we compare SAU with the latest SOTA in-processing AT-based method, i.e. , Composite AT (CAT) [13].

We first highlight the following differences between CAT and SAU:

SAU is theoretically motivated by the relationship between adversarial robustness and backdoor risk, while CAT is empirically driven by some experimental observations. Therefore, our method has a theoretical guarantee for the performance against various backdoor attacks, while CAT does not.

SAU fine-tunes a backdoored model with a small clean dataset, while CAT trains a backdoorfree model from scratch on a poisoned dataset. Therefore, SAU is more efficient than CAT.

Then, we conduct experiments on defending against backdoor attacks and summarize the results in Table 9 and 10.

The experimental results show that our method is superior to CAT, as evidenced by the following aspects:

SAU achieves significantly higher accuracy (ACC) and lower attack success rate (ASR) than CAT on both poisoning ratios.

SAU is more robust to different poisoning ratios than CAT. In contrast, CAT fails on a high poisoning ratio (10%) (also observed in Table XI of Gao et al. [13]).

2. Comparison with Post-training AT-based methods. The most related methods to SAU are the post-processing AT-based method including I-BAU and AFT, and we highlight the following differences between them:

I-BAU is proposed under the assumption that the same trigger are used for all poisoned samples which may not be true for more advanced attacks, while SAU is shown to be effective for attacks with multi-trigger and sample-specific trigger theoretically and empirically.

AFT, which is to perform AT on clean dataset, is motivated by some empirical observation, while SAU is motivated by a novel analysis on the relation between adversarial examples and poisoned samples.

Besides, the experimental comparison between SAU and I-BAU in Section 4.2 and the experimental between SAU and AFT (Vanilla AT on Clean Dataset) in Section 4.3 show that SAU outperforms I-BAU and AFT by a large margin.

B.4 Choice of perturbation and generalization ability of SAU

In this section, we discuss the choice of perturbation set and show that the proposed method, i.e. , SAU, can generalize to attacks with excessive-magnitude triggers.

1. Type of adversarial perturbation (AP) and perturbation set. The theoretical analysis and the proposed method SAU is independent of the AP type and AP set (not only L p ball). For sample space X , define the set G = { g ( x, ∆) -x | x ∈ X} and the general perturbation set S = { T ( x, ϵ ) -x | x ∈ X} for adversarial attack T with learnable parameter ϵ ∈ Ω such that an adversarial example can be generated by ˜ x ϵ = T ( x, ϵ ) . Then, the theoretical analysis and proposed method can be applied if G ⊆ S , where the AP can be any type, such as additive AP, Spatially Transformed AP [56], or Perceptual AP [25], etc.

2. Effect of perturbation set S . The key of SAU is to consider Shared Adversarial Example (SAE) as surrogates for poisoned samples. Given models, the set of SAE is determined by S : larger S leads to larger set of SAE, and vice versa Therefore, the perturbation set S plays an important role in the effectiveness of SAU. Specifically, adopting a larger S can cover unknown triggers but also increases the chance of picking SAE which is irrelevant to poisoned samples, and may weaken the defense performance. Meanwhile, some poisoned samples may be beyond the scope of SAE if a smaller S is adopted.

3. Generalization to excessive-magnitude trigger. In practice, we find that the proposed method SAU is effective for various attacks, even if Assumption 1 does not hold. For example, SAU with L ∞ norm bound 0.2 can still effectively mitigate SSBA attack whose average L ∞ trigger norm is much larger than 0.2, revealing the generalization of SAU to attacks with excessive-magnitude trigger.

To investigate the SAU's generalization ability, we conduct experiments on BadNets with various strengths. Specifically, for BadNets with a 3x3 patched trigger, we alter the pixel value of the trigger from 0.2 to 1.0.

Table 12 shows the results for defending various BadNets attacks by SAU following the setting in Section 4.2, from which we can find that SAU can consistently mitigate the backdoor even when the trigger is beyond the perturbation set . One reasonable explanation is that although the poisoned samples are not included in SAEs, there exists SAE that has a similar effect as poisoned samples as shown in Figure 4, resulting in strong generalization ability of SAU.

4. Effect of perturbation type. To explore the effect of perturbation types, we extend SAU to incorporate two other perturbation types and denote the resulting methods by Spatial SAU and Ensemble SAU , respectively. Specifically, for spatial SAU, we employ spatial adversarial attack [56] to generate spatial SAEs by spatial transformation, following the same setting as [13]. For Ensemble SAU, the perturbation set S is composed of a collection of subsets, including L 1 ball (bound 500), L 2 ball (bound 10), L ∞ ball (bound 0.2), and the spatial AP set. Then, for each batch, Ensemble SAU randomly chooses one subset of perturbation to generate the SAEs. We compare SAU , Spatial SAU and Ensemble SAU , following the setting in Section 4.

Poisoned Sample

Shared Adversarial Example

Non-Shared Adversarial Example

38.75

80.55

From Table 13, we compare SAU, Spatial SAU and Ensemble SAU, following the setting in Section 4. From the table, we find that Spatial SAU works well on CIFAR-10 dataset but fails to defend BadNets and SSBA attacks on Tiny ImageNet. For WaNet, spatial SAU can effectively defend it on both datasets. We remark that the Ensemble SAU achieves the lowest ASR in all experiments . However, as Ensemble SAU has larger perturbation set, its accuracy is lower than SAU by 1 2%.

In summary, our theoretical analysis and the proposed method SAU are flexible to the perturbation set, and SAU can be generalized to attack with excessive trigger norms. Moreover, the Ensemble SAU serves as a stronger variant of SAU and can be effective for various attacks.

B.5 SAU with generated datasets

Like most other post-processing methods [31, 55, 47, 28, 59], SAU assumes that a small clean dataset is available for conducting defense, which can be obtained by ways such as buying from a trustworthy data supplier, using modern generative models, collecting from the internet, or applying some data cleansing techniques.

In this section, we explore a more practical case for defense, i.e. , using the generated dataset, to evaluate our method. Specifically, we conduct experiments on synthetic datasets and compare SAU with NAD. We test both methods with a synthetic dataset CIFAR-5m [36] which provides generated CIFAR-10-like images whose distribution is close but not identical to CIFAR-10, and can be regarded as OOD data. We build mixed datasets by randomly picking samples from CIFAR-10 and CIFAR-5m with a mixed ratio , and evaluate both methods on the mixed datasets. A larger mixed ratio indicates more synthetic data in the mixed dataset. The experiment results are summarized in the following tables.

From Table 14, we can find that all methods can work on the mixed dataset but as the mixed ratio increase, the accuracy (ACC) may decrease due to the distribution difference between the two datasets. It's notable that the model distillation-based method NAD is influenced by the quality of the dataset most, with a reduction of 4.91% in ACC, while SAU is quite robust to the data quality. Another interesting phenomenon is that the ASR of ANP decreases when more synthetic data is used, which shows that the distribution of data may influence the performance of backdoor defense.

All those observations motivate us to conduct research on the relationship between data quality and backdoor defense in the future.

B.6 Resistance to defense-aware/adaptive Attacks

When the attacker is aware of the defender's adversarial training strategy, the attacker can train the backdoored model using Adversarial Training (AT) on the poisoned dataset, such that the model can be resistant to adversarial attack, hindering the generation of shared adversarial examples.

To evaluate the performance of SAU against such defense-aware backdoor attacks, we conduct experiments on the CIFAR-10 dataset following the settings in Section 4.2. For AT on the training dataset, we use PGD-10 attack with L ∞ norm 8/255, which is commonly used in adversarial training literature.

From Table 15, we can find that SAU can achieve an average ACC of 78.3% and ASR of 2.44%, showing its resistance to such adaptive attacks.

C Experiment details

In this section, we provide the experiment details. In our experiments, all baselines and settings are adapted from BackdoorBench [54], and we present the experiment details in this section.

C.1 Attack details

BadNets [15] is one of the earliest works for backdoor learning, which inserts a small patch of fixed pattern to replace some pixels in the image. We use the default setting in BackdoorBench.

Blended backdoor attack (Blended) [7] uses an alpha-blending strategy to fuse images with fixed patterns. We set α = 0 . 2 as the default in BackdoorBench. We remark that since such a large α produces visual-perceptible change to clean samples, the Blended Attack in this setting is very challenging for all defense methods.

Input-aware dynamic backdoor attack (Input-Aware)[37] is a training-controllable attack that learns a trigger generator to produce the sample-specific trigger in the training process of the model. We use the default setting in BackdoorBench.

Low-frequency attack (LF) [58] uses smoothed trigger by filtering high-frequency artifacts from a UAP. We use the default setting in BackdoorBench.

Sinusoidal signal backdoor attack (SIG) [2] is a clean-label attack that uses a sinusoidal signal as the trigger to perturb the clean images in the target label. We use the default setting in BackdoorBench.

Sample-specific backdoor attack (SSBA) [30] uses an auto-encoder to fuse a trigger into clean samples and generate poisoned samples. We use the default setting in BackdoorBench.

Warping-based poisoned networks (WaNet) [38] is also a training-controllable attack that uses a warping function to perturb the clean samples to construct the poisoned samples. We use the default setting in BackdoorBench.

C.2 Defense details

The details for each defense used in our experiment/discussion are summarized below:

ANP [55] is a pruning-based method that prunes neurons sensitive to weight perturbation. Note that in BackdoorBench, ANP can select its pruning threshold by grid searching on the test dataset (default way in BackdoorBench) or a given constant threshold. To produce a fair comparison with other baselines, we use a constant threshold for ANP and set the pruning threshold to 0 . 4 , which is found to produce better results than the recommended constant threshold of 0 . 2 in BackdoorBench. All other settings are the same as the default setting in BackdoorBench.

FP [31] is a pruning-based method that prunes neurons according to their activations and then fine-tunes the model to keep clean accuracy. We use the default setting in BackdoorBench.

NAD [28] uses Attention Distillation to mitigate backdoors. We use the default setting in BackdoorBench.

NC [47] first optimizes a possible trigger to detect whether the model is backdoored. Then, if the model is detected as a backdoored model, it mitigates the backdoor by unlearning the optimized trigger. We use the default setting in BackdoorBench. We remark that if the model is detected as a clean model, NC just returns the model without any changes.

EP [64] uses the distribution of activation to detect the backdoor neurons and prunes them to mitigate the backdoor. We use the default setting in BackdoorBench.

i-BAU [59] uses adversarial training with UAP and hyper-gradient to mitigate the backdoor. We use the default setting in BackdoorBench.

SAU (Ours) uses PGD to generate shared adversarial examples and unlearns them to mitigate the backdoor. For all experiments, we set λ 1 = λ 2 = λ 4 = 1 and λ 3 = 0 . 01 in (13) . Then, we run PGD 5 steps with L ∞ norm bound 0 . 2 and a large step size 0 . 2 to accelerate the inner maximization in ( 8 ). For unlearning step (fine-tuning on D cl and SAEs), we use the same setting as i-BAU and fine-tuning (FT) in BackdoorBench. We run SAU 100 epochs in CIFAR-10 and GTSRB. In Tiny ImageNet, we run SAU 20 epochsNote that the number of epochs used in our experiment largely exceeds the necessary epochs for SAU to take effect and can be further reduced to save computational costs.

Vanilla Adversarial Training (vanilla AT) [33] use PGD to generate vanilla adversarial examples and unlearn them to mitigate the backdoor. In all experiments, vanilla AT uses the same setting as SAU, including the step size, norm bound, optimizer, and learning rate, except for the experiment in Figure 1 , where we train vanilla AT 100 epochs on Blended Attack with PreAct-ResNet18 and Tiny ImageNet to show its long-term performance.

Adversarial Fine-tuning (AFT) [35] performs adversarial training on clean dataset, which is equivalent to vanilla AT.

Composite Adversarial Training (Composite AT, CAT) [13] use perturbation from different types to defend against backdoor attacks. We request the official implementation of CAT from the authors and use the recommended setting in the original paper.

Progressive Backdoor Erasing (PBE) [35] uses the adversarial attack to progressively filter the clean dataset from the training dataset and remove the backdoor. Since the code haven't been released yet and the details of some critical parameters such as the configuration of adversarial attacks, the data filtering ratio in each step, and the learning rates are not covered in the paper, we failed to produce satisfactory results. Besides, the threat

model of PBE is different from ours which makes it incomparable to SAU, and therefore, we do not include it in the experiment section.

Adaptation to batch normalization. Batch normalization (BN) has been widely used in modern deep neural networks due to improved convergence. However, recent works show that it is difficult for BN to estimate the correct normalization statistics of a mixture of distributions of adversarial examples and clean examples [57, 48]. Such a problem is magnified when we adversarially fine-tune a model with a small set of samples and may destroy the generalization ability of the model due to biased BN statistics. To address this problem, we propose the fixed BN strategy in the adversarial unlearning/training process. Note that in all experiments, the fixed BN strategy is applied to i-BAU, SAU, and vanilla AT when BN is used in the model architecture like PreAct-ResNet18.

Experiments on VGG19. For VGG19 [41], BN is not used. Therefore, the fixed BN strategy is not applied. Moreover, some methods that depend on BN, such as ANP and EP, are not applicable to VGG19.

D Additional experiments

In this section, we provide additional experiments.

D.1 Main experiments on VGG19

This section provides additional experiment results on VGG19 with CIFAR-10 (Table 16) and Tiny ImageNet (Table 17) to supplement Section 4. Table 16 shows that SAU outperforms all baselines on CIFAR-10 and VGG19, with a 40.56% decrease of ASR compared to the second lowest ASR. Moreover, SAU achieves the highest DER in 6 of 7 attacks, demonstrating a significantly better tradeoff between accuracy and ASR. It also achieves the top-2 R-ACC in 5 of 7 attacks and the best average R-ACC, indicating its good ability to recover the prediction of poisoned samples. Table 17 shows that SAU achieves the best DER in 5 of 6 attacks and the top-2 lowest ASR in 5 of 6 attacks. Although SAU only achieves the second-best average ASR in Table 17, it has significantly higher accuracy than NC, which achieves the best average ASR.

D.2 Main experiments on GTSRB

This section provides additional experiment results on GTSRB with PreAct-ResNet18 (Table 18) and VGG19 (Table 19) to supplement Section 4. In both tables, SAU achieves the top-2 lowest ASR in 4 of 6 attacks and the lowest ASR on average. Furthermore, SAU achieves the top-2 highest DER in 4 of 6 attacks for PreAct-ResNet18 (Table 18) and 5 of 6 attacks for VGG19 (Table 19).

D.3 Experiments with 5% poisoning ratio

This section reports the experiment results for comparing SAU with baselines with a poisoning ratio of 5% on CIFAR-10 and Tiny-ImageNet. As with the results for a poisoning ratio of 10%, SAU achieves the top-2 lowest ASR in most of the attacks. Tables 20 and 21 show that SAU achieves the lowest average ASR. In summary, SAU still outperforms other baselines for a poisoning ratio of 5%.

D.4 Learning curves for Shared Adversarial Risk

This section presents a plot of the learning curves for shared adversarial risk to illustrate the relationship between the backdoor risk (measured by ASR) and the shared adversarial risk. For each plot, we run SAU on CIFAR-10 with PreAct-ResNet18 and a poisoning ratio of 10% for 10 epochs. The SAR is computed on the test dataset by computing the shared adversarial example using the inner maximization step in (8). Figure 5 demonstrates that SAU can effectively reduce the shared adversarial risk, and thus, mitigate the backdoor risk. We note that in order to satisfy Assumption 1, the perturbation set adopted by SAU is much larger than the perturbation budget in the adversarial robustness area, resulting in a high adversarial risk in Figure 5.

D.5 Experiments on different PGD configurations

In this section, we test SAU using different PGD configurations to generate the shared adversarial examples. Specifically, we consider the following configurations for PGD:

Config 1: PGD with L ∞ norm bound 0.2, step size 0 . 2 , and 5 steps.

Config 2: PGD with L ∞ norm bound 0.1, step size 0 . 1 , and 5 steps.

Config 3: PGD with L ∞ norm bound 0.2, step size 0 . 1 , and 5 steps.

Config 4: PGD with L 2 norm bound 5, step size 0 . 2 , and 5 steps.

Config 5: PGD with L 1 norm bound 300, step size 0 . 2 , and 5 steps.

The experiments are conducted for CIFAR-10 with PreAct-ResNet18 and poisoning ratio 10% . We summarize the experiment results in Table 22, from which we can find the SAU can effectively mitigate backdoors under various configurations of PGD.

1

2

3

4

5

6

7

8

9

10

Epochs

0

20

40

60

80

Percentage

BadNets

Adversarial Risk

Shared Adversarial Risk

1

2

3

4

5

6

7

8

9

10

Epochs

0

20

40

60

80

LF

Adversarial Risk

Shared Adversarial Risk

1

2

3

4

5

6

7

8

9

10

Epochs

0

20

40

60

80

SSBA

Adversarial Risk

Shared Adversarial Risk

D.6 Experiments on different numbers of clean samples

This section examines the influence of clean sample size for SAU by evaluating SAU with different numbers of clean samples. The experiment is conducted on CIFAR-10 with PreAct-ResNet18 and the results are summarized in Table 23. Table 23 shows that SAU can consistently mitigate backdoors with sample sizes ranging from 2500 (5%) to 500 (1%) with high accuracy. When the sample size decreases to 50 (0.1%, 5 samples per class) or 10 (0.02%, 1 sample per class), SAU can still reduce

the backdoor to a low ASR. However, clean accuracy is difficult to guarantee with such limited clean samples.

D.7 Experiments on Multi-trigger Attacks

In this section, we evaluate our method for the multi-trigger/multi-target cases. Specifically, we use two different attacks (denoted by Attack-1 and Attack-2) with different target labels. Following Section 4.2, we test our method (SAU) against a strong baseline ANP on the CIFAR-10 dataset with a poisoning ratio of 10% and backbone PreAct-ResNet18. The experiment results are summarized in the following table, where we use ASR-1 and ASR-2 to represent the Attack Success Rate for Attack-1 and Attack-2, respectively. From Table 24, we can find that SAU achieves much higher accuracy ( 5 . 64% higher) and a much lower ASR for Attack-1 ( 26 . 46% lower) compared to ANP, although the average ASR for Attack-2 of ANP is slightly lower than that of SAU (only 0 . 13% lower). These results show that SAU outperforms the baseline in most cases, which demonstrates the effectiveness and robustness of our method in this challenging scenario.

D.8 Experiments on ALL to ALL attack

In this section, we compare SAU with other baselines on ALL to ALL attacks on CIFAR-10 with PreAct-ResNet18 and poisoning ratio 10% . Specifically, the target labels for the sample with original labels y are set to y t = ( y + 1) mod K where mod is short for "modulus". The experiment results are summarized in Table 25. From Table 25, we can find that SAU achieves the best defending performance in 5 of 7 attacks and the lowest average ASR. At the same time, SAU also achieves the best average R-ACC and average DER, which further demonstrates its effectiveness in defending against backdoor attacks with multiple targets.

Note that the ASR for the poisoned model is significantly lower than the ASR in the single-target case. Thus, Table 25 also shows that SAU can still effectively mitigate backdoor even the ASR of the poisoned model is much lower than 100% .

D.9 Comparison to Reconstructive Neuron Pruning (RNP)

In this section, we compare SAU with RNP [29] to further improve our submission. We adopted the official implementation of RNP (https://github.com/bboylyg/RNP) into the framework of BackdoorBench for fair comparison and the specific experimental settings are:

Dataset and model architecture : we compare our SAU with RNP on CIFAR-10 dataset with ResNet18 and PreAct-ResNet18.

Clean data size and poisoning ratio : We adopt clean datasets: 1% and 5%, and poisoning ratios: 0.5%, and 10% to compare their effectiveness for large and small poisoning ratios.

Hyper-parameters of RNP : We notice that the hyper-parameters of RNP are sensitive to model architecture. After carefully inspecting the pruning process of RNP, we set the pruning threshold to 0.75 for ResNet18 and 0.05 for PreAct-ResNet18 which can achieve a good overall performance across various attacks.

The experimental results are shown in Table 26,27,28. According to the results and methodologies, we briefly summarize a few important differences between them:

Sensitivity to the size of the clean dataset. SAU is effective across various sizes of clean data, while RNP achieves good performance for clean data size 0.5% 1%, as shown in Table 26 and discussed in Section 4 of [29]. Therefore, SAU can be applied to various sizes of clean datasets, especially when the defenders do not know the exact size of the training dataset.

Sensitivity to poisoning ratio. SAU is effective across various poisoning ratios, while RNP faces challenges when defending against low poisoning ratio attacks, as shown in Table 28 and discussed in Section 5 of [29].

Sensitivity to model architecture. SAU is effective and stable across various model architectures, while the parameters of RNP are sensitive to the model structure, as we observe that the pruning threshold for RNP varies significantly for various models.

Motivation. SAU is theoretically motivated by the relationship between adversarial robustness and backdoor risk, while RNP is empirically driven by some experimental observations.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Shared Adversarial Unlearning (SAU) method for mitigating backdoor attacks. The motivation behind this method is the recognition that not all adversarial samples are effective for backdoor defense. Therefore, it is important to identify the adversarial samples that truly contribute to backdoor defense during adversarial training. The authors derive an upper bound on the backdoor risk from benign model and backdoor model predictions, and optimize the generation of backdoor samples based on this bound. Experimental results demonstrate the effectiveness of the method in purifying the backdoor model.\n\nSTRENGTHS:\n- Clear research motivation, well-structured writing, and provides ample theoretical support. \n- Comprehensive comparison results with mainstream backdoor defense methods on multiple datasets, confirm its effectiveness.\n\nWEAKNESSES:\n- Adversarial training often involves complex min-max optimization problems, making it difficult to apply in practical training. However, existing defense work, such as ABL, achieves efficient backdoor removal by detecting suspicious backdoor samples and performing unlearning (gradient ascent) training. ABL's unlearning paradigm is evidently a simpler and more direct backdoor removal strategy. Therefore, it would be interesting to explore the combination of the adversarial sample generation strategy proposed in this paper with ABL's unlearning paradigm, simplifying the optimization process of the proposed adversarial training. It would be helpful if the authors could provide some theoretical or empirical results.\n\n- From Table 3 in the experiments, it is apparent that the proposed SAU method fails to effectively defend against the WaNet attack with full-image triggers on the TinyImageNet dataset (attack success rate after defense is as high as 85.75%). In other words, when the image scale or trigger norm increases, the SAU method may face defense failures. The authors need to provide a reasonable explanation and corresponding countermeasures to demonstrate the generalization performance of the SAU method.\n\n- If the attacker is aware of the defender's adversarial training strategy, can they generate corresponding backdoor trigger samples based on the adversarial training and launch attacks? In such a scenario, is the SAU defense effective?\n\n- To my knowledge, adversarial defense is sensitive to the scale of clean data. In the experiments, the authors state that the defense method requires 5% clean samples. How much will the accuracy and defense performance of SAU decline when the scale of clean samples drops below 1%?\n\n- There is a lack of comparison with more advanced defense benchmarks, such as the recently proposed RNP [1].\n\nOverall, the authors of this paper highlight the differential impact of different types of adversarial samples on backdoor defense during adversarial training. While there are optimization uncertainties and application limitations, such as the difficulty in scaling to large models, associated with mitigating backdoor attacks through adversarial training, the authors' findings provide new insights into the relationship between adversarial samples and backdoor unlearning. Therefore, if the authors can address the aforementioned concerns and related questions, I would be happy to increase the score.\n\n[1] Yige Li, Xixiang Lyu, Xingjun Ma, et al, Reconstructive Neuron Pruning for Backdoor Defense, ICML, 2023.\n\nQUESTIONS:\nRefer to the weaknesses mentioned above.\n\nLIMITATIONS:\nRefer to the weaknesses mentioned above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to defend against backdoor attacks in deep neural networks through adversarial training techniques. By exploring the connection between adversarial examples and poisoned samples, the authors propose an upper bound for backdoor risk and a bi-level formulation for mitigating backdoor attacks in poisoned models. The proposed approach can identify shared adversarial examples and unlearns them to break the connection between the poisoned sample and the target label.\n\nSTRENGTHS:\n- This paper explores the connection between adversarial examples and backdoor attacks, which is interesting.\n\n- Extensive experiments are conducted to evaluate the performance of the proposed method.\n\n- The paper is well written and easy to read.\n\nWEAKNESSES:\n- Some important related works are missing. There are some existing works that study provable/certified robustness against backdoor attacks or data poisoning [1,2,3]. However, the authors do not mention those works in the paper. It would be interesting if the authors could compare their method with those works.\n\n- This paper assumes that the trigger magnitude is bounded by a norm, which may degrade the practicality of the proposed method. In practice, it is entirely possible for the attacker to adopt triggers with very large magnitudes, and the proposed method may not work in this case.\n\n- The proposed method relies on a clean dataset to achieve the defense goal, and the quality of the dataset may affect the performance of the proposed method. However, it is not clear how to obtain a clean dataset with high quality in practice.\n\n     [1] RAB: Provable Robustness Against Backdoor Attacks. IEEE S&P 2022.\n\n     [2] Certified Robustness of Nearest Neighbors against Data Poisoning and Backdoor Attacks. AAAI 2022.\n\n     [3] BagFlip: A Certified Defense against Data Poisoning. NeurIPS 2022.\n\nQUESTIONS:\n- Compare the proposed method with existing works that study provable/certified robustness against backdoor attacks or data poisoning.\n\n- How to achieve good performance when the trigger magnitude is larger than the assumed norm?\n\n- Does the quality of the clean dataset affect the performance of the proposed method? How to obtain the dataset with high quality in practice?\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analysed the relationship between adversarial examples and poisoned examples. \nThen, this paper proposed a fine-tuning strategy to purify the poisoned model.\n\nSTRENGTHS:\n1 This paper is easy to follow. \n\n2 This paper provides some experiments that support the proposed method. \n\n3 This paper has some experimental analyses that can lead to the proposed method.\n\nWEAKNESSES:\n1 It seems this paper used many  ill-established notions such as backdoor risk, shared adversarial risk, vanilla adversarial risk, etc..  To the best of my knowledge, I haven't heard to backdoor risk in any machine learning/AI books. \n\n2 Adversarial training has the issue of high computational overhead, which could limit the practicality of the proposed method in real-world applications.\n\n3 The key reference is missing. This paper also need to compared with the paper [1]. \nPrevious work [1] has proposed to tackle with backdoor attacks with multiple adversarial perturbations (L_p adversarial attack and spatial adversarial attack). I encourage the authors to discuss the differences between both works and explore more types of adversarial attack such as spatial adversarial attacks [2] or perceptual adversarial attacks [3].  Besides, some popular backdoor attacks do not satisfy Assumption 1 such as BadNets which has a large L_p norm. \n\n[1] On the Effectiveness of Adversarial Training against Backdoor Attacks. TNNLS 2023.\n\n[2] Spatially Transformed Adversarial Examples. ICLR 2018.\n\n[3] Perceptual Adversarial Robustness: Defense Against Unseen Threat Models. ICLR 2021.\n\nQUESTIONS:\nCan the proposed method mitigate the backdoor issues of the contrastive learning [4], which is a more practical setting. \n\n[4] Poisoning and backdooring contrastive learning, ICLR2022\n\nLIMITATIONS:\nAfter the part of conclusion, the authors discussed the limitation of this work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nsummary: The paper proposes a mitigation approach for backdoor attacks in ML (attacks where a model predicts target classes for poisoned samples when triggered by the adversary). The authors analyze the relationship between backdoor risk and adversarial risk [adversarial examples and poisoned samples] to create an upper bound for the backdoor risk, identify --Shared Adversarial Examples-- between the poisoned model and the fine-tuned model (which can successfully activate the backdoor) and then formulate an optimization or a purification problem for mitigating backdoor attacks (Shared Adversarial Unlearning) to break the connection between poisoned samples and the target label, such that they are either classified correctly by the fine-tuned model or differently by the two models. The authors further thoroughly analyze their approach, comparing it with six other SOTA defense methods on seven types of backdoor attacks with different model structures and datasets.\n\nSTRENGTHS:\nThe paper targets an important problem. The approach is novel and the thorough analysis of several key metrics indicates the efficacy of the approach.\n\nWEAKNESSES:\n* Many  important part are in the supplemental materials, including some of the approaches such as the SAU implementation using SGD to unlearn the SAEs, the all to all case, and multi-trigger in threat model (section 3.1) is unclear.\n\nQUESTIONS:\n*  Type I adv. examples where hθbd and hθ can be misled to the same class represents an upper bound for back door examples. How do you account for false positives? Or is it assumed to be always 100%? \n\n* How does ASR vary in comparison to other unlearning techniques for backdoor mitigation such as I-BAU.\n\n* What are the tradeoffs associated with this approach in terms of implementation - specifically around the cost of identifying the SAEs and SAU?\n\nLIMITATIONS:\n* Writing flow can be improved. Adding a toy example would definitely improve the overall understanding of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}