Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples
Authors: Shaokui Wei
Mingda Zhang
Hongyuan Zha
Baoyuan Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: Backdoor Attack
Trustworthy AI
Backdoor Learning
TL;DR: A novel upper bound of backdoor risk is proposed with a method for purifying backdoored models using small clean datasets.

Abstract: Backdoor attacks are serious security threats to machine learning models where an adversary can inject poisoned samples into the training set, causing a backdoored model which predicts poisoned samples with particular triggers to particular target classes, while behaving normally on benign samples. In this paper, we explore the task of purifying a backdoored model using a small clean dataset. By establishing the connection between backdoor risk and adversarial risk, we derive a novel upper bound for backdoor risk, which mainly captures the risk on the shared adversarial examples (SAEs) between the backdoored model and the purified model. This upper bound further suggests a novel bi-level optimization problem for mitigating backdoor using adversarial training techniques. To solve it, we propose Shared Adversarial Unlearning (SAU). Specifically, SAU first generates SAEs, and then, unlearns the generated SAEs such that they are either correctly classified by the purified model and/or differently classified by the two models, such that the backdoor effect in the backdoored model will be mitigated in the purified model. Experiments on various benchmark datasets and network architectures show that our proposed method achieves state-of-the-art performance for backdoor defense. The code is available at https://github.com/SCLBD/BackdoorBench (PyTorch) and https://github.com/shawkui/MindTrojan (MindSpore).

Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples
Shaokui Wei Mingda Zhang'
Hongyuan Zha! Baoyuan Wu!*
School of Data Science,
The Chinese University of Hong Kong, Shenzhen (UUHK-Shezzenn), China SShenzhen Institute Artificial Intelligence and Robotics for Society, China
Abstract
Backdoor attacks are serious security threats to machine learning models where an adversary inject poisoned samples into the training set, causing backdoored model which predicts poisoned samples with particular triggers to particular target classes, while behaving normally benign samples. In this paper, we explore the task of purifying a backdoored model using a small clean dataset. By establishing the connection between backdoor risk and adversarial risk, derive novel upper bound for backdoor risk, which mainly captures the risk on the shared adversarial examples (SAEs) between the backdoored model and the purified model. This upper bound further suggests a novel bi-level optimization problem for mitigating backdoor using adversarial training techniques. To solve it, we propose Shared Adversarial Unlearning (SAU). Specifically, SAU first generates SAEs, and then, unlearns the generated SAEs such that they are either correctly classified by the purified model and/or differently classified by the two models, such that the backdoor effect in the backdoored model will be mitigated the purified model. Experiments on various benchmark datasets and network architectures show that our proposed method achieves state-of-the -hett-rrtt performance for backdoor defense. (Py (PyCocch) and
The code is available at https://github. com/SCLBDD/Backdooo wtwp:/hwww..ooo/wwwuuuuuue
(MindSpore).
Introduction
Recent decades have witnessed growing application of deep neural network (DNN) techniques in various domains. such as face recognition, autonomous driving, and medical image processing [1, 17, 32, 44]. However, DNNs are known to be vulnerable malicious attacks that can compromise their security and reliability. One of the emerging threats is the backdoor attack, where an attacker attempts to inject stealthy backdoor into a DNN model by manipulating a small portion of the training data such that the poisoned model will behave normally for clean inputs while misclassifying any input with the trigger pattern to a target label. Since the poisoned models can have real-world consequences, such as allowing attacker to gain unauthorized access to system or to cause physical harm, it is crucial to develop effective methods for backdoor defense in developing and deploying DNNs.
Adversarial training (AT) is one of the most effective methods for improving the robustness of DNNs [33]. AT is typically formulated as min-max optimization problem, where the inner maximization aims to find adversarial perturbations that fool the model, while the outer minimization to reduce the adversarial risk. Recently, several works have explored the use of AT for defending against backdoor attacks. However, these works face some limitations and challenges. Weng et al. [52] and
*Corresponds to Baoyuan Wu (wubaoyuan wwwaaaannnuuukkckc.cammmm
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Gao et al. [13] found that AT can only be effective for mitigating backdoor attacks with certain trigger patterns and may even increase the vulnerability to backdoor attacks when training from scratch on poisoned datasets. Zeng et al. [59] proposed to unlearn the universal adversarial perturbation (UAP) to remove backdoors from poisoned models. Since directly unlearning UAP leads to highly unstable performance for mitigating backdoors, they employ implicit hyper gradient to solve the min-max problem and achieve remarkable performance. However, their method assumes that the same trigger can activate the backdoor regardless of the samples it is planted on, and therefore lacks guarantee against more advanced attacks that use sample-specific and/or non-additive triggers.
In this paper, we consider the problem of purifying a poisoned model. After investigating the relationship between adversarial examples and poisoned samples, a new upper bound for backdoor risk to the fine-tuned model is proposed. Specifically, by categorizing adversarial examples into three types, we show that the backdoor risk mainly depends on the shared adversarial examples that mislead both the fine-tuned model and the poisoned model to the same class. Shared adversarial examples suggest a novel upper bound for backdoor risk, which combines the shared adversarial risk and vanilla adversarial risk. Besides, the proposed bound can be extended with minor modifications to universal adversarial perturbation and targeted adversarial perturbation. Based on the new bound, we propose a bi-level formulation to fine-tune the poisoned model and mitigate the backdoor. To solve the bi-level problem, we proposed Shared Adversarial Unlearning (SAU). SAU first identifies the adversarial examples shared by the poisoned model and the fine-tuned model. Then, to break the connection between poisoned samples and the target label, the shared adversarial examples unlearned such that they are either classified correctly by the fine-tuned model differently by the two models. Moreover, our method can be naturally extended to defend against backdoor attacks with multiple triggers and/or multiple targets. To evaluate our method, we compare it with six state-of-ue -heetrrtt (SOTA) defense methods on seven SOTA backdoor attacks with different model structures and datasets. Experimental results show that our method achieves comparable and even superior performance to all the baselines.
Our contributions are three folds: 1) We analyze the relationship between adversarial examples and poisoned samples, and derive a novel upper bound for the backdoor risk that can be generalized to various adversaria training- based methods for backdoor defense; 2) We formulate bi-level optimization problem for mitigating backdoor attacks in poisoned models based on the derived bound, and propose an efficient method to solve it; 3) We conduct extensive experiments evaluate the effectiveness of our method, and compare it with six state-of tfte-oof-the-ar defense methods on seven challenging backdoor attacks with different model structures and datasets.
2 Related work
Backdoor attack. Backdoor attack is one of the major challenges the security of DNNs. The poisoned model behaves normally on clean inputs but produces the target output when the trigger pattern is present. Based on the types of triggers, backdoor attacks can be classified into two types: fixed-pattern backdoor attacks and sample-specific backdoor attacks. BadNets [15] is the first backdoor attack that uses fixed corner white blocks as triggers. To improve the stealth of the triggers, Blended [7] is proposed to blend the trigger with the image in weighted way. Since fixed-pattern triggers can be recognized easily, sample-specis
backdoor attacks have been proposed. SSBA [30], WaNet [38], LF [58], IRBA [12]. VSSC [49] and TAT [8] use different techniques to inject unique triggers for different samples from different angles. Sleeper-agent [42] and Lira [9] optimize the target output to obtain more subtle triggers. Recently, 7hu et al. [67] proposed a learnable poisoning sample selection strategy to further boost the effect of backdoor attacks. To keep the label of the backdoor image matching the image content, LC [40] and SIG [2] use counterfactual and other methods to modify the image to deploy clean label attacks.
Backdoor defense. Backdoor defense aims to reduce the impact of backdoor attacks on deep neural networks (DNNs) through training and other means. There are three types of backdoor defense: pre-processing, in-processing, and post- proseproccssing. Pre- Pre-processing backdoor defense aims to identify the poisoned samples in the training dataset. For instance, AC [5] filters out the poisoned samples by the abnormal activation clustering phenomenon in the target class; Confusion Training [39] identifies the poisoned samples by training a poisoned model that only fits the poisoned samples; VDC [68] incorporates multimodal large language models to detect the poisoned samples. In-processing
2

---

backdoor defense reduces the effect of the backdoor during training. For instance, ABL [27] exploits the fact that the learning speed of backdoor samples faster than that of the clean sample, and splits
some poisoned samples to eliminate the backdoor by forgetting these poisoned samples; DBD [19]
divides the backdoor training process and directly inhibits the backdoor learning process; D-ST [6] splits the backdoor samples and uses semi-supervised learning by observing that the clean samples are more robust to image transformation than the backdoor samples. Post-processing backdoor defense mitigates the effect of backdoors for poisoned model by pruning or finc-tuning. For example, FP [31] prunes some potential backdoor neurons and fine-tunes the model to eliminate the backdoor effect; ANP [55] finds the backdoor neurons by adversarial perturbation to model weights; EP [64] and CLP [63] distinguish the characteristics of the backdoor neurons from the clean neurons; NAD [28] uses mild poisoned model to guide the training of the poisoned model to obtain cleaner model; I-BAU [59] finds possible backdoor triggers by the universal adversarial attack and unlearn these triggers purify the model; FT-SAM [65] boosts the performance of fine-tuning for backdoor mitigation by incorporating sharpness-aware minimization; NPD [66] learns a lightweight linear transformation layer by solving well designed bi-level optimization problem to defend against
backdoor attack.
Adversarial training. In adversarial training, models are imposed to learn the adversarial examples in the training stage and therefore, resistant to adversarial attacks in the inference stage. In one of the earliest works [14]. the adversarial examples are generated using Fast Gradient Sign Method. In [33], PGD-AT is proposed, which generates adversarial examples by running FGSM multiple steps with projection and has become the most widely used baseline for adversarial training. Some further improvements of PGD-AT include initialization improvement [21, 23], attack strategy improvement [22], and efficiency improvement [53, 62].
3 Methodology
In Section 3.1, we first introduce notations, threat model, and defense goal formulate the problem. By investigating the relationship between adversarial risk and backdoor risk, a new upper bound of backdoor risk is derived in Section 3.2, from which bi-level formulation is proposed in Section 3.3.
3.1 Preliminary
Notations. We consider a K KCllaas (K > 2) classification problem that aims to predict the label y y' of a given sample æ E X. where y' [1..... K] is the set of labels and X' is the space of samples. Let hi : X' + y be DNN classifier with model parameter O. We use 'D to denote the set of data (x, y). For simplicity, we use x E as a abbreviation for (xx y) E D. Then, for sample
D, its predicted label is
ho (x)
arg max pa (a:; O),
k= ,K
where (x; 0) is the (softmax) probability of x belonging class k.
Threat model. Let V be the set of triggers and define g X' x V - X' as the generating function for poisoned samples. Then, given trigger A V and sample E X, one can generate poisoned sample g(x; A). For simplicity, we only consider all to one case, i.e., there is only one trigger A and one target label g. The all all case and multi-trigger case are left in Appendix A. We assume that the attacker has access to manipulate the dataset and/or control the training process such that the trained model classifies the samples with pre-defined trigger A to the target labels ý while classifying clean samples normally. In addition, we define the poisoning ratio as the proportion of poisoned samples in the training dataset.
Defense goal. We consider a scenario where defender is given poisoned model with parameter Ood and a small set of clean data Da = {xx, v. Nr whose labels are not the target label, we further define the set of non-target samples as D_ {(xr, y) (æt v) Da,, y a ÿ). Note that the size of D. is small and the defender cannot train a new model from scratch using only Dat- The defender's goal is to purify the model so that the clean performance is maintained and the backdoor effect is removed or mitigated. We assume that the defender cannot access the trigger A or the target label ý.
Since we are mainly interested in samples
3

---

Problem formulation. Using the 0-1 loss [50, 60], the classification risk Rei and the backdoor risk Red with respect classifier he on Dd can be defined as:
1 Σ
(()) # ya), Roa(ho) En [oholg(a A)) ū. E D-g)
Rat
(2)
[D-9)
where Il is the indicator function and
denotes the cardinality of a set.
In (2), the classification risk concerns whether clean sample is correctly classified, while the backdoor risk measures the risk classifying a poisoned sample from D-g to target class ÿ.
In this paper, we consider the following problem for purifying a poisoned model:
min Red (he) + XRea(ho),
(3)
where À is hyper--paaaane parameter to control the tradeoff between classification risk and backdoor risk.
However, directly solving (3) is impractical due the lack of the trigger A and target label Therefore, a natural choice is to replace Rad with surrogate risk irrelevant to A and ý.
3.2 Connection between backdoor risk and adversarial risk
To tackle the above problem, we propose a novel upper bound of backdoor risk which can be relaxed surrogate of Rod that is independent of A and ý. Before that, we first decompose the adversarial risk and review the connection between backdoor risk and adversarial risk.
Adversarial risk. Given sample æ and an adversarial perturbation e S, an adversarial example E can be generated by 2. t+c where S is the set of perturbations. We define the set of adversarial example-label pair he generated by perturbation c and dataset Dat by Ac,o = {( y) ho(()) * y. (x. y) E } and abbreviate (ær. E Ac.o by C. AA... Then, the vanilla adversarial risk for hi on D-g 9 can defined as
Radu (ho) E maxe, Es I( E 0, 22 E ED-g)
(4)
ID-ş]
To bridge the adversarial examples and poison samples, provide the following assumption: Assumption 1. Assume that g(æ; A) x E S for Væ E Das
Assumption ensures that there exists E S such that æ + g(:; A). Note that the analysis and the proposed method are independent of any specific types of perturbation. More discussions about the choice of S and its influence the proposed method are discussed in Appendix B.4.
Then, we reach the first upper bound of backdoor risk:
Proposition 1. Under Assumption 1, Radv serves an upper bound Rod, iee, Rod S Radv-
Remark. Although Rade seems to be promising surrogate for Red. it neglects some essential connections between adversarial exam- ples and poisoned samples, resulting in signif- icant gap between adversarial risk and backdoor risk, as shown in Figure 1. Such gap also leads to a fluctuating learning curve for back- door risk and significant drop in accuracy in the backdoor mitigation process by adversar- ial training. Therefore, we seek to construct tighter upper bound for Rid- A key insight is that not all adversarial examples contribute to backdoor mitigation.
80
Percentage
Risk Gap:96.41%
40
Beshinor Pirk
20
40
60
80
100
Epochs
Figure 1: Example of purifying poisoned model using adversarial training Tiny ImageNet [26]. The curves for Accuracy, Backdoor Risk, and Ad- versarial Risk indicated by Green, Orange, and Purple, respectively.
To further identify AES important for mitigating backdoors, we leverage the information from the poisoned model to categorize the adversarial examples to he into three types, shown in Table 1.

---

Table 1: Different types of adversarial examples to ho-
Type Description
Definition
Mislead he and ho.,, to the same class Mislead he, but not mislead he. Mislead ho and ho. to different classes
how (Gt) = ho (Te) herd (..) of hel (Te) Nora (tte
II
hor (ate) # ho (Tte) hord (Tt) # y, no # y
Adversarial Examples Adu
Adversarial Examples hy
Adversriaa Examples to Adu
Advensarial Examples to
Type
Type
Paisoned Samspees
Peisaned Sumples
Type III
Type
Type III
Type
Figure 2: A schematic of the relationship between adversarial examples, shared adversarial examples (SAEs, Type 1) and poisoned samples. The adversarial examples for he, and he are shown in the blue and green solid ellipses, respectively. The poisoned samples are in the black dashed circle. Assume that her. and he have 100% backdoor attack success rates in the left, such that all poisoned samples are contained in SAEs. Thus, by reducing the shared adversarial examples between hes, and he (from left to right), the backdoor risk can be mitigated in hg.
Furthermore, we refer adversarial examples of Type as the shared adversarial examples between he and hew.. as defined below:
Definition (Shared Adversarial Example). Given two classifiers ho and hag an adversarial example & shared between he, and he, if and only if he, (wx) ho, (it) $ fy.
Let Dr be the subset of samples in D_g on which planting trigger can successfully ac- tivate the backdoor in how.. as summarized in Table As illustrated in Figure 2, the following proposition reveals a deeper con- nection between adversarial examples and poisoned samples.
Table 2: Table of notations.
Notation
Description/Delinitton
hend ho
Poisoned model
Purified model
Trigger
Target label
g(x, A)
Poisoned sample generated from æ
Proposition 2. For V(x, y) E D.. the poi- soned sample g(:; A) allack hg if and
S
Set perturbations
Perturbation
only if (e(gg(( (x; A)) heme (g(x; A)) # y. Furthermore, under Assumption the poi- soned sample g( A) is shared adversar- ial example between hg and howr if it can attack he-
E.
Adversarial example, x +
DJ
((r,, W
D
(ar, (). (m. De. ú)
D.
((x, y) (g(=; U. D_
Ac.o
[(Ge; (ho y. (a, E Da)
Then, the adversarial risk Rado can be decomposed to three components as below:
N Σ max 6ES
Rodo Rodv(ho) = iD_
I(xx E D.) (h) hora (ā.,e,, Di.e. E Ac.o)
Type I addversriill examples
+ T(x; E D.)T(ha (atee))((()))))))))))))))))))))) + he (Di. Di,es E Ac.,))
T(x, E D_g Ti,e E Ac.
Type III adversarial examples
vanilla udversurial exumples D- \D,
(5)
Proposition implies that the first component in (5) is essential for backdoor mitigation, as it captures the risk of shared adversarial examples between the poisoned model he and the fine-tuned model he, and the poisoned samples effective for he belong to SAEs. The second component is irrelevant
5

---

for backdoor mitigation, since adversarial examples of Type II and III are either not poisoned samples or ineffective backdoor attack against he. The third component protects the samples that are originally resistant triggers from being backdoor attacked in the mitigation process.
Thus, by removing the second component in (5), we propose the following sub-adversarial risk
N
(1e)
Σ
max [(x, E D.)ll(ho(@iee) ((rres) ( Dse E Ac, e.ES
ID_g)
+ I(a,; D- Ds, E Ac.oo).
Compared to vanilla adversarial risk Rade. the proposed sub-ade aube-adverarial risk Roub focuses on the shared adversarial examples for samples D3 while considering vanilla adversarial examples for samples in D-g D.. When D, = D i.e., he., has backdoor risk 100%, the sub-adversarial risk measures the shared adversarial risk on D D-g.
Then, we provide the following proposition to establish the relation between hackdoor risk and (sub) adversarial risk:
Proposition 3. Under Assumption for classifier he, the following inequalities hold
Roa(ho) (ssus(ho) S Radv(ho))
Therefore, Rous is tighter upper bound for Rid compared with Radv. After replacing Red with Reub in (3), we reach the following optimization problem:
min (ho(h)) + (ho))
0
Due to the space limit, the proofs of the above propositions, well as some detailed discussions and extensions will be provided in Appendix A.
3.3 Proposed method
Since the target label i is unavailable and 0-1 loss is non-differentiable, we first discuss the relaxation of Rub and then replace 0-1 loss with suitable surrogate loss functions in this section.
Relaxation. One limitation of Ress is the inaccessibility of D.. i.e., the subset of samples in D on which planting trigger can successfully activate the backdoor in he.. Since poisoned model usually has high attack success rate (ASR), the first relaxation is replacing D, with D i.c.,
[[x;; DD)))(ho((i..)) herd (Zi,e, Tie E Ac. ES
Reub (ha)
max
D M
Since is unavailable, we also replace D_g by Da in the sub-adversarial risk. As g(x; A) is not malicious for classification if the ground truth label of æ ÿ. this relaxation has negligible negative influence for mitigating backdoor risk. After the above two relaxations, we reach the following shared adversarial risk on Dai
N
Ryhare (ho)
= N
Σ max ((ho(Ei...))) = ho... Dise; E Ac, e.ES
Due to the space limit, we postpone the detailed discussion of the above two relaxations (c.g., the relaxation gaps) in Appendix A.7.
By replacing Roah with Rshore in (7), we reach the following problem:
min (ho) ( + XRsharehha).
(8)
By solving (8), we seek ho that either classifies the adversarial examples correctly (i.e., ho (a: = 4) or differently with how (i.e., he( * he.,())), (i.)), while maintaining high accuracy Det.

---

Approximation. We firstly need to approximate two indicators in (8), including: 1) I(ho * y); 2) I[ E ho
use the widely used cross-entropy (CE) loss the surrogate loss, i.e., Lat (z. y: 0) CH(p(a:; 0) y). In terms of the latter indicator that measures the shared adversarial risk, different surrogate losses could be employed for different usages:
he.... (+.)) The former indicator is for the clean accuracy,
To generate shared adversarial examples, we first decompose the second indicator below:
I( E Ac.o n As 0r. (hhe (i+) = he. (#+)),
which covers two components standing for fooling her and he simultaneouslly, and keeping the same predicted label for he.. and he, respectively. For the first component,
CE on both he and he... surrogate loss, i.c.,
Lade (Te, y; 0) = (CE(p(ã,: 0).9) CE(p(@ 0bd),
(10)
For the second component, we measure the distance between e 0) and p( Pbd) by Jensen- Shannon divergence [11] and adopt the following surrogate loss:
Lishare eee 0) = -JS(p( 0), p( Obd)).
(11)
To unlearn the shared adversarial examples, we adopt the following surrogate loss
Lsar (Te; y: 0 = (ū + v) log (1 Pi(
(12)
where ho...( (.)). By reducing Learr the prediction of #. by ho, i.e., ho (+e) ) forced differ from he... (ate) if ho (Øc) # y. therefore, reducing the shared adversarial risk (SAR).
Overall objective. By combining all the above surrogate losses with the linear weighted summation mode, we propose the following bi-level objective:
N
min
Σ
( Yei 0) dzLoar (3i,e; ye: 0)}
N
(13)
s.t.
E,
arg max Aa AaLid (āre, yi 0) + Ao (Zi,,, 0), i I,..,,N, e
where \1, d2, da d indicate trade-off weights.
Optimization algorithm. We propose an optimization algorithm called Shared Adversarial Unlearning (SAU, Algorithm 1), which solves (13) by alternatively updating 0, and e*. Specifically, the model parameter is updated using stochastic gradient descent [3], and the perturbation e" is generated by projected gradient descent [33].
Algorithm Shared Adversarial Unlearning
Input: Training set D&t poisoned model he... PGD step size y) 0, number of PGD steps T Tader perturbation set S. max iteration number T.
Initialize he her..
for t = 0,.., 1
for Each mini-batch in Da do Initialize perturbation c.
for tadto 0, ..., Tade do
Compute gradient g of w.r.t. the inner maximization objective in (13).
Update e = IIs + 7) sign(g)) where I[ is the projection operation. end for
Update 0 w.r.t the outer minimization objective (13).
end for end for

---

4 Experiment
4.1 Experiment settings
Backdoor attack. We compare SAU with 7 popular state-of-the-ar (SOTA) backdoor attacks, including BadNets [15]. Blended backdoor attack (Blended) [7]. input-aware dynamic backdoor attack (Input-Aware) low frequency attack (LF) [58], sinusoidal signal backdoor attack (SIG) [2], sample-specific backdoor attack (SSBA) [30], and warping- -banen poisoned networks (WaNet) [38]. To make a fair and trustworthy comparison, we use the implementation and configuration from BackdoorBench [54], a comprehensive benchmark for backdoor evaluation. By default, the poisoning ratio is set to 10% in all attacks, and the 0t label is set to be the target label. We evaluate all the attacks on 3 benchmark datasets, CIFAR-I0 [24]] Tiny ImageNet [26], and GTSRB [43] using two [18] and VGG19 [41]. Due to space constraints, the results for GTSRB and VGG19 are postponed to Appendix D. Note that for clean label attack SIG, the 10% poisoning ratio can only be implemented for CIFAR-10. More attack details are left in Appendix C.
networks:
Backdoor defense. We compare our method with 6 SOTA backdoor defense methods: ANP [55], Fine-pruning (FP) [31], NAD [28], NC [47]. EP [64] and i-BAU [59]. By default, all the defense methods can access 5% benign training data. We follow the recommended configurations for SOTA defenses as in BackdoorBench [54]. For our method, we choose to generate the shared adversarial example with Projected Gradient Descent (PGD) [33] with Lec norm. For all experiments, PGD steps with norm bound 0.2 and we set X X2 = A = and X3 = 0,01. More details about defense settings and additional experiments can be found in Appendix C and D.
run
Evaluation metric. We use four metries to evaluate the performance of each defense method: Accuracy on benign data (ACC), Attack Success Rate (ASR), Robust Accuracy (R-ACC) and Defense Effectiveness Rating (DER). R-ACC measures the proportion of the poisoned samples classified to their true label, and ASR measures the proportion of poisoned samples misclassified to the target label. Larger R-ACC and lower ASR indicate that the backdoor is effectively mitigated. Note that the samples for the target class are excluded when computing the ASR and R-ACC as done in BackdoorBench. DER e [0, 1] was proposed in [65] to evaluate the cost of ACC for reducing ASR. It is defined as follows:
(14) where AASR denotes the drop in ASR after applying defense, and AACC represents the drop in ACC after applying defense.
DER [max(0, AASR) - max AACC) + 11/2,
Note: Higher ACC, lower ASR, higher R-ACC and higher DER represent better defense performance. We use boldface and underline indicate the best and the second-best results among all defense methods, respectively, in later experimental results.
4.2 Main results
Effectiveness of SAU. To verify the effectiveness of SAU, we first summarize the experimental results on CIFAR-10 and Tiny ImageNet in Tables 3 and 4, respectively. As shown in Tables 3-4, SAU can mitigate backdoor for almost all attacks with significantly lower average ASR. For the experiments on CIFAR-10, SAU achieves the top-2 lowest ASR in of 7 attacks and very low ASR for the other 3 attacks. Similarly, SAU performs the lowest ASR in 3 attacks for Tiny ImageNet and negligible ASR in two of the other attacks. Notably, SAU fails to mitigate WaNet in Tiny ImageNet, although it can defend against WaNer on CIFAR-10. As transformation-base attack that applies image transforms to construct poisoned samples, WaNet can generate triggers with a large Loo norm. Specifically. WaNet has average Lox trigger norm 0.348 on TinyImageNet and 0.172 on CIFAR-10 10. Note that for all experiments, we generate adversarial perturbation with Loo norm less than 0.2. The large trigger norm of WaNet on Tiny ImageNet poses a challenge for AT-based methods such as i-BAU and the proposed one, which reveals an important weakness of such methods, i.c., their performance may be degraded if the trigger is beyond the perturbation set. A promising approach for this challenge is to consider adversarial examples for the union of multiple perturbation types [34, 45] and we postpone more discussion to Appendix B.
As maintaining clean accuracy is also important for an effective backdoor defense method, we also compare SAU with other baselines in terms of ACC, R-ACC and DER in Table 3 and 4. As SAU
8

---

Table 3: Results CIFAR-10 with PreAct-ResNet18 and poisoning ratio 10%.
Defense
No Defense
ANP [55]
FP 1311
NC|471
Attack
ACCC
ASR
R-ACC
DER
ACCC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
NetNets [15]
191.32
95.03
4.67
N/A
90.98
4.88
87.22
94.86
91.31
57.13
41.62
08.95
80.05
1.27
89.16
96.75
Blended [7]
93.47
99.92
0.08
N/A
92.97
81.58
13.36
37:27
93.17
99.26
0.73
50.18
93.47
99.92
0.08
50.00
Input-Aware [37]
90.67
98.26
1.06
N/A
91
01.04
1.32
86.71
96.47
91.74
0.04
44.54
99.11
92.61
0.76
90.87
98.77
LF [F|SS]
90.19 98.28
0.71
N/A
92
39.99
55.03
79.37
92.90
98.97
50.01
91.62
SIG [2]
84.48 98.27
1.72
N/A
83.36
36.42
43.67
80.36
89.10
26.20
20.61
56.03
84.48
98.
1.72
50.00
SSBA [30]
92.88 97.86
1.90
NIA
92.62
60.17
36.69
68.71
92.54
63.50
15.36
57.01
90.99
0.58
87.04
97.69
WaNet [38]
91.25 73
9.70
N/A
91
2.22
54
93.17 76
91 46
0.00
69. 73
94.32
80
7.53
85
91.10
Average
91.04
96.91
2.94
N/A
90
00.69
32 84
58.75
81.83
83
91.75
$2.31
27.66
72.23
90.57
29.96
63.06
83.06
Defense
NAD [28)
EP [64]
I-BAU [39]
SAU (Ouns)
Attack
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACCC
ASR
R-ACC
DER
Nets [15]
89.87
2.14
68.71
72
89 06
1,88
89,51
95.75
89.15
1.21
88.88
95.83
60.31
1.53
88.81
96.74
Blended [7]
92.17
97.60
2.14
50.47
92.44
73.37
13.90
53.23
90.56
90.96
6.14
64.89
95.63
Imput-Awcan [37]
93.18
1.68
91.12
96.29
89.86
2.23
85.20
97.61
90.29
63.36
32.70
67:26
91.59
1:27
88.54
98..9
I. [58]
92:37
47.83
47.49
75.31
91.82
85.98
12.77
55.97
89.00
21.83
64.37
96.67
90.32
4.18
81.54
96.12
SIG [2]
90.02
10.66
64.20
91.81
83.1
0.26
56.68
98.12
85.85
1,28
55.19
98.49
58.56
1.67
57.96
98..30
SSBA [30]
91.91
77.4
20.86
30.74
92.33
10.07
78.00
93.32
$8.15
2.17
77 77:28
96.48
90.84
1.79
86 63
97.01
WaNet [38]
22.98
72.60
81.38
90.09
12.54
50.96
90.91
89. 10
93.01
91.26
90.28
94.36
Average
91.81 37.2
55.32
79.53
89.9
34.22 26
53.26
80.76
88.87
15.32
65.82
89.61
90.41
2.51
79.69
96.52
Table 4: Results Tiny ImageNet with PreAct- 8 and poisoning ratio 10%.
Defense
No Defense
ANP
[65]
FP [31]
NC [47]
Artack
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
BadNets [15]
56.23
100.0
0.0
N/A
13.13
51.
17.76
51.52
0.30
50.82
Blended [7]
66.03
99.71
0.22
N/A
43.93
6.11
17.28
90.75
51.89
95.94
2.1
19.81
52.55
93:21
3.96
51.51
Imput-Aware [37]
57.45
98.85
1.06
N/A
35.39
0,00
11.82
85.40
55.28
62.92
21.7
66.88
56.20
0.08
52.19
98.76
LF [58]
55.97
98.57
0.97
N/A
15
62.30
14.49
63.00
51.41
95.25
2.42
19.4
52.99
85.56
8.07
55.022
SSBA [30]
55.22
97.71
1.68
N/A
43
36.36
56.53
17.24
61.66
50.47
88.87
6.26
52.04
52.47
53.47
27.17
70.75
WaNet [38]
0.36
N/A
36
53.33
0.23
Average
56.28
99.06
0.72
N/A
41
33
20.83
22.66
77.12
52:44
74.48
6.31
68.68
53.18
38.78
31.64
74.00
Defense
NAD [28]
EP [64]
i-BAU (59)
SAU (Ours)
Attack
ACC
ASR
R-ACC
DER
R-ACC
DER
ACC
R-ACC
DER
ACCC
ASR
R-ACC
BadNots [15]
16.37
0.27
45.61
91.93
52.30
0.03
52.05
98.02
52.67
98.31
1.59
19.06
51.52
0.53
51.15
97.
Blended [7]
46.89
94:90
2.63
47.79
51 51.86
56.03
12.31
69.75
52.73
92.96
4.15
51.72
50.30
0.06
25.27
96.96
Iaput-Awaae [37]
17.91
1.86
13.13
93.73
57.23
0:21
21.74
99.20
50.63
56.12
21.86
67.96
54.11
0.33
12
97.59
LF [58]
45.45
50.49
20.21
68.78
53.33
75.41
13.06
60:26
12.77
88.04
6.82
53.67
02.65
0.97
35.87
97.14
SSBA [30]
15.32
57.32
19.14
11.07
77
52.52
T1.77
60.12
51.85
0.11
WaNet [58]
16.98
0.43
43.99
94.63
56.21
0.23
53.13
99.34
54.40
94.86
3.92
51.12
54.65
85.75
10:35
55.80
Average
34.23
29.17
73.50
58
29.33
76.55
52.62
84.18
54.81
52.51
14.62
33.55
84.57
adopts adversarial examples to mitigate backdoors, it has negative effects on clean accuracy, resulting in slightly lower clean accuracy compared to the best one. However, SAU achieves the best average R-ACC, which demonstrates its effectiveness in recovering the prediction of poisoned samples, i.e., classifying the poisoned samples correctly. Besides, the best average DER indicates that SAU achieves a significantly better tradeoff' between clean accuracy and backdoor risk.
Influence of poisoning ratio. To study the influence of the poisoning ratio on the effectiveness of SAU, we test SAU with varying poisoning ratios from 10% 50%. As shown in Table 5, SAU is still able to achieve remarkable performance even if half of the data is poisoned.
Table 5: Results on CIFAR-10 10 with PreAct-ResNert and different poisoning ratios.
Prisoning Ratio
10%
20%
30%
40%
50%
Attack
Matric
No Defence
Ours
No Defence
Ours
No Defence
Ours
No Defence
Ours
No Defence
Ours
BadNets [15]
ACC
91.82
89.05
90.17
89.27
88,32
88.49
86.16
87.30
83.61
86.93
ASR
93.79
1.53
96.12
1.17
97.33
1.72
97.78
1.72
98.39
2.31
Blended [7]
ACC
93.69
90.96
93.00
91.02
92.78
89.49
91.64
89.64
91.26
88.92
ASR
99.76
6.14
99.92
7.99
99.98
11.58
99.96
5,36
100.00
4.63
LF [58]
ACC
93.01
90.32
92.60
91.33
92.01
89.65
92.03
89.09
90.91
89.02
ASR
99.06
4.18
99.54
1.90
99,76
2.02
99.84
1.94
99.89
1.77
9

---

4.3 Ablation Study
In this section, we conduct ablation study to investigate each component of SAU. Specifically, SAU is composed of two parts: generating shared adversarial examples according to the maximization objective in (13) denoted by A, and reducing shared adversarial risk according to the minimization objective in (13), denote by B. To study the effect of these parts, replace them with the correspond- ing part vanilla adversarial training, denoted by C and D, respectively. Then, we run each variant 20 epochs on Tiny ImageNet with PreAct-ResNere 8 and poisoning ratio 10%. Each experiment is repeated five times, and the error bar only shown in Figure for simplicity.
As shown in Table 6, the maximization objective for generating shared adversarial examples (compo- nent A) is the key to reducing ASR, and component can help to alleviate the hurt to clean accuracy. When replacing the component B with the mini- mization step vanilla AT (D), it suffers from a drop of clean accuracy, although it can also work for mitigating backdoor effectively. Compared to
Table 6: Results on Tiny ImageNet with differ- ent variants of SAU and Vanilla AT.
Atlack
BadNets
Blended
LF
Defense
|ACC
ASR | ACC ASR
ACC
ASR
A-B (Ours)
|50.60
0.27
|61.47 1.02 51.31
0.72
[37.21
0.30
33..35
1.77
36.68
0.17
C+B
533005
28.57
53.16
69.22
52.57
68.65
vanilla AT, SAU achieves significantly lower ASR and higher ACC, with a much more stable learning curve, shown in Figure 3. We remark that the decrease of clean accuracy for A+D and vanilla AT demonstrates that imposing model to learn overwhelming adversarial examples may severely hurt the clean accuracy, phenomenon widely studied in the area of adversarial training [46, 60].
C+D (Vanilla AT)
46.66
2.48 | [46.39 38.03 46.69
13.68
BadNets
Blended
LF
100
100
1.0
80
60
60
60
20
20-
20
10
10
Epochs
Epochs
Epochs
Defense
A+B (Ours)
A+D
C+8
C+D (Vanilla AT)
Metric
ACC
ASR
Figure 3: Learn curves for SAU, Vanilla AT and their variants, averaged over five runs.
Conclusion
In conclusion, this paper proposes Shared Adversarial Unlearning, method to defend against backdoor attacks in deep neural networks through adversarial training techniques. By developing better understanding of the connection between adversarial examples and poisoned samples, propose a novel upper bound for backdoor risk and a bi-level formulation for mitigating backdoor attacks in poisoned models. Our approach identifies shared adversarial examples and unlearns them to break the connection between the poisoned sample and the target label. We demonstrate the effectiveness of our proposed method through extensive experiments, showing that it achieves comparable to, and even superior, performance than six different state-of-the-a defense methods on seven SOTA backdoor attacks with different model structures and datasets. Our work provides a valuable contribution to the field of backdoor defense in deep neural networks, with potential applications in various domains.
Limitation and future work. One important direction for future work, and a current challenge, is the accessibility of clean samples. A valuable approach to this challenge is to consider the samples from other domains and the generated samples.
Structure of Appendix. The detailed proof, analysis, and extension of propositions are given Appendix A. Some important discussions such as the computation cost, the scalability, and the generalization ability of the proposed method are provided in Appendix B. The details of the experiments are provided in Appendix C. Additional experiments on different settings such as model structures, poisoning ratios, and sizes of clean datasets given in Appendix D.
10

---

Acknowledgments and Disclosure of Funding
This work supported by the National Natural Science Foundation of China under grant No. 62076213, Shenzhen Science and Technology Program under grant No. RCYX20210609900057005 and No. ZDSYS2021 1021 1415025, and No. JCYJ2021032412000 1032 and the CAAI-Huawei MindSpore Open Fund, and the Guangdong Provincial Key Laboratory of Big Data Computing, the Chinese University Hong Kong, Shenzhen.
No. S0SS1100020000000000000000

---

References
[1] Insaf Adjabi, Abdeldjalil Ouahabi, Amir Benzaoui, and Abdelmalik Taleb-Ahmed. Past, present, and future of face recognition: A review. Electronics, 9(8): 188, 2020.
[2] Mauro Barni, Kassem Kallas, and Benedetta Tondi. new backdoor attack in enns by training set corruption without label poisoning. In ICIP 2019, pages 101-105. IEEE, 2019. 2, 8, 9, 26, 27, 28, 30, 31, 32, 33, 34
[3] Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. In NeurlPS 2007,
volume 20, 2007. 7
[4] Shuwen Chai and Jinghui Chen. One-shot neural backdoor erasing via adversarial weight masking. In NeurlPS 2022, 2022. 17
[5] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, lan Molloy, and Biplay Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In Workshop on Artificial Intelligence Safety. CEUR-WS, 2019.
[6] Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In NeurlPS 2022, volume 35, pages 9727-9737, 2022. 3
[7] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv e-prints, pages arXiv-1712, 2017. 2, 8, 9, 26, 28, 29, 30, 31, 32, 33
[8] Ziyi Cheng, Baoyuan Wu, Zhenya Zhang, and Jianjun Zhao. Tat: Targeted backdoor attacks against visual object tracking. Pattern Recognition, 142:109629, 2023,
[9] Khoa Doan, Yingjie Lao. Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. In ICCV 2021, pages 11946-1 1956, 2021. 2
[10] Yinpeng Dong, Xiao Yang, Zhijie Deng, Tianyu Pang, Zihao Xiao, Hang Su, and Jun Zhu. Black-box detection backdoor attacks with limited information and data. ICCV 2021, pages 16482-16491, 2021. 17
[11] Bent Fuglede and Flemming Topsoe. Jensen-shannon divergence and hilbert space embedding. In ISIT 2004, page 31. IEEE, 2004. 7
[12] Kuofeng Gao, Jiawang Bai, Baoyuan Wu, Mengxi Ya, and Shu-Tao Xia. Imperceptible and robust backdoor attack in 3d point cloud. arXiv preprint arXiv:2208.08052, 2022. 2
[13] Yinghua Gao, Dongxian Wu, Jingfeng Zhang, Guanhao Gan, Shu-Tao Xia, Gang Niu, and Masashi Sugiyama. On the effectiveness of adversarial training against backdoor attacks. IEEE Transactions on Neural Networks and Learning Systems, 2023. 2, 22, 23, 24, 27
[14] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In ICLR 2015, 2015. 3
[15] Tianyu Gu, Kang Liu, Brendan Dolin-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. JEEE Access, 1447230-472444 2019. 2, 23, 24, 26, 28, 29, 30, 31, 32, 33, 34
[16] Junfeng Guo, Ang Li, and Cong Liu. AEVA: Black-box backdoor detection using adversarial extreme value analysis. In ICLR 2022, 2022. 17
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR 2016, pages 770-778, 2016.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV 2016, pages 630-645. Springer, 2016.
[19] Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In ICLR 2022. OpenReview.net, 2022. 3
12

---

[20] Jinyuan Jia, Yupei Liu, Xiaoyu Cao, and Neil Zhenqiang Gong, Certified robustness of nearest neighbors against data poisoning and backdoor attacks. In AAAI 2022, volume 36, pages 9575-9583, 2022. 22
[21] Xiaojun Jia, Yong Zhang, Xingxing Wei, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. sioor-guieeddee adversarial initialization for fast adversarial training. In ECCV 2022, pages 567-584, 2022.
[22] Xiaojun Jia, Yong Zhang, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. LAS-AT: adversarial training with learnable attack strategy. In CVPR 2022, pages 13388- 11399, 2022. 3
[23] Xiaojun Jia, Yong Zhang, Baoyuan Wu, Jue Wang, and Xiaochun Cao. Boosting fast adversarial training with learnable adversarial initialization. IEEE Transactions on Image Processing, 31: 4417-4430, 2022. 3
[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 8
[25] Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In ICLR 2021, 2021. 23
[26] Ya Le and Xuan Yang. Timy imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. 4,
[27] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In NeurIPS 2021, pages 14900-14912, 2021.
3
[28] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural at- tention distillation: Erasing backdoor triggers from deep neural networks. In ICLR 2021. OpenReview.net, 2021. 3, 8, 9, 21, 25, 26, 27, 28, 29, 30, 33
[29] Yige Li, Xixiang Lyu, Xingjun Ma, Nodens Koren, Lingjuan Lyu, Bo Li, and Yu-Gang Jiang. Reconstructive neuron pruning for backdoor defense. ICML 2023, 2023. 32, 33, 34
[30] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran Hc, and Siwei Lyu. Invisible backdoor attack with cample-specine triggers. In ICCV 2021, pages 16463-16472, 2021. 8, 9, 23, 24, 27, 28, 29, 30, 31, 32, 33
[31] Kang Liu, Brendan backdooring attacks on deep neural networks. In RAID 2018, pages 273-294. Springer, 2018. 8, 9, 21, 25, 27, 28, 29, 30, 31, 33
Davit,,, and Siddharth Garg. Fine-pruning: Defending against
[32] Liangkai Liu, Sidi Lu, Ren Zhong, Baofu Wu, Yongtao Yao, Qingyang Zhang, and Weisong Shi. Computing systems for autonomous driving: State of the art and challenges. IEEE Internet of Things Journal, 1116646964866, 2020.
[33] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR 2018, 2018. 1, 7, 8, 27
[34] Pratyush Maini, Xinyun Chen, Bo Li, and Dawn Song. Perturbation type categorization for multiple adversarial perturbation robustness. In UA/ 2022, pages 1317-1327. PMLR, 2022. 8
[35] Bingxu Mu, Zhenxing Niu, Le Wang, Xue Wang. Qiguang Miao, Rong Jin, and Gang Hua. Progressive backdoor erasing via connecting backdoor and adversarial attacks. In CVPR 2023, pages (0495-20503, 2023. 22, 27
[36] Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. In ICLR 2021, 2021. 25
[37] Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In NeurIPS 2020, 2020. 8, 9, 26, 28, 29, 30, 31
[38] Tuan Anh Nguyen and Anh Tuan Tran. Wanet imperceptible warping-based backdoor attack. In ICLR 2021. OpenReview.net, 2021. 2, 8, 9, 23, 24, 27, 28, 29, 30, 31, 32, 33
13

---

[39] Xiangyu Qi, Tinghao Xie, Jiachen T Wang, Tong Wu, Saeed Mahloujifar, and Prateek Mittal. Towards a proactive {ML] approach for detecting backdoor poison samples. In 32nd USENIX Security Symposium (USENIX Security 23), pages 1685-1702, 2023. 2
[40] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Du- mitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In NeurlPS 2018, volume 31, 2018. 2
[41] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR 2015, 2015. 8, 28
[42] Hossein Souri, Liam Fowl, Rama Chellappa. Micah Goldblum, and Tom Goldstein. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. In NeurlPS 2022, volume 35, pages 19178, 2022.
[43] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian lgel. The german traffic sign recognition benchmark: multi-class classification competition. In IJCNN 2011, pages 1453-1460. IEEE, 2011. 8
[44] J-Donald Tournier, Robert Smith, David Raffelt, Rami Tabbara, Thijs Dhollander, Maximilian Pietsch, Daan Christiaens, Ben Jeurissen, Chun-Hung Yeh, and Alan Connelly. Mrtrix3: A fast, flexible and open software framework for medical image processing and visualisation. Neuroimage, 202:1161:: 2019.
[45] Florian Tramer and Dan Boneh. Adversarial training and robustness for multiple perturbations. 2019.
In NeurlPS 2019, volume 32, pages
[46] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustesss may be at odds with accuracy. In ICLR 2019, 2019. 10
[47] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks In 2019 IEEE Symposium on Security and Privacy, pages 707-723, IEEE, 2019. 8, 9, 21, 25, 27, 28, 29, 30, 31, 33
[48] Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, and Zhangyang Wang. Re- moving batch normalization boosts adversarial training. In ICML 2022, pages 23433-23445. PMLR, 2022. 28
[49] Ruotong Wang, Hongrui Chen, Zihao Zhu, Li Liu, Yong Zhang, Yanbo Fan, and Baoyuan Wu. Robust backdoor attack with visible, semantic, sample-specific, and compatible triggers. arXiv preprint 23X.::23066008166 2023. 2
[50] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. ICLR 2020, 2020. 4
[51] Maurice Weber, Xiaojun Xu, Bojan Karlaš, Ce Zhang, and Bo Li. Rab: Provable robustness against backdoor attacks. In 2023 IEEE Symposium on Security and Privacy (SP), pages
1311-1328. IEEE, 2023. 21
[52] Cheng-Hsin Weng, Yan-Ting Lee, and Shan-Hung Brandon Wu. On the trade-off between adversarial and backdoor robustness. In NeurIPS 2020, volume 33, pages 11973-11983, 2020. 1.
[53] Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In ICLR 2020, 2020. 3
[54] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen, Backdoorbench: A comprehensive benchmark of backdoor learning. In NewrlPS Datasets and Benchmarks Track 2022, 2022. 8, 26
[55] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NewrIPS 2021, pages 16913-16925, 2021. 3, 8, 9, 21, 24, 25, 26, 27, 29, 30, 31, 32,
14

---

[56] Chaowei Xiao, Jun- Yan Zhu, Bo Li, Warren He. Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In ICLR 2018, 2018. 23, 24
[57] Cihang Xie and Alan L. Yuille. Intriguing properties of adversarial training at scale. In ICLR 2020, 2020. 28
[58] Yi Zeng, Won Park, Z. Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: frequency perspective. In ICCV 2021, pages 16453-16461. IEEE, 2021. 2, 8, 9, 26, 28, 29, 30, 31, 33
[59] Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In ICLR 2022. OpenReview.net, 2022. 2, 3, 8, 9, 17, 21, 22, 25, 27, 28, 29, 30, 31, 33
[60] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML. 2019, pages 1472-7482. PMLR, 2019. 4, 10
[61] Yuhao Zhang, Aws Albarghouthi, and Loris D'Antoni. Bagflip: A certified defense against data poisoning. In NeurIPS 2022, volume 35, pages 31474-31483, 2022. 21
[62] Haizhong Zheng, Ziqi Zhang, Juncheng Gu, Honglak Lee, and Atul Prakash. Efficient adver- sarial training with transferable adversarial examples. In CVPR 2020, pages 1178-1187, 2020.
3
[63] Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. In European Conference on Computer Vision, pages 175-191. Springer, 2022.
[64] Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Pre-activation distributions expose backdoor neurons. In NeurlPS 2022, 2022. 3, 8, 9, 21, 27, 29, 30, 31,
[65] Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. ICCV 2023, pages 4466-4477, 2023. 3,
[66] Mingli Zhu, Shaokui Wei, Hongyuan Zha, and Baoyuan Wu. Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features. arXiv preprint 166677 2023.
[67] Zihao Zhu, Mingda Zhang. Shaokui Wei, Li Shen, Yanbo Fan Fan, and Baoyuan Wu. Boost- ing backdoor attack with a learnable poisoning sample selection strategy. arXiv preprint w.Xiv:2307.07328. 2023.
[68] Zihao Zhu, Mingda Zhang, Shaokui Wei, Bingzhe Wu, and Baoyuan Wu. Vdc: Versatile data cleanser for detecting dirty samples via visual- lingustin inconsistency. arXiv preprint 2xXiv:2309.16211 2023.
15

---

A
Proofs and extension
In this section, we provide detailed proof and analysis of the propositions, methods, and extensions.
A.1 Proof of Proposition
Proof. Under Assumption any sample a D- we have
(1); A)) D) z [(g((;; A) E Ag(m;))) ) S max I( E Ac,o);
kES
since g(x;; A) x E. Therefore, we have
Roa(ho) S (hoo
A.2 Proof of Proposition 2
Proof. By the definition of backdoor attack, poisoned sample g(æ; A) is said attack he success- fully and only if he (e(g((; A) ÿ.
As homa (9(x; A)) ý for all a E D. (definition of D.), we have ho A)) hare (g(x; A)) # and only if ( A) can attack he successfully, for all æ E D..
Furthermore, under Assumption 1, g(æ; A) is an shared adversarial example of ho and how (g(g( A)) how (g(æ; A)) # y by the Definition
A.3 Proof of Proposition 3
Proof. By Proposition 1 and the decomposition of Rade in Section 3.2, we have
Rea(ho) S Rudi (ho). sus (ho) S (ho).
Therefore, we discuss the relation between (ho and Rhub(ho) now.
Recall that
N
Rea)(ho) DD ij
max [[00; (h))(h)(@@ie))) (fite))) Pics E Ac,o) E.ES
(15)
+ [(x, E D-g \ Do, E Acno)}.
Then, under Assumption 1, we have
For Ds, Proposition
yields that
T(ho ax; A)) () T(ho (:::; A)) = hom (g(:;; A)))
(16)
< max (x))) = hom (#e): ) de E Ac.o).
For D-6 D., similar as Proposition 1, we have
T(ho(g(m;; A)) = Ú) S T(g(x; A) E Au S max T( Ac,o).
EES
Therefore, we have
Rea(ho) S Reu.(ho).
So, under Assumption 1, the following inequalities holds:
Roa(he) < Rous (ho) < Rade (he)-
16

---

A.4 Extension to Universal Adversarial Perturbation
Universal Adversarial Perturbation (UAP) has achieved remarkable performance in previous studies [59, 4], especially for defending against backdoor attacks with fixed additive triggers. Now, we extend Rub to UAP and show that shared adversarial examples also benefit adversarial training with UAP.
We denote the Universal Adversarial Risk by
Raaado maxess Eisl
T( E Ac, E ED-g)
(D-g)
To bridge UAP and backdoor trigger, we first provide the following assumption: Assumption 2. Assume that the g(x, A) = x + for all æ E Det-
Assumption says that the poisoned samples planted with fixed additive trigger. Then, under Assumption 2 and Assumption we have
Roo(he) S Ruadv S Radv.
by the fact that A E S.
For clarity, we replace ie E A by ha +e) # y and Ce by x + which makes the rest derivation easier to follow.
Then, similar Roub. we define the universal sub-adversarial risk Rusub as below:
1
Rusub(ho)
max CES
Σ (1 + hora (x + + y. D.)+
(
T(ho(a + <) + y, x D e (D))
which considers shared universal adversarial risk on D, and vanilla universal adversarial risk D-g \ D,.
By the definition of Rusud we can easily find Rusus S Rundo- Furthermore, have
Roa(ho) EN I(ho(w A) 0.00 E D--))
ID-gl
EN (I(ho(x + A) Nora (xc + +)) D.) + I(ho([h A) -. (D.))
DD_gl
maxees Ei [T(holæ + + = hor (or E), E D.) I(ho +6) D-g D.))
\D))
DD=gl
Rha)
Therefore, reach the following proposition
Proposition 4. Under Assumption 2 and Assumption for classifier he, the following inequalities
hold
Rra(ha) Rusub S Ruvad < RRads (ha)-
Therefore, the shared adversarial examples also benefit adversarial training with UAP.
A.5 Extension to Targeted Adversarial Training
In the previous sections, we assume that the defender no access to either the trigger or target label. Since there are already some works for backdoor trigger detection Guo et al. [16], Dong et al. [10], we discuss the case where the defender can estimate target label ý and show that shared adversarial examples also be beneficial for adversarial training with target label.
17

---

Given target label ÿ. we define the Targeted Adversarial Risk as
Rtadw(ho) i= ms T(ho( = D. E EDD_ D-u))
(17)
(D-
By the definition of i. we have Rsa(ha) Readi (ho) under Assumption 1. Then, have
N
1
Reade (he)
Σ
max [[x; D.)l(ho(ōi.e)) h =0)
D_-9
(18)
+ I(x E Dz) D.)I(ho((iie)) ú. hor (ās,er) + ÿ) + I(x; D-g \ Der ho (x =0)
Then, similar as Rebb, we define the targeted sub-adversarial risk Resub by removing the second component in Reado (ha):
N
Resut (ha)
1
Σ
max [(x; E D.)[((h(((r.e)) = (Zi.e)) = D) e,
D 9
+ I(x E D-g he
which considers shared targeted adversarial risk on D, and vanilla targeted adversarial risk on D-9" Dz..
Since the removed component irrelevant to backdoor risk, have Rud S Reaub under Assump- tion 1.
Therefore, reach the following proposition
Proof. Under Assumption 2 and Assumption 1, for a classifier he, the following inequalities hold
Rho( I (ho) I Riadu (he) S Radv (ha).
Therefore, the shared adversarial examples also benefit adversarial training with target label.
Moreover, since (he) only consider shared adversarial example with target label ÿ. the following proposition holds,
Proof. Under Assumption 2 and Assumption 1, for a classifier hg. the following inequalities hold Rha(ho) & Resus(ho) S Rhub(ho) S Radv (ho).
The above proposition shows that Risub (he) is tighter bound of (ho( than Roub(ho)
A.6 Extension to Multi-aarge/MMulttriigeer
In previous sections, we assume that there only one trigger and one target for all samples. Now, we consider to a more general case where a set triggers V and set of target labels Y are given. This is more challenging case since there may be multiple triggers and/or multiple targets for one sample.
We define T to be mapping from the sample to the corresponding set of feasible trigger-target pairs. Therefore, given a sample 2,, the set of all feasible trigger-target pairs is given by T(x)). Then, Assumption is extended to all triggers accordingly, i.e., g(x, A) S, (AA 9) E T(x).
18

---

In this setting, we define that classifier he is said to be attacked by poisoned sample generated by if there exists (A. f) T()) such that (g A)) / y.
Then, the (maximum) backdoor risk on Dai is defined to be
Rho( Ei=1 (a [(ho A.)) Ds + y)
(19)
Dael
Similarly, the adversarial risk on De is defined as
Radv
Σ max (I(holg(z,, e,)) # g)). cE
Dal
We define D, = {x E Det ho., (g(x, A)) = 9 * y. V(A, 9) E T(x)} to the set of samples on which planting any feasible trigger activate the corresponding backdoor. Then, the corresponding adberarrllll risk defined as
N Σ max (1(ha 4.) = how (g(o,,( # v, D.)
Reub Dael E,E
+ ((haag( G)) # y. 2z E Da (.)}
which considers shared adversarial risk on D3 and vanilla targeted adversarial risk Da \ D..
By definition, we have Roub S Radv
Moreover, we have
N Red(ho) (. (( A.)) 51 4)
IDal
N
max [Dol 1=1 ( (ET(.))
[I(ho(g(x;, Ai)) = her. (9(x,, (ol,,,
Ai) = - # v)
+ [(holy(a;; Ad) # y, how (o(a;, A:)) # it)
max (m.
(I(ho(g(a;; Ai)) hord (g(as, Ai)) # n. D.)
Dat
"Ihhoog A.) , y. Ed
<
max cEES (I(ho (g E:)) = hew (g((i, + v, E D.)
Dal
+ I(ho(g(z;, ci)) + y. E Da (D)]
Reub-
Therefore, for backdoor attack with multiple targets and/or multiple triggers, the proposed sub- adversarial risk Roub still tighter upper bound of R&d compared to Rade.
A.7 Relaxation gap
To build practical objective for defending against a backdoor attack, two relaxations are applied to Reud. Here, we discuss the relaxation gap the relaxing process:
Replace D. by DDgg-
Since D. is not accessible, the first relaxation is replace D, by D_g and consider shared adversarial examples all samples in D-g. Denote (ho)) to be the relaxed risk after replacing
19

---

D, by --g. Since Roub can be decomposed as follows:
(ha)
Σ max CEE
[((e; E (.))(hh(@ūree) = (tt,es)) E Ac
D
==1
+ I(x; D \ Ki.e E Acno))
max (D-g E.ES
Σ
I(x, E D ED- (-s)T((h( = (#r.e)) Diie Ac,,))
+ I(ar ED- (a),(h)((s,e)) (i),))) Hi,e E Ac,
We can find that the relaxation gap is
Roub R1 5
max la, D-g (.)((he(( + her Die. E Ac.,) cES
D_9
D Iax E (D,)
1 Red (howa).
And the relaxation gap GAP1 is bounded by - (hea(h Since a poisoned model has usually has a high ASR for the backdoor attack, the gap GAP1 usually negligible.
Remark: We remark that the experiment results show that method also works well for the poisoned models with low ASR. See Section D for more details.
Replace D-g by Dat-
The second relaxation is replace D-g by Da when ý is not accessible. Let D3 = Des D-g. Then, the risk after the second relaxation is:
Rahare
= N
max ((hho(ir..)) = ham (ii,,;), E Ac,oo)
ESES
1=1
D_g
(Iax, D (h)))(h(( home (Brca)) Di & Acco) EE
max
N
1Dyl 1
Σ
max Max e (() hom. (āi,e.) Hiiee E Acn.o). cEES
N
(
where the first component is R1 bcaled
ID al and the second component is the vanilla adversarial n
[Dyl risk on Dj, scaled by N So, the relaxation gap for the second relaxation is
Dgl N
R1 Rohure
R1
(Dg)
Σ
max I(x: Dg)[[(ho( how (B:,e))) T. 1) c,
N
D2
We can find that the generalization gap is related to the
[Pol N i.e., the portion of samples whose
labels are target labels, and negligible when [Dol
is small.
N
Remark. For using Rshare (ho) to mitigate the backdoor, the influence of [D_al can be eliminated
by altering the trade-off parameter Problem (8). Also, adding triggers to samples labeled is harmless during the testing/deployment phase. So, the vanilla adversarial risk on Dy has negligible negative impact on defending the backdoor. Therefore, replacing D-g with Da has little negative impact on mitigating the backdoor.
20

---

B Discussion
In this section, we provide a comprehensive discussion of the proposed method, including its efficiency, comparison to other methods, choice of perturbation, generalization ability to attack with seccessive-magnituee triggers, performance on the generated dataset, and resistance to defense- aware/adaptive attacks.
B.1 Efficiency and Computational Cost
Here, would like emphasize that SAU is efficient and effective defense method that can mitigate backdoor attacks with acceptable overhead for the following reasons:
SAU is post-processing method that fine-tunes the backdoor model on clean dataset with a small size. For instance, we can use only 500 samples from CIFAR-10 to fine-tune the model by SAU (see Appendix D.6 for more details), which takes 1 1.52 seconds/epoch to fine-tune PreAct- PreAct-ResNetl 8 with RTX 4090. Therefore, it can be executed efficiently without requiring large amount of data computation resources.
SAU only takes few epochs to take effect, which further reduces the computational cost of executing it in practice. As shown in Appendix D.4, SAU can achieve a low Attack Success Rate and a high Accuracy in a few epochs.
To further demonstrate the efficiency of SAU, we compare its average runtime with other state- of-the-art defense methods against attacks. We the same experimental setting as in Section where the poisoning ratio is 10% and the backbone model is Pre 88. All experiments are conducted on a server with GPU RTX 4090 and CPU AMD EPYC 7543 32-Core Processor. As different methods adopt different learning paradigms for backdoor mitigation, we measure the average runtime for each defense method to take effect on CIFAR-10 10 and Tiny ImageNet (ASR < 5% and ACC > 85% on CIFAR-10, 10, or ASR < 5%, ACC > 45% on Tiny ImageNet). Note that if defend method cannot reach the criteria for "Take effect", the maximum runtime reported. The results are summarized in the following table.
Table 7: Time to take effect for different methods, averaged over different attacks
Dataset
ANP [55]
FP [31]
NC [47]
[28]
[64]
i-BAU [59]
SAU (Ours)
CIFAR-10
289.86s
266.57s
825.86s
75.14s
65.71s
47,00s
43.86s
Tiny ImageNet
1086.50s
318.30s
27359.70s
227.01s
141.67s
621.33s
262.60s
From Table 7, SAU is faster than most of the existing methods, except for NAD and EP, which shows that SAU is an efficient and effective defense method.
B.2 Comparison with Certified Robustness methods
In backdoor defense, methods can be categorized into two classes: empirical methods which aim to develop effective and efficient methods for backdoor defense, and certified methods which aim to build provable framework for backdoor robustness. As our method an empirical method, we would like to point out that our method is different from the existing works on certified robustness in several aspects, which makes them incomparable or impractical to apply to our setting. Specifically:
Efficiency and scalability. Certified methods usually have very high computation costs and are infeasible to apply to large-scale deep neural networks, while our method is efficient and scalable. For example, the representative methods RAB [51] and BagFlip [61] rely on voting over 1000 smoothed models for prediction and are only feasible for some simple neural networks and simple classification problems. To apply these methods to setting, we would need to train 1000 PreAct-ResNer models on CIFAR- 10 from scratch, which would take over 600 hours (25 days) and 620GB of space for single attack server with one RTX 3090.
Applicability to model architecture. Many certified methods are restricted to some simple or classical models, while method is general and applicable to any deep neural network
21

---

architecture. For example, the method in [20] is designed for only k-Nearest Neighbors (kNN) and radius Nearest Neighbors (rNN) models with some feature extractor and cannot be applied to modern deep neural networks.
Difference in the threat model. Most certified methods focus on the threat model where the training dataset accessible to defenders while our method focuses on the threat model where only poisoned model and few clean samples are given.
Therefore, we believe that our method is more practical and effective for defending against backdoor attacks in deep neural networks than the existing works on certified robustness from the perspective of efficiency, scalability, and model structures.
B.3 Comparison with other AT based methods
Here, we compare the proposed method SAU with other representative Adversarial Training based methods from the following aspects:
Threat Model: whether the training dataset is accessible and whether the extra clean dataset is accessible.
Types of Perturbation: whether universal adversarial perturbation is supported.
Types of Adversarial Attack: whether Targeted Adversarial Attack is supported.
The answer "Yes" and "No" are indicated by / and x, respectively. The "Optional" means the method supports but not limited specific answer. We summarize the result in Table 8.
Table 8: Comparison between SAU with other AT based methods
Method
Training Dataset
Extra Clean Dataset
UAP
Targeted Attack
AT from Scratch [52]
X
X
Composite AT [13]
X
X
i-BAU [59]
V
X
PBE [35]
X
X
AFT [35]
x
x
SAU (Ours)
Optional
Optional
We remark that two methods, i.e., PBE and AFT are proposed in Mu et al. [35] for different threat models, where PBE is designed to filter clean dataset from training dataset, and AFT is to perform AT on the given clean datasct. Therefore, AFT is equivalent to the Vanilla AT in the Figure and Section 4.3.
As summarized in Table the current AT-based methods can be roughly categorized into two types based on their threat model: in-training (given training dataset) and post-training (given poisoned model and extra clean dataset), and the difference threat model make methods from different categories incomparable.
Comparison with In-training AT-based methods. Although SAU is not comparable with the in- pe-training AT-based method due to the underlying assumption of the threat model, we would like to adopt different threat models for different methods highlight the advantages of SAU compared to the in-training AT-based methods. Therefore, we compare SAU with the latest SOTA in-processing bT-be method, i.c., Composite AT (CAT) [13].
We first highlight the following differences between CAT and SAU:
SAU is theoretically motivated by the relationship between adversarial robustness and back- door risk, while CAT is empirically driven by some experimental observations. Therefore, our method has a theoretical guarantee for the performance against various backdoor attacks, while CAT does not.
SAU fine-tunes a backdoored model with small clean dataset, while CAT trains a backdoor- free model from scratch on poisoned dataset. Therefore, SAU is more efficient than CAT.
22

---

Then, we conduct experiments on defending against backdoor attacks and summarize the results in Table 9 and 10.
Table 9: Experiments Results with poisoning ratio 1%
Method
No Detense
No Defems
No Delense CCT [13]
CAT [13]
CAT [13]
SAU (Ours) SAU (Ours)
SAU (Ours)
Attack
Acc
ASR
R-ACC
Acc
ASR
R-ACC
Acc
ASR
R-ACC
BadNets [15]
93.14
24.73
24.24
75.52
2.50
74.73
91.25
0.94
91.16
Impur-Aware
91.74
79.18
19.89
75.24
72.48
23.97
92.20
3.63
84,06
SSBA [30]
24.89
71.40
91.34
WaNet [38]
90.65
12.63
79.94
75.21
2.68
74.46
91.84
1.23
89.89
Average
60.00
37.24
61.14
91.66
1.65
88.39
Table 10: Experiments Results with poisoning ratio 10%
Method
No Defense
No Defense
No Defense
CAT
CAT [13]
CAT [15]
SAU (Ours)
SAU (Ours)
SAU (Ours)
Attack
Acc
ASR
R-ACC
Acc
ASR
R-ACC
Acc
ASR
R-ACC
BadNets [15]
91.32
95.03
4.67
74.42
92.49
6.21
89.31
1.53
88.81
Input-Aware
90.67
98.26
1.66
74.21
96.81
2.88
1.27
88.54
SSBA [10]
92.88
97.8
1.99
74.29
28.29
57.49
90.84
1.79
85.83
[38]
Average
91.53
95.21
4.52
74.39
55.61
34,91
92.54
1.78
85.72
The experimental results show that our method is superior to CAT, as evidenced by the following aspects:
SAU achieves significantly higher accuracy (ACC) and lower attack success rate (ASR) than CAT on both poisoning ratios.
SAU is more robust different poisoning ratios than CAT. contrast, CAT fails on a high poisoning ratio (10%) (also observed in Table XI of Gao et al. [13]).
2. Comparison with Post-training AT-based methods. The most related methods to SAU are the post-processing AT-based method including I-BAU and AFT, and we highlight the following differences between them:
I-BAU is proposed under the assumption that the same trigger are used for all poisoned samples which may not be true for more advanced attacks, while SAU shown to effective for attacks with multi-trigger and sample-specific trigger theoretically and empirically.
AFT, which is to perform AT on clean dataset, is motivated by some empirical observation, while SAU is motivated by novel analysis on the relation between adversarial examples and poisoned samples.
Besides, the experimental comparison between SAU and I-BAU Section 4.2 and the experimental between SAU and AFT (Vanilla AT on Clean Dataset) in Section 4.3 show that SAU outperforms I-BAU and AFT by a large margin.
B.4 Choice of perturbation and generalization ability of SAU
In this section, discuss the choice of perturbation set and show that the proposed method, i.e., SAU, can generalize to attacks with excessive-magnituee triggers.
1. Type of adversarial perturbation (AP) and perturbation set. The theoretical analysis and the proposed method SAU is independent of the AP type and AP set (not only Lp ball). For sample space X. define the set G {I( A) 2/Ø X'} and the general perturbation set 5 = (T(z, z/c e X) for adversarial attack with learnable parameter e E Ω such that an adversarial example can be generated by & T(x, e). Then, the theoretical analysis and proposed method can be applied C S, where the AP can be any type, such as additive AP, Spatially Transformed AP [56], or Perceptual AP [25], etc.
23

---

2. Effect of perturbation set S. The key of SAU is to consider Shared Adversarial Example (SAE) as surrogates for poisoned samples. Given models, the set of SAE determined by S: larger leads to larger set of SAE, and vice versa Therefore, the perturbation set S plays an important role in the effectiveness of SAU. Specifically, adopting a larger S can cover unknown triggers but also increases the chance of picking SAE which is irrelevant to poisoned samples, and may weaken the defense performance. Meanwhile, some poisoned samples may be beyond the scope of SAE if smaller S is adopted.
Generalization excessive-magnitude trigger. In practice, we find that the proposed method SAU is effective for various attacks, even if Assumption does not hold. For example, SAU with L norm bound 0.2 can still effectively mitigate SSBA attack whose average Loo trigger norm is much larger than 0.2, revealing the generalization of SAU to attacks with excessive-mannitud trigger.
Table SAU for SSBA Attack
Dataset
L
L2
Loo
ACC
ASR
R-ACC
CIFAR-10
108.33 3.29
0.33
90.99
0.58
87.04
Tiny
581.65
8,26 0.43 51.85
0.11
36.36
To investigate the SAU's generalization ability, we conduct experiments BadNets with various strengths. Specifically, for BadNets with 3x3 patched trigger, we alter the pixel value of the trigger from 0.2 1.0.
Table 12: Defend against BadNets [15] Attack with Various Pixel Value
No Defense
No Defense
ANP [55]
ANP [55]
SAU (Ours)
SAU (Ours)
Pixel
L
ACC
ASR
ACC
ASR
ACC
ASR
0.2
0.42
91.69
95.60
91.30
2.64
90.01
1.38
0,4
0.31
91.61
97.79
91.44
1.00
88.43
1.59
0.6
0.35
91.94
96.43
91.72
7.81
90.49
1.30
0.8
0.48
91.72
93.20
91.06
2.37
89.23
1.36
1.0
0.67
91.32
95.03
90.88
4.88
89.31
1.53
Table 12 shows the results for defending various BadNets attacks by SAU following the setting in Section 4.2, from which we can find that SAU can consistently mitigate the backdoor even when the trigger is beyond the perturbation set. One reasonable explanation is that although the poisoned samples are not included in SAEs, there exists SAE that has similar effect as poisoned samples as shown in Figure 4, resulting in strong generalization ability of SAU.
4. Effeet of perturbation type. To explore the effect of perturbation types, we extend SAU to incorporate two other perturbation types and denote the resulting methods by Spatial SAU and Ensemble SAU, respectively. Specifically, for spatial SAU, we employ spatial adversarial attack [56] to generate spatial SAEs by spatial transformation, following the same setting as [13]. For Ensemble SAU, the perturbation set composed of collection of subsets, including L1 ball (bound 500), L2 ball (bound 10), L ball (bound 0.2), and the spatial AP set. Then, for each batch, Ensemble SAU randomly chooses one subset of perturbation to generate the SAFs. We compare SAU, Spatial SAU and Ensemble SAU, following the setting in Section 4.
Table 13: Defend against BadNets [15] Attack with Various Pixel Value
No Defense No Defeuse
SAU SAU
Spatial SAU
Spatial SAU
Ensemble SAU
Ensemble SAU
Dataset
Attack
ACC
ACC
ASR
ASR
CIFAR-10
BadNets [15]
91.32
95.03
89.31
1.53
88.27
0.97
88.1 16
0.67
CIFAR-10
SSBA [30]
92.88
97.86
90.84
1.79
89.65
4.43
89.92
0.81
CIFAR-10
WaNet [38]
91.25
89.73
91.26
1.02
90.21
4.34
89.61
0.66
ImageNet
BaiNNets [15]
56.23
100
51.52
0.53
50.68
99.81
49.32
01.89
Tay ImtgeNet
SSBA [30]
55.22
97.71
51.85
0.11
49.79
79.72
49.73
0.05
Tmy ImageNet
WaNet [38]
56.78
54.65
54.65
85.75
50.12
0.7
51.14
0.15
24

---

Polsoned Sample
Shared Adverser Exammple
Non-Shared Adversorial Example
38.75
80.55
Figure 4: Visualization of Features of Poisoned Sample, Shared Adversarial Example and Non- shared Adversarial Example latent space for BadNets Attack with PreAct-Res Adversarial Example and Adversarial Example L2 distance between the features of the Poisoned sample and Shared Adversarial Example is 38.75, while the 1.2 distance between the features of the Poisoned sample and the Non-shared Adversarial Example is 80.55, showing that the Shared Adversarial Example can still be close to the Poisoned Sample in latent space even if the trigger is beyond perturbation set.
Both Shared generated by PGD-5 with Loo norm 0.2. The
From Table 13, we compare SAU, Spatial SAU and Ensemble SAU, following the setting in Section 4. From the table, find that Spatial SAU works well on CIFAR- 10 dataset but fails to defend BadNets and SSBA attacks on Tiny ImageNet. For WaNet, spatial SAU can effectively defend on both datasets. We remark that the Ensemble SAU achieves the lowest ASR in all experiments. However, as Ensemble SAU has larger perturbation set, its accuracy is lower than SAU by 2%.
In summary, theoretical analysis and the proposed method SAU are flexible to the perturbation set, and SAU can be generalized to attack with excessive trigger norms. Moreover, the Ensemble SAU serves stronger variant of SAU and can be effective for various attacks.
B.5 SAU with generated datasets
Like most other post-processing methods [31, 55, 47, 28, 59], SAU assumes that a small clean dataset is available for conducting defense, which can be obtained by ways such buying from trustworthy data supplier, using modern generative models, collecting from the internet, or applying some data cleansing techniques.
In this section, we explore a more practical case for defense, i.e., using the generated dataset, to evaluate our method. Specifically, we conduct experiments on synthetic datasets and compare SAU with NAD. We test both methods with a synthetic dataset CIFAR [36] which provides generated CIFAR-10-like images whose distribution is close but not identical to CIFAR-10, and can be regarded as OOD data. We build mixed datasets by randomly picking samples from CIFAR- 10 and CIFAR-5m with mixed ratio, and evaluate both methods on the mixed datasets. A larger mixed ratio indicates more synthetic data in the mixed dataset. The experiment results are summarized in the following tables.
From Table 14, we can find that all methods can work on the mixed dataset but as the mixed ratio increase, the accuracy (ACC) may decrease due to the distribution difference between the two datasets. It's notable that the model distillation-ban quality of the dataset most, with a reduction of 4.91 % in ACC, while SAU is quite robust to the data quality. Another interesting phenomenon is that the ASR of ANP decreases when more synthetic data used, which shows that the distribution of data may influence the performance of backdoor defense.
method NAD influenced by the
25

---

Table 14: Results on Generated Dataset
ANP [55]
ANP [SS]
NAD [28]
NAD [28]
SAU (Ours) SAU (Ours)
Mixed Ratio
ACC
ASR
ACC
ASR
ACC
ASR
0.0
90.88
4.88
89.87
2.14
89.31
1.53
90.68
87.96
1.88
89.83
1.63
90.21
87.69
1.88
89.36
1.60
90.31
86.69
2.19
1.66
0.8
89.37
0.56
85.90
1.86
88.13
1.79
88.74
84.96
2.48
88.32
1.01
All those observations motivate conduct research on the relationship between data quality and backdoor defense in the future.
B.6 Resistance to defense-awarr//daattie Attacks
When the attacker is aware of the defender's adversarial training strategy, the attacker train the backdoored model using Adversarial Training (AT) on the poisoned dataset, such that the model can be resistant to adversarial attack, hindering the generation of shared adversarial examples.
To evaluate the performance of SAU against such defense- aware backdoor attacks, we conduct experiments on the CIFAR-10 dataset following the settings in Section 4.2. For AT on the training dataset, we use PGD-10 attack with Lex norm 8/255, which is commonly used in adversarial training literature.
Table 15: Experiments Results for Defense-aware Attack
No Defense
No Defense
SAU (Ours) SAU (Ours)
Attack
ACC
ASR
ACC
ASR
BadNets [15]
79.89
93.46
79.83
2.39
Input-Aware
79.87
96.60
78.64
4.84
SIG [2]
73.50
99.50
76.41
0.10
Average 77.75
96.52
78.30
2.44
From Table 15, we can find that SAU can achieve an average ACC of 78.3% and ASR of 2.44%, showing its resistance to such adaptive attacks.
C Experiment details
In this section, we provide the experiment details. In our experiments, all baselines and settings are adapted from BackdoorBencl [54], and we present the experiment details in this section.
C.1 Attack details
BadNets [15] is one the earliest works for backdoor learning, which inserts a small patch of fixed pattern replace some pixels in the image. We use the default setting in BackdoorBench.
Blended backdoor attack (Blended) [7] uses blagha-beending strategy to fuse images with fixed patterns. We set a 0.2 as the default in BackdoorBenchh We remark that since such large produces pisuallperceptible change to clean samples, the Blended Attack in this setting is very challenging for all defense methods.
Input-aware dynamic backdoor attack ([nput-Awaree]37] is cantroggcontrollaale attack that learns a trigger generator to produce the sample-spectn trigger in the training process of the model. We use the default setting in BackdoorBench.
Low-frequency attack (LF) [58] uses smoothed trigger by filtering high-frequency artifacts from UAP. We use the default setting in BackdoorBench.
26

---

Sinusoidal signal backdoor attack (SIG) [2] is clean-label attack that uses sinusoidal signal as the trigger to perturb the clean images the target label. We use the default setting in BackdoorBench.
Sample-specific backdoor attack (SSBA) [30] uses an auto-encoder to fuse a trigger into clean samples and generate poisoned samples. We use the default setting in BackdoorBench.
Warping-based poisoned networks (WaNet) [38] is also training-controllan attack that uses a warping function to perturb the clean samples to construct the poisoned samples. We use the default setting in BackdoorBench.
C.2 Defense details
The details for each defense used in our experiment/diiccusion are summarized below:
ANP [55] is a muming-baseed method that prunes neurons sensitive weight perturbation. Note that in BackdoorBench, ANP can select its pruning threshold by grid searching on the test dataset (default way in Becch)orBenchh a given constant threshold. To produce fair comparison with other baselines, we use a constant threshold for ANP and set the pruning threshold 0.4, which is found to produce better results than the recommended constant threshold of 0.2 in BackdoorBench. All other settings are the same as the default setting in BackdoorBench.
FP [31] is pruning-basee method that prunes neurons according their activations and then fine-tunes the model to keep clean accuracy. We use the default setting in BackdoorBench.
NAD [28] uses Attention Distillation to mitigate backdoors. We use the default setting in BackdoorBench.
NC [47] first optimizes a possible trigger to detect whether the model is backdoored. Then, if the model is detected as backdoored model, it mitigates the backdoor by unlearning the optimized trigger. We use the default setting in BackdoorBench. We remark that if the model detected as a clean model, NC just returns the model without any changes.
EP [64] uses the distribution of activation to detect the backdoor neurons and prunes them to mitigate the backdoor. We use the default setting in BackdoorBench.
i-BAU [59] uses adversarial training with UAP and hyper-gradient to mitigate the backdoor. We use the default setting in BackdoorBench.
SAU (Ours) uses PGD to generate shared adversarial examples and unlearns them to mitigate the backdoor. For all experiments, we set A = X = X and X3 = 0.01 in (13). Then, we run PGD 5 steps with Loc: norm bound 0.2 and large step size 0.2 to accelerate the inner maximization in (8). For unlearning step (fine-tuning on D., and SAEs), we use the same setting as i-BAU and fine-tuning (FT) in BackdoorBench. We run SAU 100 epochs in CIFAR-I0 and GTSRB. In Tiny ImageNet, we run SAU 20 epochsNote that the number of epochs used in our experiment largely exceeds the necessary epochs for SAU to take effect and can be further reduced to save computational costs.
Vanilla Adversarial Training (vanilla AT) [33] use PGD to generate vanilla adversarial examples and unlearn them to mitigate the backdoor. In all experiments, vanilla AT uses the same setting as SAU, including the step size, norm bound, optimizer, and learning rate, except for the experiment in Figure where we train vanilla AT 100 epochs on Blended Attack with PreActtRReseei 18 and Tiny ImageNet to show its long-term performance.
Adversarial Fine-tuning (AFT) [35] performs adversarial training on clean dataset, which is equivalent vanilla AT.
Composite Adversarial Training (Composite AT, CAT) [13] use perturbation from different types to defend against backdoor attacks. We request the official implementation of CAT from the authors and use the recommended setting in the original paper.
Progressive Backdoor Erasing (PBE) [35] uses the adversarial attack to progressively filter the clean dataset from the training dataset and remove the backdoor. Since the code haven't been released yet and the details of some critical parameters such as the configuration adversarial attacks, the data filtering ratio in each step, and the learning rates are not covered in the paper, we failed to produce satisfactory results. Besides, the threat
27

---

model of PBE is different from ours which makes it incomparable to SAU, and therefore, we do not include the experiment section.
Adaptation to batch normalization. Batch normalization (BN) has been widely used in modern deep neural networks due to improved convergence. However, recent works show that is difficult for BN to estimate the correct normalization statistics of mixture of distributions of adversarial examples and clean examples [57, 48]. Such problem is magnified when we adversarially fine-tune model with a small set of samples and may destroy the generalization ability of the model due to biased BN statistics. To address this problem, we propose the fixed BN strategy in the adversarial unlearning/tranning process. Note that in all experiments, the fixed BN strategy is applied i-BAU, SAU, and vanilla AT when BN is used in the model architecture like PreAct-ResNetle
Experiments on VGG19. For VGG19 [41], BN is not used. Therefore, the fixed BN strategy not applied. Moreover, some methods that depend on BN, such as ANP and EP, are not applicable to VGG19.
D Additional experiments
In this section, we provide additional experiments.
D.1
Main experiments on VGG19
Table 16: Results on CIFAR-10 with VGG19 and poisoning ratio 10%.
Defense
No Defense
FP [31]
NC [47)
Attack
ACC
ASR R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
BadNets [1S]
89.36
95.93
N/A
89.23
92. 92.61
6.82
51.6
86
88.01
96.72
Blended [7]
90.17
99.12
0.82
N/A
90.07
99.11
0.82
49.96
85.92
1.79
74.13
96.54
Aware [37]
77.60
94.59
4.79
N/A
86.77
11.79
53.91
77.67
94.58
4.79
[F[58]
88.91
93.93
5.62
N/A 88,98
91.8
7.46
51.07
85.35
9.99
72.79
90.18
[2]
81.60
99.8
N/A 84.52
99.93
0.07
50.0
$1.69
90.80
0.12
50.00
SSBA [30]
89.48
91.86
7.29
N/A
89.40
89 66
9.22
51 61.06
89.48
91.86
7.29
50,00
WaNet [38]
88.43
88.9
10.3
N/A
89.61
73.39
21.57
77.76
88.89
10.30
50.01
Average
86.54
94.88
4.68
N/A
87.20
90.47
8.68
52.19
85.20
55.42
36.78
69.06
Defense
NAD [28]
i-BAU [59]
SAU (Ours)
Attack
ACC
ASR
R R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R RACCC
DER
BadNets [15]
88.51
38.17
58.30
77.96
87.
25.
63.24
84.34
86.71
0.08
32.77
96.60
Blended [7]
88.35
93.08
52.11
88.61
59.86
32.31
68.85
87.01
23.74
Input-Aware [37]
75.70
23.36
54.71
84.62
72.15
26.22
42.51
81.41
75.
12.49
46.19
89.76
I.F [F[58]
87.85
59.38
35.01
66.73
87.97
79.10
18.04
56.93
86.65
6.43
72.51
SIG [2]
86.01
99.18
0.77
50.31
85.06
90.89
5.27
54.46
84.87
0.11
10.82
99.84
SSBA [30]
87.65
37.54
51.58
76.21
88.08
3.74
76. 98
93.36
87.3
1.66
78.70
94.01
WaNet [38]
90.82
44.93
51.18
71.98
90.31
25.
87
81
87.07
5.32
83,37
91.11
Average
86.27 56.52
37.27
68.56
85.71
48
43.75
74.41
84.96
3.92
49.73
94.46
This section provides additional experiment results on VGG19 with CIFAR- (Table 16) and Tiny ImageNet (Table 17) supplement Section 4. Table 16 shows that SAU outperforms all baselines on CIFAR-10 10 and VGG19, with a 40.56% decrease of ASR compared to the second lowest ASR. Moreover, SAU achieves the highest DER in 6 of attacks, demonstrating a significantly better tradeoff' between accuracy and ASR. It also achieves the top-2 R-ACC in 5 of 7 attacks and the best average R-ACC, indicating its good ability to recover the prediction of poisoned samples. Table 17 shows that SAU achieves the best DER in 5 of 6 attacks and the top-2 lowest ASR in 5 of attacks. Although SAU only achieves the second- best average ASR in Table 17, it has significantly higher accuracy than NC, which achieves the best average ASR.
D.2 Main experiments on GTSRB
This section provides additional experiment results on GTSRB with PreAct-ResNer188 (Table 18) and VGG19 (Table 19) to supplement Section 4. In both tables, SAU achieves the top-2 lowest ASR in of 6 attacks and the lowest ASR on average. Furthermore, SAU achieves the top-2 highest DER in of 6 attacks for PreAct-Res (Table 18) and of 6 attacks for VGG19 (Table 19).
28

---

Table 17: Results Tiny ImageNet with VGG19 and poisoning ratio 10%.
Defense
No Defense
FP [31]
NC [47]
Attack
ACC:
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
BadNets [1S]
12.26
99.99
0.0
N/A
40.70
99.65 09.63
0.20
19.1
26.47
0.53
25.93
91.33
Blended [7]
43.5
99.32
0.32
N/A
41.82
98.67
0.71
49.48
35.93
0.03
18.87
95.86
Input-Aware [37]
16.36
99.18
0.38
N/A
45.95
99.02
0.69
50.03
38.51
0.45
36.00
95.59
LF [58]
43.14
95.41
NA
85.
54.34
10.35
5.37
81.02
SSBA [30]
41.67
96.57
1.71
N/A
40.04
69.27
9.10
62.84
28.23
0.21
22.86
91.46
WaNet [38]
43.6
99.85
0.09
N/A
99.31
0.46
49.9
8.14
7.13
Average
43.42
98.44
0.8
N/A
42.16
91.85
2.83
52.28
24.64
0.53
19.36
83.91
Defense
NAD [28]
i-BAU [59]
SAU (Ours)
Attack
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACCC
ASR
R-ACC
DER
BadNets [15]
37.68
96.40
2.02
19.5
10.10
100.0
0.0
18.92
38. 25
0.066
14.32
97.96
Blended [7]
38.50
95.98
1.64
49.21
40.11
98.88
0.47
48.52
38.24
0.01
6.10
97.02
Input-Aware [37]
36.14
2.79
31.87
93.23
11.49
58.66
18.55
69:17
12.87
6.91
35.15
94.52
LF [58]
38.06
81.10
7.40
54.62
86.89
6.05
32.8
39.15
15.46
20.31
87.98
SSBA [30]
36.27
84.86
5.78
53.15
38
93.93
87.81
4.96
53.01
36.31
0.32
5.98
95.45
WaNet [38]
34.80
63.91
13,98
63.61
39.87
0.62
48.69
37.04
20.02
95.38
Average
36.94
70.84
10.45
59.05
40.61
88 49
5.16
53.06
38.64
4.22
17.48
88.33
Table 18: Results on GTSRB with PreAct-ResNetlh and poisoning ratio 10%.
Defense
No Defense
ANP
[31]
NC [47]
Attack
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
BadNets
97.24
59.25
96.79
98:21
0.09
70.93
79.58
97.48
0.01
97.45
79.62
Bleended [7]
98.58
99.99
0.0
N/A
98.75
99.82
0.18
50.08
08.38
100.0
0.0
49.9
97.76
8.03
59.74
95.57
Input-Awure [37]
97.26
92.74
7.23
N/A
99.14
0.00
96.09
96.37
98.08
2.32
88.91
95.21
98.55
0.01
96.71
96.56
LF [58]
97.93
99.57
0.12
NIA
97.80
81.38
16.45
59.03
97.59
99.7
0.27
19.83
97.97
1.31
77.37
99.11
SSDA [30]
97.98
09.56
0.49
N/A
97.86
98.73
1.13
50.36
97.75
99.46
0.45
49.94
97.72
0.29
87.30
99.50
[38]
97.74
5.39
0.00
97.12
97.62
53.03
Average
97.79
90.89
3.02
N/A
98.07
46.68
51.29
68.91
97.94
64.94
28.21
61.07
37.96
1.61
86.10
88.18
Defonse
NAD [28]
EP [64]
[-BAU [59]
SAU (Ours)
Attack
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ASR
R-ACC
DER
BadNets [15]
98.69
0.63
98.00
79.31
97.69
10.91
88.22
74.26
96..03
0.02
96.07
79.01
98.0
0.01
97.92
79.62
Blended
98.61
100.0
0.00
50.0
97.9
100.0
0.0
49.66
95.65
86.45
6.52
35.31
98.73
3.41
34.42
97_36
Inpul-Aware [37]
98.27
40.65
58.68
76.04
98.5
49.19
49.5
71.62
97.93
1.94
93.58
95.4
98.09
0.01
93.02
96.36
LF [58]
98.14
51.83
10.02
73.87
97.59
98.2
0.72
50.02
95.59
17.3
32.16
89.97
96.08
0.37
16.89
95. 68
SSBA [30]
97.95
99.39
97.66
1.27
58.10
0.45
40.33
WaNet [38]
98.61
0.56
97 63
96.81
96.96
8.97
82 93
92
96.92
0.11
96.25
96. 66
97.86
0.02
97.12
97. 11
Average
98.38
48.84
49.15
08.02
97.72
61.21
02.07
17.85
63.78
80.60
0.71
63.33
88.31
D.3 Experiments with 5% poisoning ratio
This section reports the experiment results for comparing SAU with basclines with poisoning ratio of 5% on CIFAR-10 and Imageee As with the results for a poisoning ratio of 10%, SAU achieves the top-2 lowest ASR in most of the attacks. Tables 20 and 21 show that SAU achieves the lowest average ASR. In summary, SAU still outperforms other baselines for a poisoning ratio of 5%.
D.4 Learning curves for Shared Adversarial Risk
This section presents plot of the learning curves for shared adversarial risk to illustrate the rela- tionship between the backdoor risk (measured by ASR) and the shared adversarial risk. For each plot, we run SAU CIFAR-10 with The SAR is computed on the test dataset by computing the shared adversarial example using the inner maximization step in (8). Figure demonstrates that SAU effectively reduce the shared adversarial risk, and thus, mitigate the backdoor risk. We note that in order to satisfy Assumption 1, the perturbation set adopted by SAU is much larger than the perturbation budget in the adversarial robustness area, resulting in a high adversarial risk in Figure 5.
and poisoning ratio of 10% for 10 epochs.
D.5 Experiments on different PGD configurations
In this section, we test SAU using different PGD configurations to generate the shared adversarial examples. Specifically, we consider the following configurations for PGD:
* Config I: PGD with Loo norm bound 0.2, step size 0.2, and steps.
29

---

Table 19: Results on GTSRB with VGG19 and poisoning ratio 10%.
Defense
No Defense
FP [31]
[47]
Attack
ACC:
ASR R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
BadNets [1S]
96.25
57.11
1.83
N/A
97.02
2.89
18.64
77.11
96.86
0.00
96.64
78.56
Blended [7]
95.98
99.86
0.04
N/A
96.48
99.79
0.08
50.03
95.32
20.80
38.37
89.20
Input-Aware [37]
96.03
76.69
22.35
N/A
97.08
24.09
13.09
76.3
96.
0.02
94.22
88.33
LF [58]
N/
98.
93.25
67.61
SSBA [30]
96.43
99.31
0.43
N/A
96.45
99.0
0.66
50.16
96.56
1.03
79.49
99.14
WaNet [38]
95.27
92.09
N/A
96.90
68.48
96.84
94.67
Average
95.84
87.31
6.01
N/A
96.54
65.4
20.49
59.39
95.92
3.83
78.50
85.56
Defense
NAD [28]
i-BAU [59]
SAU (Ours)
Attack
ACC
ASR
R-ACC
DER
ACC
ASR
RAACC
DER
ACC
ASR
R-ACC
DER
BadNets [1S]
96.78
73.52
26.73
50.0
95.81
0.01
911.70
78.
96.21
0.02
79.69
78.53
Blended [7]
96.60
96.27
0.55
50.79
95.11
0.79
16.55
99.10
95.45
1.79
12.61
98.77
Input-Aware [37]
97.77
53.03
-1..91
61.83
96.92
10.06
72.08
83
96.48
0.01
11.34
88.34
LF [58]
50.68
22.45
95.17
SSBA [30]
96.38
98.38
0.95
50.44
94.57
2.55
79
97 45
95.92
7.44
31.49
95.68
WaNet [38]
30.34
57.96
76.38
92
97.04
5.80
85.
Average
96.62
76.6
22.26
56.73
95.80
3.42
58.21
N5.72
96.04
2.53
39.85
86.26
Table 20: Results on CIFAR-10 10 with PreAct-ResNedt 8 and poisoning ratio 5%.
Defense
No Defense
ANP [S5]
FP [31]
NC NC1471
Artack
ACC
R-ACC
DER
R-ACC
ACC
ACC
R-ACC
BadNets [15]
92.64
88. 74
10.78
N/A
12.38
3.10
90.13
92.60
92.47
17.47
77-9
$5.55
80.89
1.17
89.70
92.41
[7]
0.39
50.63
Iaput-Aware [37]
91.52
90.2
8.91
N/A
91.64
1,88
85.00
94.56
91.89
52.74
44.49
68.73
91.51
90.21)
8.91
50.00
LF [58]
93.35
98.03
1.86
NIA
92.60
96.98
2.87
50.15
93.14
98.37
1.53
93.35
98.03
1.86
50.00
SIG [2]
93.64
97.09
2.9
N/A
93.09
92.38
7.39
32.08
93.27
99.79
0.2
49.81
93.65
97.09
2.90
50.00
SSBA [30]
00.27
94.91
4.82
N/A
93.13
52.66
43.68
71.06
93.05
87.14
12.09
53.77
91.04
0.80
87.93
95.94
WaNet
N/A
91.76
Average
92.84 93.41
6.16
N/A
92
00
19.50
45.90
71.75
92.77
68.64
28.69
62.53
92.12
67. 67.49
29 31
62.62
Defense
NAD [28]
EP [64]]
I-BAU [59)
SAU (Ours)
Attack
R-ACC
BudNets [15]
91.00
4.73
87.93
91.2
91. 04
0.98
91.26
93.08
58.8
2.51
87.88
91.20
90.54
1.36
90.112
92.64
[7]
5.64
67.56
Amput-Aware [37]
93.07
97.12
2.68
50.0
90.59
1.51
85.09
93.88
91.34
7.63
80.11
91.19
91.12
2.03
82.11
93.88
LF 158]
92.98
91.23
5.3
51.72
91.65
83.71
15.22
36.31
87.25
34.61
41.31
78.66
91.31
2.59
78.93
96.70
SIG [2]
92.49
2.93
19.48
92.1
1.66
60.29
97.00
86.44
5.73
18.86
92.08
90.87
0.97
51.98
96.08
SSBA [30]
92.
88.63
10.58
52.75
92
12.14
78.70
90.89
87.15
3.86
78.57
97.47
91.15
1.43
85.87
95.65
WaNet
16.16
3.58
Average
92.64
75.88
22.4
38.96
01.63
33.01
56.35
79.61
88.44
13.50
68.28
57.02
91.13
2.51
78.30
94.61
. Config 2: PGD with L. norm bound 0.1, step size 0.1, and steps.
Config 3: PGD with Loo norm bound 0.2, step size 0.1, and steps.
Config 4: PGD with La norm bound 5, step size 0.2, and 5 steps.
Config 5: PGD with L, norm bound 300, step size 0.2, and 5 steps.
The experiments are conducted for CIFAR-10 with PreAct-ResNet18 and poisoning ratio 10%. We summarize the experiment results Table 22, from which we can find the SAU can effectively mitigate backdoors under various configurations of PGD.
30

---

Table 21: Results on Tiny ImageNet with PreAct-ResNet18 and poisoning ratio 5%.
Defense
No Defense
ANP [55]
FP 1311
NC [47]
Atlack
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
NedNees, [15]
66.36
99.96
0.13
NIA
44.54
0.02
43.50
94.01
13.00
96.83
2.86
49.53
52.97
65.94
25.45
65.26
Blended [7]
56.69
98. 75
0.94
N/A
15.19
92.89
3.67
47.18
62.96
94.17
3:21
50.28
53.63
96.32
2,21
19.59
Input-Aware [37]
67.87
98.25
1.51
N/A
49.38
0.17
48.84
94.90
56.27
89.87
6.52
53.39
56.48
0.49
53.17
98.18
LF [58]
2.75
N/A
17:21
71.05
53.11
51.49
67.51
SSBA [30]
56.86
95.69
3.06
N/A
44.93
50.44
20.23
06.66
13.26
79.97
10.63
56.06
$4.04
0.04
38. 80
96.41
WaNet [58]
56.83
98.36
1.21
NIA
29.42
0.03
27.29
85.46
54,18
93.36
51.18
53.02
0.27
50.56
97.14
Average
56.88
97.82
1.6
N/A
12
30.34
26.84
72.74
53
90.22
51.94
53.00
38
31.13
74.03
Defense
NAD [28]
EP [6:1
i-AU [59]
SAU (Ours)
Attack
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
BudNetts [15]
45.51
34.63
34.01
77.19
51.58
0.10
$1.06
97.49
53.60
96.58
3.17
50.26
51.94
0.78
50.78
97.33
Blended [7]
46.72
88.89
4.96
49.94
53:13
82.26
8.68
36.47
52.76
92.01
4.7
61.41
02.47
3.40
24.90
95.57
Inpus-Aware [37]
1.04
45.83
94.03
57.09
51.27
98.27
56.13
36.90
79.76
53.92
0.11
50.89
97.10
LI [58]
46.99
63.9
14.66
61. 18
54,04
$3.50
10.06
54.78
52.45
90.81
5.77
50.48
52.57
14.27
23.78
73.81
SSBA [30]
46.65
42.45
24.65
71.51
53.87
52.07
23.44
70-31
52.65
90.72
50.38
53.57
0.02
39.77
96.19
WaNet [38]
48.81
2.44
39.54
92.95
51.98
0.12
$2.62
97.70
52.00
93.43
4.91
50.55
50.15
81.75
12.37
54.97
Average
46.89
38.80
27.27
70.97
53.95
36.54
32.84
76.00
53.43
83.42
10.9
54.60
02.44
21.72
33,75
80.71
BadNets
SSBA
80-
Percentage
60
Accuracy
Accuracy
Accuracy
Sackddoor
Bachdoor
Backdoor
40-
Advenartal Risk
Adversortal Rick
40
Advernrtal Riok
Shered Advernarial Rikk
Shared Adversatial Rick
Shared Adverariia Rok
20-
20
20
01
10
4
10
Epochs
Epochs
Epochs
Figure 5: Learn curves for Accuracy, Backdoor Risk, Adversarial Risk, and Shared Adversarial Risk.
Table 22: Results for SAU with different PGD configurations
Defense
No Defense
Config
(Default)
Config 2
Attack
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
DER
BadNets [15]
91.32
95.03
4.67
N/A
89.31
1.53
88.81
95.74
89.56
0.80
89.60
96.24
Blended [7]
93.47
99.92
0.08
N/A
90.96
6.14
64.89
95.63
91.05
6.06
61.11
95.72
Input-Aware
90.67
98.26
1.66
N/A
91.59
1.27
88.54
98.49
90.93
1.56
81.76
98.35
LF [58]
93.19
99.28
0.71
N/A
90.32
4.18
81.54
96.12
90.83
3.76
77.93
96.58
SIG [2]
84.48 96. 27
1.72
N/A
88.56
1.67
57.96
98.30
88.29
0.94
49.41
98.66
SSBA [30]
92.88
97.86
N/A
1.79
85,83
97.01
89.81
80 29
96.58
WaNet [38]
91.25
89.73
9.76
N/A
91.26
1.02
90.28
94.36
86.22
3.12
86.89
90.79
Average
91.01
2.94
N/A
90,41
2.51
79.69
96.52
89.33
2.55
75.71
96.13
Defense
Contig
Config
Config 5
Attack
ACC
ASR
R-ACC
DER
ACC
ASR
R-ACC
ACC
R-AOC
DER
BadNets [15]
89.69
2.21
89.19
95.60
89.78
1.11
89.38
96.19
88.01
1.29
88.63
95.73
Blended [?]
90.98
6.12
61.64
95.65
91.58
8.31
59.09
94.86
91.49
5.83
63.04
96.05
Inpul-Aware
90.75
1.86
86.21
96.20
91.30
0.94
84.24
98.66
91.21
1.20
86.60
98.53
L.F [S8]
91.05
3.00
81.23
97.07
87.06
1.94
73.72
95.60
91.29
5.12
80.26
96.13
SIG [2]
87.18
1.32
55.12
98.47
88.20
88
0.78
49.09
98.74
86.69
55.64
95.09
SSBA [30]
90.30
2.68
82.04
96.34
90.65
2.50
84.40
96.52
90.48
1.62
84.19
96.92
WaNet [38]
91.14
1.60
87.
90.79
1.13
94.07
90.55
1.82
93.61
Average
90.17
2.68
77.63
96.45
89.91
2.40
75.66
96.38
90.11
2.71
78.21
96.11
D.6 Experiments on different numbers of clean samples
This section examines the influence of clean sample size for SAU by evaluating SAU with different numbers of clean samples. The experiment conducted on CIFAR-10 10 with PreAct-ResNetI' 8 and the results are summarized in Table 23. Table 23 shows that SAU can consistently mitigate backdoors with sample sizes ranging from 2500 (5% to 500 (1 %) with high accuracy. When the sample size decreases to 50 (0.1 %, 5 samples per class) or 10 (0.02%, sample per class), SAU can still reduce
31

---

the backdoor to low ASR. However, clean accuracy is difficult to guarantee with such limited clean samples.
Table 23: Results for SAU with different numbers of clean samples
Defense
Defense
SAU
-2000
SAU
11000
SAU
SAU
SAU
-10
Atlack
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
BadNets [15]
91.32
95.03
89.31
1.53
89.24
1.31
83.04
2.17
67.86
1.41
51.43
14.38
Blended [7]
93.17
99.92
90.96
6.14
88.82
2.53
87.71
5.41
73.95
11.62
43.51
0.00
LF [58]
93.19
99.28
90.32
88.00
6.79
88.69
66.69
0.30
54.95
12.62
D.7
Experiments on Multi-trigger Attacks
In this section, we evaluate our method for the multi tetii-rrgggermmui-tttaeee target cases. Specifically, we use two different attacks (denoted by Attack-1 and Attack-2) with different target labels. Following Section 4.2, we test our method (SAU) against strong baseline ANP on the CIFAR-10 10 dataset with poisoning ratio of 10% and backbone PreAct-ResNet18 8. The experiment results are summarized in the following table, where we use ASR-1 and ASR-2 to represent the Attack Success Rate for 1 and Attack-2, respectively. From Table 24, we find that SAU achieves much higher accuracy (5.64% higher) and a much lower ASR for A (26.46% lower) compared to ANP, although the average ASR for Attack-2 of ANP is slightly lower than that of SAU (only 0.13% lower). These results show that SAU outperforms the baseline in most cases, which demonstrates the effectiveness and robustness of our method in this challenging scenario.
Table 24: Results for defending against Multi multi-trigger Attacks
No Defense
No Defense No Defense
ANP [55]
ANP [5S]
ANP [55]
SAU (Ours)
SAU (Ours) SAU (Ours)
Attack-1
Attack-2
ACC
ASR-1
ASR-2
Blended [7]
BadNets [15]
90.22
99.27
95.03
85.55
37.07
0.01
$8.93
2.21
0.54
Aware
RadNets [15]
89.71
4.48
LF [S8]
BadNets [15]
90.16
98.08
95.15
84.21
79.11
2.30
88.45
2.93
0.34
SIG
BadNets
82.12
98.48
95.03
76.43
26.92
0.14
85.82
0.67
SSBA [30]
BadNets [15]
89.69
95.13
95.11
83
20.03
0.14
87.60
1.38
0.52
WaNet [8]]
BadNets [15]
89.44
90.21
95.40
82.21
5,53
0.01
$8.02
3.12
0.67
Average
88.56
93.30
95.08
82.31
28.86
0.44
87.95
0.57
D.8 Experiments ALL to ALL attack
In this section, we compare SAU with other baselines on ALL ALL attacks on CIFAR-10 with PreAct-ResNet18 and poisoning ratio 10%. Specifically, the target labels for the sample with original labels y are set to y/- (y + 1) mod K where mod is short for "modulus". The experiment results are summarized in Table 25. From Table 25, we can find that SAU achieves the best defending performance in best average R-ACC and average DER, which further demonstrates its effectiveness in defending against backdoor attacks with multiple targets.
attacks and the lowest average ASR. At the same time, SAU also achieves the
Note that the ASR for the poisoned model is significantly lower than the ASR in the single-target case. Thus, Table 25 also shows that SAU can still effectively mitigate backdoor even the ASR of the poisoned model is much lower than 100%.
D.9 Comparison to Reconstructive Neuron Pruning (RNP)
In this section, we compare SAU with RNP [29] to further improve our submission. We adopted the official implementation of RNP btttp::/github.co//bboo cow/bboylyg/RNPP into the framework of Backdoor- Bench for fair comparison and the specific experimental settings are:
Dataset and model architecture: compare SAU with RNP CIFAR-10 dataset with ResNet18 and PreAct-ResNette
Clean data size and poisoning ratio: We adopt clean datasets: 1 % and 5%, and poisoning ratios: 0.5%, and 10% to compare their effectiveness for large and small poisoning ratios.
32

---

Table 25: Results for defending against ALL to ALL attacks
Detense
No Defense
ANP [55]
IP [31
NC NCJ47]
ACC
R-ACC
BadNets [15]
91.89
74.42
18.06
N/A
92.33
2.56
90.12
81.03
83.91
1.72
83.71
82.36 16110
01.88
74.41
18.67
50.0
Blended
93.67
86.69
36.39
86.69
Inpur-Aware
91.92
83.69
7.23
N/A
91.04
0.85
$6.34
90.98
86.69
2.01
51.00
81.72
91
81.49
8.75
50.74
LF [58]
93.84
89.98
2.44
N/A
91.10
2.58
90.31
84.54
1.27
80.8
89.44
93.84
89.96
2.44
SIG
93.4 90.61
N/A
92.83
86.56
2. 2.69
51.74
69.7
06
93.40
90.61
1,08
SSRA [30]
93.46
87.84
3.7
N/A
X1.29
7.10
$4.36
90
82.97
2.48
50.26
87.44
93.46
87 87.84
3.7
50.0
WaNet [36]
1.00
86
10.
Average
92.58
84.54
6.03
N/A
92.30
21
68.73
81:22
84.74
2.75
77.02
86.97
92.48
84.23
6.75
50:11
Defense
NAD [28]
EP [64]
I-BAU [59)
SAU (Ours)
ACC
R-ACC
DER
ACCC
ASR
R-ACC
R-ACC
ACC
ASR
R-ACC
BadNets [15]
80.49
2.51
80.54
80.25
58.72
3,00
87.94
84..2
189.39
1.29
89.90
85 05.3211
90.69
1.02
90.61
86.10
Alended
76.85
5.63
55.88
82.12
91.69
82.62
3.06
51.05
72:20
87.32
-1.47
2.17
82.08
91.16
Input- Aware
82.54
1.75
76.86
86.28
90:2
1.13
87.60
90.42
80.40
6.03
51.10
87.57
91.74
1.57
89.12
90.82
LF [S8]
13.68
2.83
74.56
83.43
91.88
84.64
5.97
51.68
80.38
7.88
64.68
88.81
91.07
1.11
86.84
93.34
SIG
93.19
SSBA [30]
74.12
3.58
69.61
82.46
90.33
41.83
46.59
71.44
88
56
6.49
73.49
88 22
91
1.84
85.53
91.94
WaNer [35]
65.26
82.59
85.97
86.66
74.74
11.13
50.3
1.63
90.98
88.48
91.73
1.43
89.34
$6.58
Average
78.98
3.44
71.21
83.75
90.06
53.44
35.06
64.29
89.95
4.82
80.52
88 42
91 40
1.56
86.81
90.77
Hyper-parameters RNP: We notice that the hyper-parametes of RNP are sensitive to model architecture. After carefully inspecting the pruning process of RNP, we set the pruning threshold 0.75 for ResNet18 and 0.05 for PreAct-ResNet18 which can achieve good overall performance across various attacks.
The experimental results are shown in Table 26,27,28. According to the results and methodologies, we briefly summarize few important differences between them:
Sensitivity to the size of the clean dataset. SAU is effective across various sizes of clean data, while RNP achieves good performance for clean data size 0.5% 1%, as shown in Table 26 and discussed in Section 4 of [29]. Therefore, SAU can be applied to various sizes of clean datasets, especially when the defenders do not know the exact size of the training dataset.
Sensitivity to poisoning ratio. SAU is effective across various poisoning ratios, while RNP faces challenges when defending against low poisoning ratio attacks, as shown in Table 28 and discussed in Section 5 of [29].
Sensitivity to model architecture. SAU is effective and stable across various model architectures, while the parameters of RNP sensitive the model structure, observe that the pruning threshold for RNP varies significantly for various models.
Motivation. SAU is theoretically motivated by the relationship between adversarial robust- ness and backdoor risk, while RNP is empirically driven by some experimental observations.
Table 26: Result on ResNet-16 with Poisoning Ratio 10%
Attack
Clean Ratio
No Defense
No Defense
RNP [29]
RNP [29]
SAU (Ours)
SAU (Ours)
ACC
ASR
ACC
ASR
ACC
ASR
BadNets [15]
1%
91.46
95.31
88.9
2.14
83.62
2.04
Input-Aware
1%
90.31
98.86
89.47
6.60
87.16
4.79
SIG [2]
1%
84.64
99.69
80.88
0.03
84.08
0.37
BadNets [15]
5%
91.46
95.31
10.65
0.00
89.25
1.33
Input-Aware
5%
90.31
98.86
10.09
0.00
88.20
0.60
SIG []]
5%
84.64
99.69
10.00
0.00
89.07
3.99
33

---

Table 27: Result on PreAct-ResNet-18 with Poisoning Ratio 10%
Attack
Clean Ratio
No Defense
No Defense
RNP [29]
RNP [29]
SAU (Ours)
SAU (Ours)
ACC
ASR
ACC
ASR
ACC
ASR
BadNets [15]
1%
91.32
95.03
66.86
0.00
85.12
2.52
Input-Aware
1%
90.67
98.26
79.88
0.00
88.56
2.79
SIG [2]
1%
84.48
98.27
74.47
0.04
84.64
0.64
BadNets [1S]
5%
91.32
95.03
10.00
0.00
89.31
1.53
Input-Aware
5%
90.67
98.26
10.00
0.00
91.59
1.27
SIG [2]
5%
84.48
98.27
11.56
0.00
88.56
1.67
Table 28: Result on ResNet-18 18 with Poisoning Ratio 0.5%
No Defense
No Defense
RNP [29]
RNP |A9 (Ours) SAU (Ours)
Attack
ACC
ASR
ACC
ASR
ACC
ASR
BadNets [15]
93.73
67.06
91.46
11.54
82,63
2.46
Input-Aware
90.78
97.68
87.28
67.74
87.77
4.60
SIG [2]
93.68
85.02
79.91
14.76
87.71
0.14
34


Reviews:
[{"review_id": "4bY2txKHus", "reviewer": "Reviewer_qs3i", "summary": "summary: The paper proposes a mitigation approach for backdoor attacks in ML (attacks where a model predicts target classes for poisoned samples when triggered by the adversary). The authors analyze the relationship between backdoor risk and adversarial risk [adversarial examples and poisoned samples] to create an upper bound for the backdoor risk, identify --Shared Adversarial Examples-- between the poisoned model and the fine-tuned model (which can successfully activate the backdoor) and then formulate an optimization or a purification problem for mitigating backdoor attacks (Shared Adversarial Unlearning) to break the connection between poisoned samples and the target label, such that they are either classified correctly by the fine-tuned model or differently by the two models. The authors further thoroughly analyze their approach, comparing it with six other SOTA defense methods on seven types of backdoor attacks with different model structures and datasets.\n\n", "questions": "*  Type I adv. examples where hθbd and hθ can be misled to the same class represents an upper bound for back door examples. How do you account for false positives? Or is it assumed to be always 100%? \n\n* How does ASR vary in comparison to other unlearning techniques for backdoor mitigation such as I-BAU.\n\n* What are the tradeoffs associated with this approach in terms of implementation - specifically around the cost of identifying the SAEs and SAU?\n\n", "limitations": "* Writing flow can be improved. Adding a toy example would definitely improve the overall understanding of the paper. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper targets an important problem. The approach is novel and the thorough analysis of several key metrics indicates the efficacy of the approach. \n", "weaknesses": "\n* Many  important part are in the supplemental materials, including some of the approaches such as the SAU implementation using SGD to unlearn the SAEs, the all to all case, and multi-trigger in threat model (section 3.1) is unclear.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Suggestion for paper layout**\n\n**R1:** Thanks for your constructive suggestion. We will update the layout in the revised manuscript by moving the suggested contents and other important contents from the supplementary material to the main manuscript, to make it more self-contained and legible. Thanks again for your helpful suggestion. \n \n \n**Q2. Multi-trigger in threat model (section 3.1) is unclear.**\n\n**R2:** We appreciate your interest in the threat model of multi-trigger cases. We address the **multi-trigger and multi-target setting** of the threat model in **Supplemental Materials A.6**. There, we formally define the Multi-target/Multi-trigger threat model and extend our theoretical analysis (Proposition 3) to this case. \n\nWe have also evaluated our method for the multi-trigger/multi-target cases. Specifically, we use two different attacks (denoted by Attack-1 and Attack-2) with different target labels. Following Section 4, we test our method (SAU) against a strong baseline ANP on the CIFAR-10 dataset with a poisoning ratio of 10% and backbone PreAct-ResNet18. The experiment results are summarized in the following table, where we use ASR-1 and ASR-2 to represent the Attack Success Rate for Attack-1 and Attack-2, respectively. These results show that SAU outperforms the baseline in most cases, which demonstrates the effectiveness and robustness of our method in this challenging scenario.\n\n**Table 1: Experiment Results for Multi-trigger and Multi-target Case**\n\n|||NO Defense|NO Defense|NO Defense|ANP|ANP|ANP|SAU|SAU|SAU|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nAttack-1|Attack-2|ACC|ASR-1|ASR-2|ACC|ASR-1|ASR-2|ACC|ASR-1|ASR-2\nBlended|BadNet|90.22|99.27|95.03|85.55|37.07|**0.01**|**88.93**|**2.21**|0.54\nInputAware|BadNet|89.71|78.62|94.77|82.21|4.48|**0.01**|**88.88**|**4.14**|0.70\nLF|BadNet|90.16|98.08|95.15|84.21|79.11|2.30|**88.45**|**2.93**|**0.34**\nSIG|BadNet|82.12|98.48|95.03|76.43|26.92|**0.14**|**85.82**|**0.60**|0.67\nSSBA|BadNet|89.69|95.13|95.11|83.25|20.03|**0.14**|**87.60**|**1.38**|0.52\nWaNet|BadNet|89.44|90.21|95.40|82.21|5.53|**0.01**|**88.02**|**3.12**|0.67\nAverage||88.56|93.30|95.08|82.31|28.86|**0.44**|**87.95**|**2.40**|0.57\n\n**Q3. Type I adv. examples where $h_{\\theta}^{bd}$ and $h_{\\theta}$ can be misled to the same class represents an upper bound for back door examples. How do you account for false positives? Or is it assumed to be always 100%?**\n\n**R3:** Thank you for your interest in shared (Type I) adversarial examples (SAEs) and the proposed upper bound for backdoor examples. We would like to clarify that **not all shared adversarial examples are poisoned samples, and there is a possibility of false positives.** Note that SAEs is a subset of standard adversarial examples. Therefore, **the false positive rate of SAEs is significantly lower than that of standard adversarial examples**, which leads to remarkable improvement in reducing ASR and maintaining clean accuracy. However, there may still be some non-poisoned samples among the shared adversarial examples, which motivates further research on more precise identification of backdoor samples among adversarial examples in the future.\n\n**Q4. How does ASR vary in comparison to other unlearning techniques for backdoor mitigation such as i-BAU.**\n\n**R4:** We appreciate your interest in the comparison between SAU and i-BAU. Here we summarize comparison from Table 2 and Table 3 in our paper:\n* On CIFAR-10, SAU outperforms i-BAU in 5 of 7 attacks with an average ASR reduction of 12.81%. \n* On Tiny ImageNet, SAU outperforms i-BAU in all 6 attacks, with an average ASR reduction of 69.56%.\n\nThese results demonstrate that our method is superior to i-BAU in mitigating backdoor effect.\n\n**Q5. What are the tradeoffs associated with this approach in terms of implementation, specifically around the cost of identifying the SAEs and SAU?**\n\n**R5:** Thank you for your concern about the computation cost of SAU. We would like to emphasize that **SAU is an efficient and effective defense method** which can mitigate backdoor attacks with acceptable overhead for the following reasons:\n1. SAU is a post-processing method that fine-tunes the backdoor model on a clean dataset with a small size. For instance, we can use only 500 samples from CIFAR-10 to fine-tune the model by SAU (see **Supplemental Material D.6** for more details), which takes 1.52 seconds/epoch to fine-tune PreAct-ResNet18 with RTX 4090. Therefore, it can be executed efficiently without requiring a large amount of data or computation resources.\n2. SAU only takes a few epochs to take effect, which further reduces the computational cost of executing it in practice. As shown in **Supplemental Material D.4**, SAU can achieve a low Attack Success Rate and a high Accuracy in a few epochs.\n\nTo further demonstrate the efficiency of SAU, we compare its average runtime with other state-of-the-art defense methods against attacks. We use the same experimental setting as in Section 4, where the poisoning ratio is 10% and the backbone model is PreAct-ResNet18. All experiments are conducted on a server with GPU RTX 4090 and CPU AMD EPYC 7543 32-Core Processor. Table 2 shows the average runtime for each defense method to take effect on CIFAR-10 and Tiny ImageNet (ASR<5% and ACC>85% on CIFAR-10, or ASR<5%, ACC>45% on Tiny ImageNet). Note that if a defend method cannot reach the criteria for \"Take effect\", the maximum runtime is reported. As we can see, **SAU is faster than most of the existing methods, except for NAD and EP**. \n\n**Table 2: Computation Time**\n|Dataset|ANP|FP|NC|NAD|EP|i-BAU|SAU|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nCIFAR-10|289.86s|266.57s|825.86s|75.14s|65.71s|47.00s|43.86s\nTiny ImageNet|1086.50s|318.30s|27359.70s|227.01s|141.67s|621.33s|262.60s\n\n**Q6. Suggestion for a Toy Example.**\n\n**R6:** We appreciate your constructive suggestion for a toy example and we will add a toy example in the revised version for demonstration. Thank you again for your helpful suggestion."}, {"review_id": "rRyi34YfBy", "reviewer": "Reviewer_Do9c", "summary": "This paper analysed the relationship between adversarial examples and poisoned examples. \nThen, this paper proposed a fine-tuning strategy to purify the poisoned model. ", "questions": "Can the proposed method mitigate the backdoor issues of the contrastive learning [4], which is a more practical setting. \n\n[4] Poisoning and backdooring contrastive learning, ICLR2022", "limitations": "After the part of conclusion, the authors discussed the limitation of this work. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1 This paper is easy to follow. \n\n2 This paper provides some experiments that support the proposed method. \n\n3 This paper has some experimental analyses that can lead to the proposed method. ", "weaknesses": "1 It seems this paper used many  ill-established notions such as backdoor risk, shared adversarial risk, vanilla adversarial risk, etc..  To the best of my knowledge, I haven't heard to backdoor risk in any machine learning/AI books. \n\n2 Adversarial training has the issue of high computational overhead, which could limit the practicality of the proposed method in real-world applications.\n\n3 The key reference is missing. This paper also need to compared with the paper [1]. \nPrevious work [1] has proposed to tackle with backdoor attacks with multiple adversarial perturbations (L_p adversarial attack and spatial adversarial attack). I encourage the authors to discuss the differences between both works and explore more types of adversarial attack such as spatial adversarial attacks [2] or perceptual adversarial attacks [3].  Besides, some popular backdoor attacks do not satisfy Assumption 1 such as BadNets which has a large L_p norm. \n\n[1] On the Effectiveness of Adversarial Training against Backdoor Attacks. TNNLS 2023.\n\n[2] Spatially Transformed Adversarial Examples. ICLR 2018.\n\n[3] Perceptual Adversarial Robustness: Defense Against Unseen Threat Models. ICLR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. For the concern about terminology**\n\n**R1.** We appreciate your interest in our work and the terminology we use. We would like to clarify the following points:\n\n* As backdoor learning is an emerging area, we follow the typical paradigm of scientific research like [4,5] to first formulate the question by risk and then replace the risk with surrogate loss. In this way, we first defined the backdoor risk to investigate the relationship between adversarial example and poisoned sample. Therefore, **backdoor risk is not covered by current AI/ML books since it's proposed by our paper and one of contributions**.\n\n* The detailed explanation for some terminology is provided below:\n     1. **Backdoor Risk** is defined as the empirical probability that a backdoored model predicts poisoned sample with trigger to target label. Since backdoored model predicts the poisoned samples to the target class, a lower backdoor risk indicates the model is less likely to be affected by triggers.\n\n     2. **Adversarial Risk** is widely used in the adversarial learning community to measure the performance of adversarial attack and defense[4,5]. It is defined as the empirical probability that a model misclassifies an adversarially perturbed sample. Furthermore, the term 'vanilla' means standard version of something. Therefore, vanilla adversarial risk is simply adversarial risk and we will clarify it in the revised version.\n\n     3. **Shared Adversarial Risk** is a novel concept proposed in this work and one of our contributions, which provides a tighter upper bound for backdoor risk and suggests a new perspective for backdoor mitigation. It is defined as the empirical probability that two models predict an adversarial example to the same wrong label.\n\nThe formal mathematical definition of Backdoor Risk, Adversarial Risk, and Shared Adversarial Risk are presented in Section 3.\n\n**Q2. For the concern about computation cost**\n\n**R2:** Thanks. We would like to refer you to **Q5 of Reviewer QS3I** for a comprehensive analysis of the computation cost, which shows that **SAU is an efficient and effective defense method for backdoor mitigation**.\n\n**Q3. Suggestion for comparison to [1].**\n \n**R3:** Thank you for your constructive suggestion. Since the earliest publication date of [1] (early access on 14 June 2023) is after the submission deadline of NeurIPS 2023, we didn't find it and compare with it. We would like to mention that **our submission is not expected to compare to work that appeared only a month or two before the deadline, according to the latest policy of NeurIPS 2023 (see NeurIPS-FAQ).**\n\nHowever, we are still glad to highlight the following differences between our method and Composite Adversarial Training (CAT) proposed in [1]:\n\n1. SAU is theoretically motivated by the relationship between adversarial robustness and backdoor risk, while CAT is empirically driven by some experimental observations. Therefore, our method has a theoretical guarantee for the performance against various backdoor attacks, while CAT does not.\n2. SAU fine-tunes a backdoored model with a small clean dataset, while CAT trains a backdoor-free model from scratch on a poisoned dataset. Therefore, SAU is more efficient than CAT.\n\nWe have also compared the performance of SAU and CAT on CIFAR-10 dataset following the settings in Section 4. The hyper-parameters for CAT are the same as [1]. The results are shown in the following tables:\n\n**Table 1: Experiments Results with poisoning ratio 1%**\n\n|Method|No Defense|No Defense|No Defense|CAT|CAT|CAT|SAU (Ours)|SAU (Ours)|SAU (Ours)|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nAttack|Acc|ASR|R-ACC|Acc|ASR|R-ACC|Acc|ASR|R-ACC\nBadNets|93.14|74.73|24.24|75.52|2.50|74.73|**91.25**|**0.94**|**91.16**\nInput-Aware|91.74|79.18|19.89|75.24|72.48|23.97|**92.20**|**3.63**|**84.06**\nSSBA|93.43|73.44|24.89|74.94|3.07|71.40|**91.34**|**0.79**|**88.46**\nWaNet|90.65|12.63|79.94|75.21|2.68|74.46|**91.84**|**1.23**|**89.89**\nAverage|92.24|60.00|37.24|75.23|20.18|61.14|**91.66**|**1.65**|**88.39**\n\n**Table 2: Experiments Results with poisoning ratio 10%**\n\n|Method|No Defense|No Defense|No Defense|CAT|CAT|CAT|SAU (Ours)|SAU (Ours)|SAU (Ours)|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nAttack|Acc|ASR|R-ACC|Acc|ASR|R-ACC|Acc|ASR|R-ACC\nBadNets|91.32|95.03|4.67|74.42|92.49|6.21|**89.31**|**1.53**|**88.81**\nInput-Aware|90.67|98.26|1.66|74.21|96.81|2.88|**91.59**|**1.27**|**88.54**\nSSBA|92.88|97.8|1.99|74.29|28.29|57.49|**90.84**|**1.79**|**85.83**\nWaNet|91.25|89.73|9.76|74.62|4.87|73.07|**90.41**|**2.51**|**79.69**\nAverage|91.53|95.21|4.52|74.39|55.61|34.91|**92.54**|**1.78**|**85.72**\n\nThe experimental results show that our method is superior to CAT, as evidenced by the following aspects:\n\n1. SAU achieves significantly higher accuracy (ACC) and lower attack success rate (ASR) than CAT on both poisoning ratios. \n2. SAU is more robust to different poisoning ratios than CAT. In contrast, CAT fails on a high poisoning ratio (10%) (also observed in Table XI of [1]). \n\n**Q4. For exploring more types of adversarial perturbations and the concern about Assumption 1.**\n\n**R4.** Thanks. We would like to refer you to the **Common Response** for more comprehensive analysis and experiments for SAU with different adversarial perturbations such as the suggested **Spatial Adversarial Perturbation (Spatial SAU)** and **Ensemble Adversarial Perturbation (Ensemble SAU)**, and the generalization ability of SAU.\n\n**Q5. Suggestion for contrastive learning**\n\n**A5:** We appreciate your insightful suggestion for applying SAU to contrastive learning. However, the current theoretical analysis and method are designed for classification problems and cannot be directly applied to contrastive learning where the label of samples are not accessible.\n\nReferences\n\n[4] Improving adversarial robustness requires revisiting misclassified examples, ICLR 2019\n\n[5] Theoretically principled trade-off between robustness and accuracy, ICML 2019"}, {"review_id": "6NtDWTiIMi", "reviewer": "Reviewer_mKLw", "summary": "This paper proposes a method to defend against backdoor attacks in deep neural networks through adversarial training techniques. By exploring the connection between adversarial examples and poisoned samples, the authors propose an upper bound for backdoor risk and a bi-level formulation for mitigating backdoor attacks in poisoned models. The proposed approach can identify shared adversarial examples and unlearns them to break the connection between the poisoned sample and the target label.", "questions": "- Compare the proposed method with existing works that study provable/certified robustness against backdoor attacks or data poisoning.\n\n- How to achieve good performance when the trigger magnitude is larger than the assumed norm?\n\n- Does the quality of the clean dataset affect the performance of the proposed method? How to obtain the dataset with high quality in practice?\n", "limitations": "The authors have discussed the limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper explores the connection between adversarial examples and backdoor attacks, which is interesting.\n\n- Extensive experiments are conducted to evaluate the performance of the proposed method.\n\n- The paper is well written and easy to read.", "weaknesses": "- Some important related works are missing. There are some existing works that study provable/certified robustness against backdoor attacks or data poisoning [1,2,3]. However, the authors do not mention those works in the paper. It would be interesting if the authors could compare their method with those works.\n\n- This paper assumes that the trigger magnitude is bounded by a norm, which may degrade the practicality of the proposed method. In practice, it is entirely possible for the attacker to adopt triggers with very large magnitudes, and the proposed method may not work in this case.\n\n- The proposed method relies on a clean dataset to achieve the defense goal, and the quality of the dataset may affect the performance of the proposed method. However, it is not clear how to obtain a clean dataset with high quality in practice.\n\n     [1] RAB: Provable Robustness Against Backdoor Attacks. IEEE S&P 2022.\n\n     [2] Certified Robustness of Nearest Neighbors against Data Poisoning and Backdoor Attacks. AAAI 2022.\n\n     [3] BagFlip: A Certified Defense against Data Poisoning. NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.Suggestion of comparison to provable/certified backdoor defense methods [1,2,3].** \n\n**R1:** We acknowledge the importance of certified robustness against backdoor attacks. In backdoor defense, methods can be categorized into two classes: **empirical methods which aim to develop effective and efficient methods for backdoor defense, and certified methods which aim to build a provable framework for backdoor robustness**. As our method is an empirical method, we would like to point out that our method is different from the existing works on certified robustness in several aspects, which makes them incomparable or impractical to apply to our setting. Specifically:\n\n1. **Efficiency and scalability.** Certified methods usually have very high computation costs and are infeasible to apply to large-scale deep neural networks, while our method is efficient and scalable. For example, the recommended papers [1] and [3] rely on voting over **1000 smoothed models** for prediction and are only feasible for some simple neural networks and simple classification problems. To apply these methods to our setting, we would need to train 1000 PreAct-ResNet models on CIFAR-10 from scratch, which would take over **600 hours (25 days) and 620GB of space for a single attack on a server with one RTX 3090**. That's why we cannot compare our method with [1,2].\n\n2. **Applicability to model architecture.** Many certified methods are restricted to some simple or classical models, while our method is general and applicable to any deep neural network architecture. For example, the recommended paper [2] is designed for only k-Nearest Neighbors (kNN) and radius Nearest Neighbors (rNN) models with some feature extractor and cannot be applied to modern deep neural networks.\n\nTherefore, we believe that our method is more practical and effective for defending against backdoor attacks in deep neural networks than the existing works on certified robustness. We will add more discussion on this point in the revised version of the paper. \n\n**Q2. For the concern about trigger magnitude**\n\n**R2:** Thanks. We would like to emphasize that our theoretical analysis and the proposed method are independent of the choice of perturbation set $S$. Therefore, various perturbation types/sets (not only $L_p$ ball) can be adapted to SAU. We refer you to the **Common Responses** for a more comprehensive analysis of different types of adversarial perturbations and the generalization ability for SAU, including defending against BadNets with various strengths. \n\nWe remark that the **Ensemble SAU** in the **Common Responses** which generates Shared Adversarial Examples by a set of perturbation types, achieves the lowest ASR in all experiments, demonstrating its effectiveness for backdoor mitigation against various attacks.\n\nWe hope the above discussion and the analysis in the **Common Response** could address your concern.\n\n**Q3. For the concern of how to collect clean data and the influence of data quality**\n\n**R3:** We would like to emphasize that accessing a small clean dataset is a common and reasonable assumption for most post-processing methods for backdoor defense, such as [5,6,7,8,9]. There are some possible ways to obtain a clean dataset in practice, such as buying from a trustworthy data supplier, using modern generative models, collecting from the internet, or applying some data cleansing techniques [10].\n\nWe acknowledge that the influence of data quality on post-processing methods is interesting. To investigate the influence of data quality for post-processing backdoor defense methods, we conduct experiments on synthetic datasets and compare SAU with two SOTA methods, i.e., **ANP** and **NAD**. Specifically, we test the three methods with a synthetic dataset CIFAR-5m [11] which provides generated CIFAR-10-like images whose distribution is close but not identical to CIFAR-10, and can be regarded as OOD data. We build mixed datasets by randomly picking samples from CIFAR-10 and CIFAR-5m with a **mixed ratio**, and evaluate the three methods on the mixed datasets. A larger mixed ratio indicates more synthetic data in the mixed dataset. The experiment results are summarized in the following tables. \n\n**Table 1: Experiment for Backdoor Defense on Mixed Datasets**\n||ANP|ANP|NAD|NAD|SAU|SAU|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:\n**Mixed Ratio**|ACC|ASR|ACC|ASR|ACC|ASR\n0.0|90.88|4.88|89.87|2.14|89.31|1.53\n0.2|90.68|2.10|87.96|1.88|89.83|1.63\n0.4|90.21|1.11|87.69|1.88|89.36|1.60\n0.6|90.31|1.51|86.69|2.19|88.44|1.66\n0.8|89.37|0.56|85.90|1.86|88.13|1.79\n1.0|88.74|0.40|84.96|2.48|88.32|1.01\n\nWe can find that **all methods can work on the mixed dataset but as the mixed ratio increase, the accuracy (ACC) may decrease due to the distribution difference between the two datasets**. It's notable that the model distillation-based method NAD is influenced by the quality of the dataset most, with a reduction of 4.91% in ACC, while **SAU is quite robust to the data quality**. Another interesting phenomenon is that the ASR of ANP decreases when more synthetic data is used, which shows that the distribution of data may influence the performance of backdoor defense. All those observations motivate us to conduct research on the relationship between data quality and backdoor defense in the future.\n\nReference\n\n[5] Adversarial neuron pruning purifies backdoored deep models. NeurIPS 2021.\n\n[6] One-shot neural backdoor erasing via adversarial weight masking. NeurIPS 2022.\n\n[7] Adversarial unlearning of backdoors via implicit hypergradient. ICLR 2021.\n\n[8] Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. SP 2019\n\n[9] Reconstructive Neuron Pruning for Backdoor Defense, ICML 2023.\n\n[10] Effective Backdoor Defense by Exploiting Sensitivity of Poisoned Samples. \nNeurIPS 2022.\n\n[11] The Deep Bootstrap Framework: Good Online Learners are Good Offline \nGeneralizers. ICLR 2020."}, {"review_id": "a6VMMGtzgz", "reviewer": "Reviewer_1V93", "summary": "This paper proposes a Shared Adversarial Unlearning (SAU) method for mitigating backdoor attacks. The motivation behind this method is the recognition that not all adversarial samples are effective for backdoor defense. Therefore, it is important to identify the adversarial samples that truly contribute to backdoor defense during adversarial training. The authors derive an upper bound on the backdoor risk from benign model and backdoor model predictions, and optimize the generation of backdoor samples based on this bound. Experimental results demonstrate the effectiveness of the method in purifying the backdoor model.", "questions": "Refer to the weaknesses mentioned above.", "limitations": "Refer to the weaknesses mentioned above.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear research motivation, well-structured writing, and provides ample theoretical support. \n- Comprehensive comparison results with mainstream backdoor defense methods on multiple datasets, confirm its effectiveness.\n", "weaknesses": "- Adversarial training often involves complex min-max optimization problems, making it difficult to apply in practical training. However, existing defense work, such as ABL, achieves efficient backdoor removal by detecting suspicious backdoor samples and performing unlearning (gradient ascent) training. ABL's unlearning paradigm is evidently a simpler and more direct backdoor removal strategy. Therefore, it would be interesting to explore the combination of the adversarial sample generation strategy proposed in this paper with ABL's unlearning paradigm, simplifying the optimization process of the proposed adversarial training. It would be helpful if the authors could provide some theoretical or empirical results.\n\n- From Table 3 in the experiments, it is apparent that the proposed SAU method fails to effectively defend against the WaNet attack with full-image triggers on the TinyImageNet dataset (attack success rate after defense is as high as 85.75%). In other words, when the image scale or trigger norm increases, the SAU method may face defense failures. The authors need to provide a reasonable explanation and corresponding countermeasures to demonstrate the generalization performance of the SAU method.\n\n- If the attacker is aware of the defender's adversarial training strategy, can they generate corresponding backdoor trigger samples based on the adversarial training and launch attacks? In such a scenario, is the SAU defense effective?\n\n- To my knowledge, adversarial defense is sensitive to the scale of clean data. In the experiments, the authors state that the defense method requires 5% clean samples. How much will the accuracy and defense performance of SAU decline when the scale of clean samples drops below 1%?\n\n- There is a lack of comparison with more advanced defense benchmarks, such as the recently proposed RNP [1].\n\nOverall, the authors of this paper highlight the differential impact of different types of adversarial samples on backdoor defense during adversarial training. While there are optimization uncertainties and application limitations, such as the difficulty in scaling to large models, associated with mitigating backdoor attacks through adversarial training, the authors' findings provide new insights into the relationship between adversarial samples and backdoor unlearning. Therefore, if the authors can address the aforementioned concerns and related questions, I would be happy to increase the score.\n\n[1] Yige Li, Xixiang Lyu, Xingjun Ma, et al, Reconstructive Neuron Pruning for Backdoor Defense, ICML, 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. For the combination of ABL**\n\n**A1:** Thanks. We would like to point out that ABL is an in-processing method that can access the whole training dataset for poisoned sample detection and unlearns the suspicious backdoor samples, while our method SAU is a post-processing method and can only use a small clean dataset. Therefore, ABL's unlearning paradigm cannot be directly applied to SAU. We also emphasize that simply replacing the unlearning step of SAU by gradient ascent still leads to a bi-level optimization problem and may not simplify the optimization process.\n\n**Q2. For the concern of generalization ability of the proposed methods and magnitude of trigger norm**\n\n**R2:** Thanks. We refer you to the **Common Response** where we discuss the generalization ability of SAU and test SAU with other perturbations such as the ensemble adversarial perturbation (Ensemble SAU). Note that Ensemble SAU serves as a strong variant of SAU and can achieve better defense performance, e.g., **ASR 0.15% for WaNet on Tiny ImageNet**.\n\n**Q3. For the concern of adaptive attacks**\n\n**R3:** When the attacker is aware of the defender's adversarial training strategy, the attacker can train the backdoored model using Adversarial Training (AT) on the poisoned dataset, such that the model can be resistant to adversarial attack, hindering the generation of shared adversarial examples.\n\nTo evaluate the performance of SAU against such defense-aware backdoor attacks, we conduct experiments on CIFAR-10 dataset following the settings in Section 4. For AT, we use PGD-10 attack with $L_{\\infty}$ norm 8/255. **From Table 1, we can find that SAU can achieve an average ACC of 78.3% and ASR of 2.44%, showing its resistance to such adaptive attacks.**\n\n**Table 1: Experiment Results for Adaptive Attack**\n||No Defense|No Defense|SAU|SAU|\n:-:|:-:|:-:|:-:|:-:\n**Attack**|ACC|ASR|ACC|ASR\nBadNets|79.89|93.46|79.83|2.39\nInput-Aware|79.87|96.60|78.64|4.84\nSIG|73.50|99.50|76.41|0.10\nAverage|77.75|96.52|78.30|2.44\n\n**Q4. For the concern about dataset size**\n\n**R4:** Thank. We would like to refer you to **Supplementary Material D.6** where we test SAU with various numbers of clean samples (from 5% to 0.02%) and show that SAU can be effective for various sizes of datasets.\n\n**Q5. For the suggestion of comparison with RNP [1].**\n\n**R5:** Since the earliest publication date of [1] is very close to the submission deadline of NeurIPS 2023, we didn't find it and compare with it. We would like to mention that **our submission is not expected to compare to work that appeared only a month or two before the deadline, according to the latest policy of NeurIPS 2023 (see NeurIPS-FAQ)**. \n\nHowever, we are still very glad to add the comparison with RNP [1] to further improve our submission. We adopted the official implementation of RNP into the framework of BackdoorBench for fair comparison and the specific experimental settings are:\n\n* **Dataset and model architecture**: we compare our SAU with RNP on CIFAR-10 dataset with ResNet18 and PreAct-ResNet18.\n* **Clean data size and poisoning ratio**: We adopt clean datasets: 1% and 5%, and poisoning ratios: 0.5%, and 10% to compare their effectiveness for large and small poisoning ratios. \n* **Hyper-parameters of RNP**: We notice that the hyper-parameters of RNP are sensitive to model architecture. After carefully inspecting the pruning process of RNP, we set the pruning threshold to 0.75 for ResNet18 and 0.05 for PreAct-ResNet18 which can achieve a good overall performance across various attacks.\n\n**Experimental results and analysis**\nThe experimental results are shown in Table 2,3,4. According to the results and methodologies, we briefly summarize a few important differences between them:\n\n* **Sensitivity to the size of the clean dataset.** SAU is effective across various sizes of clean data, while RNP achieves good performance for clean data size 0.5%~1%, as shown in Table 2 and discussed in Section 4 of [1]. Therefore, SAU can be applied to various sizes of clean datasets, especially when the defenders do not know the exact size of the training dataset.\n\n* **Sensitivity to poisoning ratio.** SAU is effective across various poisoning ratios, while RNP faces challenges when defending against low poisoning ratio attacks, as shown in Table 4 and discussed in Section 5 of [1].\n\n* **Sensitivity to model architecture.** SAU is effective and stable across various model architectures, while the parameters of RNP are sensitive to the model structure, as we observe that the pruning threshold for RNP varies significantly for various models.\n\n* **Motivation.** SAU is theoretically motivated by the relationship between adversarial robustness and backdoor risk, while RNP is empirically driven by some experimental observations.\n\n**Table 2: Result on ResNet-18 with Poisoning Ratio 10%**\n|||No Defense|No Defense|RNP|RNP|SAU|SAU|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\n**Attack**|**Clean Ratio**|ACC|ASR|ACC|ASR|ACC|ASR\nBadNets|1%|91.46|95.31|88.94|2.14|83.62|2.04\nInput-Aware|1%|90.31|98.86|89.47|6.60|87.16|4.79\nSIG|1%|84.64|99.69|80.88|0.03|84.08|0.37\nBadNets|5%|91.46|95.31|10.65|0.00|89.25|1.33\nInput-Aware|5%|90.31|98.86|10.09|0.00|88.20|0.60\nSIG|5%|84.64|99.69|10.00|0.00|89.07|3.99\n\n**Table 3: Result on PreAct-ResNet-18 with Poisoning Ratio 10%**\n|||No Defense|No Defense|RNP|RNP|SAU|SAU|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\n**Attack**|**Clean Ratio**|ACC|ASR|ACC|ASR|ACC|ASR\nBadNets|1%|91.32|95.03|66.86|0.00|85.12|2.52\nInput-Aware|1%|90.67|98.26|79.88|0.00|88.56|2.79\nSIG|1%|84.48|98.27|74.47|0.04|84.64|0.64\nBadNets|5%|91.32|95.03|10.00|0.00|89.31|1.53\nInput-Aware|5%|90.67|98.26|10.00|0.00|91.59|1.27\nSIG|5%|84.48|98.27|11.56|0.00|88.56|1.67\n\n**Table 4: Result on ResNet-18 with Poisoning Ratio 0.5%**\n||No Defense|No Defense|RNP|RNP|SAU|SAU|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:\n**Attack**|ACC|ASR|ACC|ASR|ACC|ASR\nBadNets|93.73|67.06|91.46|11.54|82.63|2.46\nInput-Aware|90.78|97.68|87.28|67.74|87.77|4.60\nSIG|93.68|85.02|79.91|14.76|87.71|0.14"}]

Author rebuttal:
# Common Response

**Q1. Concerns about the perturbation set $\mathcal{S}$ in Assumption 1 that restricts the trigger magnitude, including its practicality, and its generalization to backdoor attack with excessive-magnitude trigger.** 

**A1:** Thanks for the insightful comments. The concerns are raised from Assumption 1, which says that $g(x;\Delta) - x \in \mathcal{S}$ for a sample $x$ and its corresponding poisoned sample $g(x,\Delta)$, which ensures that there exists $\epsilon \in S$ such that $x+ \epsilon = g(x;\Delta)$. We would like to clarify the following points:
1. **Type of adversarial perturbation (AP) and perturbation set.** The theoretical analysis and the proposed method SAU is independent of the AP type and AP set **(not only $L_p$ ball)**. For sample space $\mathcal{X}$, define the set $G=\\{g(x,\Delta)-x|x\in \mathcal{X}\\}$ and the general perturbation set $S=\\{T(x,\epsilon)-x|x\in \mathcal{X}\\}$ for adversarial attack $T$ with learnable parameter $\epsilon\in \Omega$ such that an adversarial example can be generated by $\tilde x_\epsilon=T(x,\epsilon)$. Then, **the theoretical analysis and proposed method can be applied if $G\subseteq S$, where the AP can be any type, such as additive AP, Spatially Transformed AP [1], or Perceptual AP [2], etc.** 
2. **Effect of perturbation set $S$.** The key of SAU is to consider Shared Adversarial Example (SAE) as surrogates for poisoned samples. Given models, **the set of SAE is determined by S: larger S leads to larger set of SAE, and vice versa** Therefore, the perturbation set $S$ plays an important role in the effectiveness of SAU. Specifically, adopting a larger $S$ can cover unknown trigger but also increases the chance of picking SAE this is irrelevant to poisoned samples, and may weaken the defense performance. Meanwhile, some poisoned samples may be beyond the scope of SAE if a smaller $S$ is adopted.
3. **Generalization to excessive-magnitude trigger.** In practice, we find that SAU is effective for various attacks, even if Assumption 1 does not hold. For example, SAU with $L_\infty$ norm bound 0.2 can still effectively mitigate SSBA attack whose average $L_\infty$ trigger norm is much larger than 0.2, revealing the generalization of SAU to attacks with excessive-magnitude trigger.

	**Table 1: SAU for SSBA Attack**
	|Dataset|$L_1$|$L_2$|$L_\infty$|ACC|ASR|R-ACC|
	:-:|:-:|:-:|:-:|:-:|:-:|:-:
	CIFAR-10|108.33|3.29|0.33|90.99|0.58|87.04
	Tiny|581.65|8.26|0.43|51.85|0.11|36.36

	To investigate the SAU's generalization ability, we conduct experiments on BadNets with various strengths. Specifically, for BadNets with 3x3 patched trigger, we alter the trigger's pixel value from 0.2 to 1.0. Table 2 shows the results for defending various BadNets attacks by SAU following the setting in Section 4, from which we can find that **SAU can consistently mitigate the backdoor even when the trigger is beyond the perturbation set**. One reasonable explanation is that although the poisoned samples are not included in SAEs, there exists SAE that has similar effect as poisoned samples as shown in Fig. 1 in the below **Supplementary PDF**, resulting in strong generalization ability of SAU.

	**Table 2: Defend against BadNets Attack with Various Pixel Value**
	|||No Defense|No Defense|ANP|ANP|SAU|SAU|
	:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:
	Pixel|L_inf|ACC|ASR|ACC|ASR|ACC|ASR
	0.2|0.42|91.69|95.60|91.30|2.64|90.01|1.38
	0.4|0.31|91.61|97.79|91.44|1.00|88.43|1.59
	0.6|0.35|91.94|96.43|91.72|7.81|90.49|1.30
	0.8|0.48|91.72|93.20|91.06|2.37|89.23|1.36
	1.0|0.67|91.32|95.03|90.88|4.88|89.31|1.53

4. **Effect of perturbation type:** As pointed by Reviewer 1V93, SAU fails to defend WaNet attack on Tiny ImageNet, which generates poisoned samples by spatial transform and has much larger $L_\infty$ trigger norm. Following the constructive suggestion from Reviewer Do9c, we extend SAU to incorporate two other perturbation types and denote the resulting methods by **Spatial SAU** and **Ensemble SAU**, respectively. Specifically, for spatial SAU, we employ spatial adversarial attack to generate spatial SAEs by spatial transformation. For Ensemble SAU, the perturbation set $S$ is composed of a collection of subsets, including $L_1$ ball (bound 500), $L_2$ ball (bound 10), $L_\infty$ ball (bound 0.2), and the spatial AP set. Then, for each batch, Ensemble SAU randomly chooses one subset of perturbation to generate the SAEs.

	In **Table 3**, we compare SAU, **Spatial SAU** and **Ensemble SAU**, following the setting in Section 4. From the table, we find that Spatial SAU works well on CIFAR-10 dataset but fails to defend BadNets and SSBA attacks on Tiny ImageNet. For WaNet, spatial SAU can effectively defend it on both datasets. We remark that the **Ensemble SAU achieves the lowest ASR in all experiments**. However, as Ensemble SAU has larger perturbation set, its accuracy is lower than SAU by 1~2%.

	**Table 3: Comparison between SAU, Spatial SAU and Ensemble SAU**
	|||No Defense|No Defense|SAU|SAU|Spatial SAU|Spatial SAU|Ensemble SAU|Ensemble SAU|
	:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:
	Dataset|Attack|ACC|ASR|ACC|ASR|ACC|ASR|ACC|ASR
	CIFAR-10|BadNets|91.32|95.03|**89.31**|1.53|88.27|0.97|88.16|**0.67**
	CIFAR-10|SSBA|92.88|97.86|**90.84**|1.79|89.65|4.43|89.92|**0.81**
	CIFAR-10|WaNet|91.25|89.73|**91.26**|1.02|90.21|4.34|89.61|**0.66**
	Tiny ImageNet|BadNets|56.23|100|**51.52**|0.53|50.68|99.81|49.32|**0.49**
	Tiny ImageNet|SSBA|55.22|97.71|**51.85**|0.11|49.79|79.72|49.73|**0.05**
	Tiny ImageNet|WaNet|56.78|54.65|**54.65**|85.75|50.12|0.7|51.14|**0.15**

**In summary**, our theoretical analysis and the proposed method SAU are flexible to the perturbation set, and SAU can be generalized to attack with excessive trigger norms. Moreover, the Ensemble SAU serves as a stronger variant of SAU and can be effective for various attacks.

[1] Spatially Transformed Adversarial Examples.

[2] Perceptual Adversarial Robustness: Defense Against Unseen Threat Models.


Meta-review: Two reviewers tend to reject this paper. One reviewer's rejection reason is that "I am not satisfied with the newly defined notations of backdoor risk and shared adversarial risk". AC feels that this is not a strong reason to reject this paper. AC also read the rebuttal to reviewer mKLw since this reviewer did not respond. AC feel that rebuttal has addressed the concern. Given that, AC feels this paper has the value to the community and tends to accept this paper. AC hopes the authors can add the valuable reviewers' comments into the camera-ready version.
Decision: Accept (poster)