SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shaokui Wei $^{1}$  Mingda Zhang $^{1}$  Hongyuan Zha $^{1,2}$  Baoyuan Wu $^{1*}$
Backdoor attacks are serious security threats to machine learning models where an adversary can inject poisoned samples into the training set, causing a backdoored model which predicts poisoned samples with particular triggers to particular target classes, while behaving normally on benign samples. In this paper, we explore the task of purifying a backdoored model using a small clean dataset. By establishing the connection between backdoor risk and adversarial risk, we derive a novel upper bound for backdoor risk, which mainly captures the risk on the shared adversarial examples (SAEs) between the backdoored model and the purified model. This upper bound further suggests a novel bi-level optimization problem for mitigating backdoor using adversarial training techniques. To solve it, we propose Shared Adversarial Unlearning (SAU). Specifically, SAU first generates SAEs, and then, unlearns the generated SAEs such that they are either correctly classified by the purified model and/or differently classified by the two models, such that the backdoor effect in the backdoored model will be mitigated in the purified model. Experiments on various benchmark datasets and network architectures show that our proposed method achieves state-of-the-art performance for backdoor defense. The code is available at https://github.com/SCLBD/BackdoorBench (PyTorch) and https://github.com/shawkui/MindTrojan (MindSpore).
Recent decades have witnessed a growing application of deep neural network (DNN) techniques in various domains, such as face recognition, autonomous driving, and medical image processing [1, 17, 32, 44]. However, DNNs are known to be vulnerable to malicious attacks that can compromise their security and reliability. One of the emerging threats is the backdoor attack, where an attacker attempts to inject stealthy backdoor into a DNN model by manipulating a small portion of the training data such that the poisoned model will behave normally for clean inputs while misclassifying any input with the trigger pattern to a target label. Since the poisoned models can have real-world consequences, such as allowing an attacker to gain unauthorized access to a system or to cause physical harm, it is crucial to develop effective methods for backdoor defense in developing and deploying DNNs.
Adversarial training (AT) is one of the most effective methods for improving the robustness of DNNs [33]. AT is typically formulated as a min-max optimization problem, where the inner maximization aims to find adversarial perturbations that fool the model, while the outer minimization is to reduce the adversarial risk. Recently, several works have explored the use of AT for defending against backdoor attacks. However, these works face some limitations and challenges. Weng et al. [52] and
*Corresponds to Baoyuan Wu (wubaoyuan@cuhk.edu.cn).
Gao et al. [13] found that AT can only be effective for mitigating backdoor attacks with certain trigger patterns and may even increase the vulnerability to backdoor attacks when training from scratch on poisoned datasets. Zeng et al. [59] proposed to unlearn the universal adversarial perturbation (UAP) to remove backdoors from poisoned models. Since directly unlearning UAP leads to highly unstable performance for mitigating backdoors, they employ implicit hyper gradient to solve the min-max problem and achieve remarkable performance. However, their method assumes that the same trigger can activate the backdoor regardless of the samples it is planted on, and therefore lacks a guarantee against more advanced attacks that use sample-specific and/or non-additive triggers.
In this paper, we consider the problem of purifying a poisoned model. After investigating the relationship between adversarial examples and poisoned samples, a new upper bound for backdoor risk to the fine-tuned model is proposed. Specifically, by categorizing adversarial examples into three types, we show that the backdoor risk mainly depends on the shared adversarial examples that mislead both the fine-tuned model and the poisoned model to the same class. Shared adversarial examples suggest a novel upper bound for backdoor risk, which combines the shared adversarial risk and vanilla adversarial risk. Besides, the proposed bound can be extended with minor modifications to universal adversarial perturbation and targeted adversarial perturbation. Based on the new bound, we propose a bi-level formulation to fine-tune the poisoned model and mitigate the backdoor. To solve the bi-level problem, we proposed Shared Adversarial Unlearning (SAU). SAU first identifies the adversarial examples shared by the poisoned model and the fine-tuned model. Then, to break the connection between poisoned samples and the target label, the shared adversarial examples are unlearned such that they are either classified correctly by the fine-tuned model or differently by the two models. Moreover, our method can be naturally extended to defend against backdoor attacks with multiple triggers and/or multiple targets. To evaluate our method, we compare it with six state-of-the-art (SOTA) defense methods on seven SOTA backdoor attacks with different model structures and datasets. Experimental results show that our method achieves comparable and even superior performance to all the baselines.
Our contributions are three folds: 1) We analyze the relationship between adversarial examples and poisoned samples, and derive a novel upper bound for the backdoor risk that can be generalized to various adversarial training-based methods for backdoor defense; 2) We formulate a bi-level optimization problem for mitigating backdoor attacks in poisoned models based on the derived bound, and propose an efficient method to solve it; 3) We conduct extensive experiments to evaluate the effectiveness of our method, and compare it with six state-of-the-art defense methods on seven challenging backdoor attacks with different model structures and datasets.
Backdoor attack. Backdoor attack is one of the major challenges to the security of DNNs. The poisoned model behaves normally on clean inputs but produces the target output when the trigger pattern is present. Based on the types of triggers, backdoor attacks can be classified into two types: fixed-pattern backdoor attacks and sample-specific backdoor attacks. BadNets [15] is the first backdoor attack that uses fixed corner white blocks as triggers. To improve the stealth of the triggers, Blended [7] is proposed to blend the trigger with the image in a weighted way. Since fixed-pattern triggers can be recognized easily, sample-specific backdoor attacks have been proposed. SSBA [30], WaNet [38], LF [58], IRBA [12], VSSC [49] and TAT [8] use different techniques to inject unique triggers for different samples from different angles. Sleeper-agent [42] and Lira [9] optimize the target output to obtain more subtle triggers. Recently, Zhu et al. [67] proposed a learnable poisoning sample selection strategy to further boost the effect of backdoor attacks. To keep the label of the backdoor image matching the image content, LC [40] and SIG [2] use counterfactual and other methods to modify the image to deploy clean label attacks.
Backdoor defense. Backdoor defense aims to reduce the impact of backdoor attacks on deep neural networks (DNNs) through training and other means. There are three types of backdoor defense: pre-processing, in-processing, and post-processing. Pre-processing backdoor defense aims to identify the poisoned samples in the training dataset. For instance, AC [5] filters out the poisoned samples by the abnormal activation clustering phenomenon in the target class; Confusion Training [39] identifies the poisoned samples by training a poisoned model that only fits the poisoned samples; VDC [68] incorporates multimodal large language models to detect the poisoned samples. In-processing
backdoor defense reduces the effect of the backdoor during training. For instance, ABL [27] exploits the fact that the learning speed of backdoor samples is faster than that of the clean sample, and splits some poisoned samples to eliminate the backdoor by forgetting these poisoned samples; DBD [19] divides the backdoor training process and directly inhibits the backdoor learning process; D-ST [6] splits the backdoor samples and uses semi-supervised learning by observing that the clean samples are more robust to image transformation than the backdoor samples. Post-processing backdoor defense mitigates the effect of backdoors for a poisoned model by pruning or fine-tuning. For example, FP [31] prunes some potential backdoor neurons and fine-tunes the model to eliminate the backdoor effect; ANP [55] finds the backdoor neurons by adversarial perturbation to model weights; EP [64] and CLP [63] distinguish the characteristics of the backdoor neurons from the clean neurons; NAD [28] uses a mild poisoned model to guide the training of the poisoned model to obtain a cleaner model; I-BAU [59] finds possible backdoor triggers by the universal adversarial attack and unlearn these triggers to purify the model; FT-SAM [65] boosts the performance of fine-tuning for backdoor mitigation by incorporating sharpness-aware minimization; NPD [66] learns a lightweight linear transformation layer by solving a well designed bi-level optimization problem to defend against backdoor attack.
Adversarial training. In adversarial training, models are imposed to learn the adversarial examples in the training stage and therefore, resistant to adversarial attacks in the inference stage. In one of the earliest works [14], the adversarial examples are generated using Fast Gradient Sign Method. In [33], PGD-AT is proposed, which generates adversarial examples by running FGSM multiple steps with projection and has become the most widely used baseline for adversarial training. Some further improvements of PGD-AT include initialization improvement [21, 23], attack strategy improvement [22], and efficiency improvement [53, 62].
In Section 3.1, we first introduce notations, threat model, and defense goal to formulate the problem. By investigating the relationship between adversarial risk and backdoor risk, a new upper bound of backdoor risk is derived in Section 3.2, from which a bi-level formulation is proposed in Section 3.3.
Notations. We consider a  $K$ -class  $(K \geq 2)$  classification problem that aims to predict the label  $y \in \mathcal{Y}$  of a given sample  $\pmb{x} \in \mathcal{X}$ , where  $\mathcal{Y} = [1, \dots, K]$  is the set of labels and  $\mathcal{X}$  is the space of samples. Let  $h_\theta: \mathcal{X} \to \mathcal{Y}$  be a DNN classifier with model parameter  $\theta$ . We use  $\mathcal{D}$  to denote the set of data  $(x, y)$ . For simplicity, we use  $\pmb{x} \in \mathcal{D}$  as a abbreviation for  $(x, y) \in \mathcal{D}$ . Then, for a sample  $\pmb{x} \in \mathcal{D}$ , its predicted label is
$$
h _ {\boldsymbol {\theta}} (\boldsymbol {x}) = \underset {k = 1, \dots , K} {\arg \max } \boldsymbol {p} _ {k} (\boldsymbol {x}; \boldsymbol {\theta}), \tag {1}
$$
where  $\pmb{p}_k(\pmb{x};\pmb{\theta})$  is the (softmax) probability of  $\pmb{x}$  belonging to class  $k$ .
Threat model. Let  $\mathcal{V}$  be the set of triggers and define  $g: \mathcal{X} \times \mathcal{V} \to \mathcal{X}$  as the generating function for poisoned samples. Then, given a trigger  $\Delta \in \mathcal{V}$  and a sample  $\pmb{x} \in \mathcal{X}$ , one can generate a poisoned sample  $g(\pmb{x}; \Delta)$ . For simplicity, we only consider all to one case, i.e., there is only one trigger  $\Delta$  and one target label  $\hat{y}$ . The all to all case and multi-trigger case are left in Appendix A. We assume that the attacker has access to manipulate the dataset and/or control the training process such that the trained model classifies the samples with pre-defined trigger  $\Delta$  to the target labels  $\hat{y}$  while classifying clean samples normally. In addition, we define the poisoning ratio as the proportion of poisoned samples in the training dataset.
Defense goal. We consider a scenario where a defender is given a poisoned model with parameter  $\theta_{bd}$  and a small set of clean data  $\mathcal{D}_{cl} = \{(x_i, y_i)\}_{i=1}^N$ . Since we are mainly interested in samples whose labels are not the target label, we further define the set of non-target samples as  $\mathcal{D}_{-\hat{y}} = \{(x, y) | (x, y) \in \mathcal{D}_{cl}, y \neq \hat{y}\}$ . Note that the size of  $\mathcal{D}_{cl}$  is small and the defender cannot train a new model from scratch using only  $\mathcal{D}_{cl}$ . The defender's goal is to purify the model so that the clean performance is maintained and the backdoor effect is removed or mitigated. We assume that the defender cannot access the trigger  $\Delta$  or the target label  $\hat{y}$ .
Problem formulation. Using the 0-1 loss [50, 60], the classification risk  $\mathcal{R}_{cl}$  and the backdoor risk  $\mathcal{R}_{bd}$  with respect to classifier  $h_\theta$  on  $\mathcal{D}_{cl}$  can be defined as:
$$
\mathcal {R} _ {c l} \left(h _ {\boldsymbol {\theta}}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {i}\right) \neq y _ {i}\right), \quad \mathcal {R} _ {b d} \left(h _ {\boldsymbol {\theta}}\right) = \frac {\sum_ {i = 1} ^ {N} \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(g \left(\boldsymbol {x} _ {i} , \Delta\right)\right) = \hat {y} , \boldsymbol {x} _ {i} \in \mathcal {D} _ {- \hat {y}}\right)}{\left| \mathcal {D} _ {- \hat {y}} \right|} \tag {2}
$$
where  $\mathbb{I}$  is the indicator function and  $|\cdot |$  denotes the cardinality of a set.
In (2), the classification risk concerns whether a clean sample is correctly classified, while the backdoor risk measures the risk of classifying a poisoned sample from  $\mathcal{D}_{-\hat{y}}$  to target class  $\hat{y}$ .
In this paper, we consider the following problem for purifying a poisoned model:
$$
\min  _ {\boldsymbol {\theta}} \mathcal {R} _ {c l} \left(h _ {\boldsymbol {\theta}}\right) + \lambda \mathcal {R} _ {b d} \left(h _ {\boldsymbol {\theta}}\right), \tag {3}
$$
where  $\lambda$  is a hyper-parameter to control the tradeoff between classification risk and backdoor risk.
However, directly solving (3) is impractical due to the lack of the trigger  $\Delta$  and target label  $\hat{y}$ . Therefore, a natural choice is to replace  $\mathcal{R}_{bd}$  with a surrogate risk irrelevant to  $\Delta$  and  $\hat{y}$ .
3.2 Connection between backdoor risk and adversarial risk
To tackle the above problem, we propose a novel upper bound of backdoor risk which can be relaxed to a surrogate of  $\mathcal{R}_{bd}$  that is independent of  $\Delta$  and  $\hat{y}$ . Before that, we first decompose the adversarial risk and review the connection between backdoor risk and adversarial risk.
Adversarial risk. Given a sample  $\pmb{x}$  and an adversarial perturbation  $\epsilon \in S$ , an adversarial example  $\tilde{\pmb{x}}_{\epsilon}$  can be generated by  $\tilde{\pmb{x}}_{\epsilon} = \pmb{x} + \epsilon$ , where  $S$  is the set of perturbations. We define the set of adversarial example-label pair to  $h_{\theta}$  generated by perturbation  $\epsilon$  and dataset  $D_{cl}$  by  $A_{\epsilon, \theta} = \{ (\tilde{\pmb{x}}_{\epsilon}, y) | h_{\theta}(\tilde{\pmb{x}}_{\epsilon}) \neq y, (\pmb{x}, y) \in D_{cl} \}$  and abbreviate  $(\tilde{\pmb{x}}_{\epsilon}, y) \in A_{\epsilon, \theta}$  by  $\tilde{\pmb{x}}_{\epsilon} \in A_{\epsilon, \theta}$ . Then, the vanilla adversarial risk for  $h_{\theta}$  on  $D_{-\hat{y}}$  can be defined as
$$
\mathcal {R} _ {a d v} \left(h _ {\boldsymbol {\theta}}\right) = \frac {\sum_ {i = 1} ^ {N} \max  _ {\boldsymbol {\epsilon} _ {i} \in \mathcal {S}} \mathbb {I} \left(\tilde {\boldsymbol {x}} _ {i , \boldsymbol {\epsilon} _ {i}} \in \mathcal {A} _ {\boldsymbol {\epsilon} _ {i} , \boldsymbol {\theta}} , \boldsymbol {x} _ {i} \in \mathcal {D} _ {- \hat {y}}\right)}{\left| \mathcal {D} _ {- \hat {y}} \right|}. \tag {4}
$$
To bridge the adversarial examples and poison samples, we provide the following assumption:
Assumption 1. Assume that  $g(\pmb{x}; \Delta) - \pmb{x} \in S$  for  $\forall \pmb{x} \in \mathcal{D}_{cl}$ .
Assumption 1 ensures that there exists  $\epsilon \in S$  such that  $x + \epsilon = g(x; \Delta)$ . Note that the analysis and the proposed method are independent of any specific types of perturbation. More discussions about the choice of  $S$  and its influence on the proposed method are discussed in Appendix B.4.
Then, we reach the first upper bound of backdoor risk:
Proposition 1. Under Assumption 1,  $\mathcal{R}_{adv}$  serves as an upper bound of  $\mathcal{R}_{bd}$ , i.e.,  $\mathcal{R}_{bd} \leq \mathcal{R}_{adv}$ .
Remark. Although  $\mathcal{R}_{adv}$  seems to be a promising surrogate for  $\mathcal{R}_{bd}$ , it neglects some essential connections between adversarial examples and poisoned samples, resulting in a significant gap between adversarial risk and backdoor risk, as shown in Figure 1. Such a gap also leads to a fluctuating learning curve for backdoor risk and a significant drop in accuracy in the backdoor mitigation process by adversarial training. Therefore, we seek to construct a tighter upper bound for  $\mathcal{R}_{bd}$ . A key insight is that not all adversarial examples contribute to backdoor mitigation.
To further identify AEs important for mitigating backdoors, we leverage the information from
Figure 1: Example of purifying poisoned model using adversarial training on Tiny ImageNet [26]. The curves for Accuracy, Backdoor Risk, and Adversarial Risk are indicated by Green, Orange, and Purple, respectively.
the poisoned model to categorize the adversarial examples to  $h_{\theta}$  into three types, as shown in Table 1.
Table 1: Different types of adversarial examples to  $h_{\theta}$ .
Type	Description	Definition
I	Mislead hθ and hθbd to the same class	hθbd(хε) = hθ(хε) ≠ y
II	Mislead hθ, but not mislead hθbd	hθbd(хε) ≠ hθ(хε), hθbd(хε) = y
III	Mislead hθ and hθbd to different classes	hθbd(хε) ≠ hθ(хε), hθbd(хε) ≠ y, hθ(хε) ≠ y
Figure 2: A schematic of the relationship between adversarial examples, shared adversarial examples (SAEs, Type I) and poisoned samples. The adversarial examples for  $h_{\theta_{bd}}$  and  $h_\theta$  are shown in the blue and green solid ellipses, respectively. The poisoned samples are in the black dashed circle. Assume that  $h_{\theta_{bd}}$  and  $h_\theta$  have  $100\%$  backdoor attack success rates in the left, such that all poisoned samples are contained in SAEs. Thus, by reducing the shared adversarial examples between  $h_{\theta_{bd}}$  and  $h_\theta$  (from left to right), the backdoor risk can be mitigated in  $h_\theta$ .
Furthermore, we refer adversarial examples of Type I as the shared adversarial examples between  $h_{\theta}$  and  $h_{\theta_{bd}}$ , as defined below:
Definition 1 (Shared Adversarial Example). Given two classifiers  $h_{\theta_1}$  and  $h_{\theta_2}$ , an adversarial example  $\tilde{x}_{\epsilon}$  is shared between  $h_{\theta_1}$  and  $h_{\theta_2}$  if and only if  $h_{\theta_1}(\tilde{x}_{\epsilon}) = h_{\theta_2}(\tilde{x}_{\epsilon}) \neq y$ .
Let  $\mathcal{D}_s$  be the subset of samples in  $\mathcal{D}_{-\hat{y}}$  on which planting a trigger can successfully activate the backdoor in  $h_{\theta_{bd}}$ , as summarized in Table 2. As illustrated in Figure 2, the following proposition reveals a deeper connection between adversarial examples and poisoned samples.
Proposition 2. For  $\forall (\pmb {x},y)\in \mathcal{D}_s$  the pois- soned sample  $g(\pmb {x};\Delta)$  can attack  $h_\theta$  if and only if  $h_{\theta}(g(\pmb {x};\Delta)) = h_{\theta_{bd}}(g(\pmb {x};\Delta))\neq y.$  Furthermore, under Assumption 1, the pois- soned sample  $g(\pmb {x};\Delta)$  is a shared adversar- ial example between  $h_\theta$  and  $h_{\theta_{bd}}$  ,if it can attack  $h_\theta$
Table 2: Table of notations.
Notation	Description/Definition
hθbd	Poisoned model
hθ	Purified model
Δ	Trigger
ŷ	Target label
g(x, Δ)	Poisoned sample generated from x
S	Set of perturbations
ε	Perturbation
ẑx̂	Adversarial example, x + ε
Dcl	{ (xi, yi ) }N i=1
D-ŷ	{ (x, y ) | (x, y) ∈ Dcl, y ≠ŷ}
Ds	{ (x, y ) | hθbd(g(x; Δ)) =ŷ, x ∈ D-ŷ}
Aε,θ	{ (ẑx̂, y ) | hθ(ẑx̂) ≠ y, (x, y) ∈ Dcl}
Then, the adversarial risk  $\mathcal{R}_{adv}$  can be decomposed to three components as below:
$$
\begin{array}{l} \mathcal{R}_{adv}(h_{\boldsymbol{\theta}}) = \frac{1}{|\mathcal{D}_{-\hat{y}}|}\sum_{i = 1}^{N}\max_{\boldsymbol{\epsilon}_{i}\in \mathcal{S}}\Bigl\{\underbrace{\mathbb{I}(\boldsymbol{x}_{i}\in\mathcal{D}_{s})\mathbb{I}(h_{\boldsymbol{\theta}}(\tilde{\boldsymbol{x}}_{i,\boldsymbol{\epsilon}_{i}}) = h_{\boldsymbol{\theta}_{bd}}(\tilde{\boldsymbol{x}}_{i,\boldsymbol{\epsilon}_{i}}),\tilde{\boldsymbol{x}}_{i,\boldsymbol{\epsilon}_{i}}\in\mathcal{A}_{\boldsymbol{\epsilon}_{i},\boldsymbol{\theta}})}_{\text{Type I adversarial examples on }\mathcal{D}_{s}} \\ + \underbrace {\mathbb {I} \left(\boldsymbol {x} _ {i} \in \mathcal {D} _ {s}\right) \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(\tilde {\boldsymbol {x}} _ {i , \boldsymbol {\epsilon} _ {i}}\right) \neq h _ {\boldsymbol {\theta} _ {b d}} \left(\tilde {\boldsymbol {x}} _ {i , \boldsymbol {\epsilon} _ {i}}\right), \tilde {\boldsymbol {x}} _ {i , \boldsymbol {\epsilon} _ {i}} \in \mathcal {A} _ {\boldsymbol {\epsilon} _ {i} , \boldsymbol {\theta}}\right)} _ {\text {T y p e I I o r I I I a d v e r s a r i a l e x a m p l e s o n} \mathcal {D} _ {s}} + \underbrace {\mathbb {I} \left(\boldsymbol {x} _ {i} \in \mathcal {D} _ {- \hat {\boldsymbol {y}}} \backslash \mathcal {D} _ {s} , \tilde {\boldsymbol {x}} _ {i , \boldsymbol {\epsilon} _ {i}} \in \mathcal {A} _ {\boldsymbol {\epsilon} _ {i} , \boldsymbol {\theta}}\right)} _ {\text {v a n i l l a a d v e r s a r i a l e x a m p l e s o n} \mathcal {D} _ {- \hat {\boldsymbol {y}}} \backslash \mathcal {D} _ {s}} \Bigg \}. \tag {5} \\ \end{array}
$$
Proposition 2 implies that the first component in (5) is essential for backdoor mitigation, as it captures the risk of shared adversarial examples between the poisoned model  $h_{\theta_{d}}$  and the fine-tuned model  $h_{\theta}$ , and the poisoned samples effective for  $h_{\theta}$  belong to SAEs. The second component is irrelevant
for backdoor mitigation, since adversarial examples of Type II and III are either not poisoned samples or ineffective to backdoor attack against  $h_{\theta}$ . The third component protects the samples that are originally resistant to triggers from being backdoor attacked in the mitigation process.
Thus, by removing the second component in (5), we propose the following sub-adversarial risk
$$
\begin{array}{l} \mathcal {R} _ {s u b} \left(h _ {\boldsymbol {\theta}}\right) = \frac {1}{\left| \mathcal {D} _ {- \hat {y}} \right|} \sum_ {i = 1} ^ {N} \max  _ {\boldsymbol {\epsilon} _ {i} \in \mathcal {S}} \left\{\mathbb {I} \left(\boldsymbol {x} _ {i} \in \mathcal {D} _ {s}\right) \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}\right) = h _ {\boldsymbol {\theta} _ {b d}} \left(\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}\right), \tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}} \in \mathcal {A} _ {\boldsymbol {\epsilon} _ {i}, \boldsymbol {\theta}}\right) \right. \tag {6} \\ \left. + \mathbb {I} \left(\boldsymbol {x} _ {i} \in \mathcal {D} _ {- \hat {y}} \backslash \mathcal {D} _ {s}, \tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}} \in \mathcal {A} _ {\boldsymbol {\epsilon} _ {i}, \boldsymbol {\theta}}\right) \right\}. \\ \end{array}
$$
Compared to vanilla adversarial risk  $\mathcal{R}_{adv}$ , the proposed sub-adversarial risk  $\mathcal{R}_{sub}$  focuses on the shared adversarial examples for samples in  $\mathcal{D}_s$  while considering vanilla adversarial examples for samples in  $\mathcal{D}_{-\hat{y}}\setminus \mathcal{D}_s$ . When  $\mathcal{D}_s = \mathcal{D}_{-\hat{y}}$ , i.e.,  $h_{\theta_{bd}}$  has backdoor risk  $100\%$ , the sub-adversarial risk measures the shared adversarial risk on  $\mathcal{D}_{-\hat{y}}$ .