SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[56] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In ICLR 2018, 2018. 23, 24
[57] Cihang Xie and Alan L. Yuille. Intriguing properties of adversarial training at scale. In ICLR 2020, 2020. 28
[58] Yi Zeng, Won Park, Z. Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV 2021, pages 16453-16461. IEEE, 2021. 2, 8, 9, 26, 28, 29, 30, 31, 32, 33
[59] Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In ICLR 2022. OpenReview.net, 2022. 2, 3, 8, 9, 17, 21, 22, 25, 27, 28, 29, 30, 31, 33
[60] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML 2019, pages 7472-7482. PMLR, 2019. 4, 10
[61] Yuhao Zhang, Aws Albarghouthi, and Loris D'Antoni. Bagflip: A certified defense against data poisoning. In NeurIPS 2022, volume 35, pages 31474-31483, 2022. 21
[62] Haizhong Zheng, Ziqi Zhang, Juncheng Gu, Honglak Lee, and Atul Prakash. Efficient adversarial training with transferable adversarial examples. In CVPR 2020, pages 1178-1187, 2020. 3
[63] Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. In European Conference on Computer Vision, pages 175-191. Springer, 2022. 3
[64] Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Pre-activation distributions expose backdoor neurons. In NeurIPS 2022, 2022. 3, 8, 9, 21, 27, 29, 30, 31, 33
[65] Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In ICCV 2023, pages 4466-4477, 2023. 3, 8
[66] Mingli Zhu, Shaokui Wei, Hongyuan Zha, and Baoyuan Wu. Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features. arXiv preprint arXiv:2306.16697, 2023. 3
[67] Zihao Zhu, Mingda Zhang, Shaokui Wei, Li Shen, Yanbo Fan Fan, and Baoyuan Wu. Boosting backdoor attack with a learnable poisoning sample selection strategy. arXiv preprint arXiv:2307.07328, 2023. 2
[68] Zihao Zhu, Mingda Zhang, Shaokui Wei, Bingzhe Wu, and Baoyuan Wu. Vdc: Versatile data cleanser for detecting dirty samples via visual-linguistic inconsistency. arXiv preprint arXiv:2309.16211, 2023. 2
A Proofs and extension
In this section, we provide detailed proof and analysis of the propositions, methods, and extensions.
Proof. Under Assumption 1, for any sample  $\pmb{x} \in \mathcal{D}_{-\hat{y}}$ , we have
$$
\mathbb {I} (h _ {\boldsymbol {\theta}} (g (\boldsymbol {x}; \Delta)) = \hat {y}) \leq \mathbb {I} (g (\boldsymbol {x}; \Delta) \in \mathcal {A} _ {g (\boldsymbol {x}; \Delta) - \boldsymbol {x}, \boldsymbol {\theta}}) \leq \max  _ {\boldsymbol {\epsilon} \in \mathcal {S}} \mathbb {I} (\tilde {\boldsymbol {x}} _ {\boldsymbol {\epsilon}} \in \mathcal {A} _ {\boldsymbol {\epsilon}, \boldsymbol {\theta}}),
$$
since  $g(\pmb {x};\Delta) - \pmb {x}\in S$
$$
\mathcal {R} _ {b d} (h _ {\boldsymbol {\theta}}) \leq \mathcal {R} _ {a d v} (h _ {\boldsymbol {\theta}}).
$$
Proof. By the definition of backdoor attack, a poisoned sample  $g(\pmb{x}; \Delta)$  is said to attack  $h_{\theta}$  successfully if and only if  $h_{\theta}(g(\pmb{x}; \Delta)) = \hat{y}$ .
As  $h_{\pmb{\theta}_{bd}}(g(\pmb{x}; \Delta)) = \hat{y}$  for all  $\pmb{x} \in \mathcal{D}_s$  (definition of  $\mathcal{D}_s$ ), we have  $h_{\pmb{\theta}}(g(\pmb{x}; \Delta)) = h_{\pmb{\theta}_{bd}}(g(\pmb{x}; \Delta)) \neq y$  if and only if  $g(\pmb{x}; \Delta)$  can attack  $h_{\pmb{\theta}}$  successfully, for all  $\pmb{x} \in \mathcal{D}_s$ .
Furthermore, under Assumption 1,  $g(\pmb{x}; \Delta)$  is an shared adversarial example of  $h_{\pmb{\theta}}$  and  $h_{\pmb{\theta}_{bd}}$  if  $h_{\pmb{\theta}}(g(\pmb{x}; \Delta)) = h_{\pmb{\theta}_{bd}}(g(\pmb{x}; \Delta)) \neq y$  by the Definition 1.
Proof. By Proposition 1 and the decomposition of  $\mathcal{R}_{adv}$  in Section 3.2, we have
$$
\mathcal {R} _ {b d} \left(h _ {\boldsymbol {\theta}}\right) \leq \mathcal {R} _ {a d v} \left(h _ {\boldsymbol {\theta}}\right),
$$
$$
\mathcal {R} _ {s u b} (h _ {\boldsymbol {\theta}}) \leq \mathcal {R} _ {a d v} (h _ {\boldsymbol {\theta}}).
$$
Therefore, we discuss the relation between  $\mathcal{R}_{bd}(h_{\pmb{\theta}})$  and  $\mathcal{R}_{sub}(h_{\pmb{\theta}})$  now.
$$
\begin{array}{l} \mathcal {R} _ {s u b} \left(h _ {\boldsymbol {\theta}}\right) = \frac {1}{\left| \mathcal {D} _ {- \hat {y}} \right|} \sum_ {i = 1} ^ {N} \max  _ {\boldsymbol {\epsilon} _ {i} \in \mathcal {S}} \left\{\mathbb {I} \left(\boldsymbol {x} _ {i} \in \mathcal {D} _ {s}\right) \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}\right) = h _ {\boldsymbol {\theta} _ {b d}} \left(\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}\right), \tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}} \in \mathcal {A} _ {\boldsymbol {\epsilon} _ {i}, \boldsymbol {\theta}}\right) \right. \tag {15} \\ \left. + \mathbb {I} \left(\boldsymbol {x} _ {i} \in \mathcal {D} _ {- \hat {y}} \backslash \mathcal {D} _ {s}, \tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}} \in \mathcal {A} _ {\boldsymbol {\epsilon} _ {i}, \boldsymbol {\theta}}\right) \right\}. \\ \end{array}
$$
Then, under Assumption 1, we have
- For  $\pmb{x} \in \mathcal{D}_s$ , Proposition 2 yields that
$$
\begin{array}{l} \mathbb {I} \left(h _ {\boldsymbol {\theta}} (g (\boldsymbol {x}; \Delta)) = \hat {y}\right) = \mathbb {I} \left(h _ {\boldsymbol {\theta}} (g (\boldsymbol {x}; \Delta)) = h _ {\boldsymbol {\theta} _ {b d}} (g (\boldsymbol {x}; \Delta))\right) \\ \leq \max  _ {\boldsymbol {\epsilon} \in S} \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(\tilde {\boldsymbol {x}} _ {\boldsymbol {\epsilon}}\right) = h _ {\boldsymbol {\theta} _ {b d}} \left(\tilde {\boldsymbol {x}} _ {\boldsymbol {\epsilon}}\right), \tilde {\boldsymbol {x}} _ {\boldsymbol {\epsilon}} \in \mathcal {A} _ {\boldsymbol {\epsilon}, \boldsymbol {\theta}}\right). \tag {16} \\ \end{array}
$$
- For  $\pmb{x} \in \mathcal{D}_{-\hat{y}} \setminus \mathcal{D}_s$ , similar as Proposition 1, we have
$$
\mathbb {I} (h _ {\boldsymbol {\theta}} (g (\boldsymbol {x}; \Delta)) = \hat {y}) \leq \mathbb {I} (g (\boldsymbol {x}; \Delta) \in \mathcal {A} _ {g (\boldsymbol {x}; \Delta) - \boldsymbol {x}, \boldsymbol {\theta}}) \leq \max  _ {\boldsymbol {\epsilon} \in \mathcal {S}} \mathbb {I} (\tilde {\boldsymbol {x}} _ {\boldsymbol {\epsilon}} \in \mathcal {A} _ {\boldsymbol {\epsilon}, \boldsymbol {\theta}}).
$$
$$
\mathcal {R} _ {b d} (h _ {\boldsymbol {\theta}}) \leq \mathcal {R} _ {s u b} (h _ {\boldsymbol {\theta}}).
$$
So, under Assumption 1, the following inequalities hold:
A.4 Extension to Universal Adversarial Perturbation
Universal Adversarial Perturbation (UAP) has achieved remarkable performance in previous studies [59, 4], especially for defending against backdoor attacks with fixed additive triggers. Now, we extend  $\mathcal{R}_{sub}$  to UAP and show that shared adversarial examples also benefit adversarial training with UAP.
We denote the Universal Adversarial Risk by
$$
\mathcal {R} _ {u a d v} (h _ {\pmb {\theta}}) = \frac {\max _ {\pmb {\epsilon} \in \mathcal {S}} \sum_ {i = 1} ^ {N} \mathbb {I} (\tilde {\pmb {x}} _ {i , \pmb {\epsilon}} \in \mathcal {A} _ {\pmb {\epsilon} , \pmb {\theta}} , \pmb {x} _ {i} \in \mathcal {D} _ {- \hat {y}})}{| \mathcal {D} _ {- \hat {y}} |}.
$$
To bridge UAP and backdoor trigger, we first provide the following assumption:
Assumption 2. Assume that the  $g(\pmb {x},\Delta) = \pmb {x} + \Delta$  for all  $\pmb {x}\in \mathcal{D}_{cl}$
Assumption 2 says that the poisoned samples are planted with a fixed additive trigger.
Then, under Assumption 2 and Assumption 1, we have
$$
\mathcal {R} _ {b d} (h _ {\boldsymbol {\theta}}) \leq \mathcal {R} _ {u a d v} \leq \mathcal {R} _ {a d v}.
$$
by the fact that  $\Delta \in S$
For clarity, we replace  $\tilde{x}_{\epsilon} \in \mathcal{A}_{\epsilon,\theta}$  by  $h_{\theta}(x + \epsilon) \neq y$  and  $\tilde{x}_{\epsilon}$  by  $x + \epsilon$  which makes the rest derivation easier to follow.
Then, similar as  $\mathcal{R}_{sub}$ , we define the universal sub-adversarial risk  $\mathcal{R}_{usub}$  as below:
$$
\begin{array}{l} \mathcal {R} _ {u s u b} (h _ {\boldsymbol {\theta}}) = \frac {1}{| \mathcal {D} _ {- \hat {y}} |} \max  _ {\boldsymbol {\epsilon} \in \mathcal {S}} \sum_ {i = 1} ^ {N} \left\{\mathbb {I} \left(h _ {\boldsymbol {\theta}} (\boldsymbol {x} + \boldsymbol {\epsilon}) = h _ {\boldsymbol {\theta} _ {b d}} (\boldsymbol {x} + \boldsymbol {\epsilon}) \neq y, \boldsymbol {x} \in \mathcal {D} _ {s}\right) + \right. \\ \left. \mathbb {I} \left(h _ {\boldsymbol {\theta}} (\boldsymbol {x} + \boldsymbol {\epsilon}) \neq y, \boldsymbol {x} \in \mathcal {D} _ {- \hat {y}} \backslash \mathcal {D} _ {s}\right) \right\}, \\ \end{array}
$$
which considers shared universal adversarial risk on  $\mathcal{D}_s$  and vanilla universal adversarial risk on  $\mathcal{D}_{-\hat{y}}\setminus \mathcal{D}_s$ .
By the definition of  $\mathcal{R}_{u s u b}$ , we can easily find  $\mathcal{R}_{u s u b} \leq \mathcal{R}_{u a d v}$ .
$$
\begin{array}{l} \mathcal {R} _ {b d} (h _ {\boldsymbol {\theta}}) = \frac {\sum_ {i = 1} ^ {N} \mathbb {I} (h _ {\boldsymbol {\theta}} (\boldsymbol {x} + \Delta) = \hat {y} , \boldsymbol {x} \in \mathcal {D} _ {- \hat {y}})}{| \mathcal {D} _ {- \hat {y}} |} \\ = \frac {\sum_ {i = 1} ^ {N} \left\{\mathbb {I} \left(h _ {\boldsymbol {\theta}} (\boldsymbol {x} + \Delta) = h _ {\boldsymbol {\theta} _ {b d}} (\boldsymbol {x} + \Delta) , \boldsymbol {x} \in \mathcal {D} _ {s}\right) + \mathbb {I} \left(h _ {\boldsymbol {\theta}} (\boldsymbol {x} + \Delta) = \hat {y} , \boldsymbol {x} \in \mathcal {D} _ {- \hat {y}} \setminus \mathcal {D} _ {s}\right) \right\}}{| \mathcal {D} _ {- \hat {y}} |} \\ \leq \frac {\operatorname* {m a x} _ {\boldsymbol {\epsilon} \in \mathcal {S}} \sum_ {i = 1} ^ {N} \left\{\mathbb {I} \left(h _ {\boldsymbol {\theta}} (\boldsymbol {x} + \boldsymbol {\epsilon}) = h _ {\boldsymbol {\theta} _ {b d}} (\boldsymbol {x} + \boldsymbol {\epsilon}) , \boldsymbol {x} \in \mathcal {D} _ {s}\right) + \mathbb {I} \left(h _ {\boldsymbol {\theta}} (\boldsymbol {x} + \boldsymbol {\epsilon}) \neq y , \boldsymbol {x} \in \mathcal {D} _ {- \hat {y}} \setminus \mathcal {D} _ {s}\right) \right\}}{| \mathcal {D} _ {- \hat {y}} |} \\ = \mathcal {R} _ {u s u b} (h _ {\boldsymbol {\theta}}). \\ \end{array}
$$
Therefore, we reach the following proposition
Proposition 4. Under Assumption 2 and Assumption 1, for a classifier  $h_\theta$ , the following inequalities hold
$$
\mathcal {R} _ {b d} (h _ {\boldsymbol {\theta}}) \leq \mathcal {R} _ {u s u b} (h _ {\boldsymbol {\theta}}) \leq \mathcal {R} _ {u a d v} (h _ {\boldsymbol {\theta}}) \leq \mathcal {R} _ {a d v} (h _ {\boldsymbol {\theta}}).
$$
Therefore, the shared adversarial examples also benefit adversarial training with UAP.
A.5 Extension to Targeted Adversarial Training
In the previous sections, we assume that the defender has no access to either the trigger or the target label. Since there are already some works for backdoor trigger detection Guo et al. [16], Dong et al. [10], we discuss the case where the defender can estimate the target label  $\hat{y}$  and show that shared adversarial examples can also be beneficial for adversarial training with target label.
Given a target label  $\hat{y}$ , we define the Targeted Adversarial Risk as
$$
\mathcal {R} _ {t a d v} \left(h _ {\boldsymbol {\theta}}\right) = \frac {\sum_ {i = 1} ^ {N} \max  _ {\boldsymbol {\epsilon} _ {i} \in \mathcal {S}} \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(\tilde {\boldsymbol {x}} _ {i , \boldsymbol {\epsilon} _ {i}}\right) = \hat {y} , \boldsymbol {x} _ {i} \in \mathcal {D} _ {- \hat {y}}\right)}{\left| \mathcal {D} _ {- \hat {y}} \right|}. \tag {17}
$$
By the definition of  $\hat{y}$ , we have  $\mathcal{R}_{bd}(h_{\pmb{\theta}}) \leq \mathcal{R}_{tadv}(h_{\pmb{\theta}})$  under Assumption 1.
$$
\begin{array}{l} \mathcal {R} _ {t a d v} \left(h _ {\boldsymbol {\theta}}\right) = \frac {1}{\left| \mathcal {D} _ {- \hat {y}} \right|} \sum_ {i = 1} ^ {N} \max  _ {\boldsymbol {\epsilon} _ {i} \in \mathcal {S}} \left\{\mathbb {I} \left(\boldsymbol {x} _ {i} \in \mathcal {D} _ {s}\right) \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}\right) = h _ {\boldsymbol {\theta} _ {b d}} \left(\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}\right) = \hat {y}\right) \right. \tag {18} \\ + \mathbb {I} (\boldsymbol {x} _ {i} \in \mathcal {D} _ {s}) \mathbb {I} (h _ {\boldsymbol {\theta}} (\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}) = \hat {y}, h _ {\boldsymbol {\theta} _ {b d}} (\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}) \neq \hat {y}) + \mathbb {I} (\boldsymbol {x} _ {i} \in \mathcal {D} _ {- \hat {y}} \setminus \mathcal {D} _ {s}, h _ {\boldsymbol {\theta}} (\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}) = \hat {y}) \Big \}. \\ \end{array}
$$
Then, similar as  $\mathcal{R}_{sub}$ , we define the targeted sub-adversarial risk  $\mathcal{R}_{tsub}$  by removing the second component in  $\mathcal{R}_{tadv}(h_{\theta})$ :
$$
\begin{array}{l} \mathcal {R} _ {t s u b} (h _ {\boldsymbol {\theta}}) = \frac {1}{| \mathcal {D} _ {- \hat {y}} |} \sum_ {i = 1} ^ {N} \max _ {\boldsymbol {\epsilon} _ {i} \in \mathcal {S}} \left\{\mathbb {I} (\boldsymbol {x} _ {i} \in \mathcal {D} _ {s}) \mathbb {I} (h _ {\boldsymbol {\theta}} (\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}) = h _ {\boldsymbol {\theta} _ {b d}} (\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}) = \hat {y}) \right. \\ \left. + \mathbb {I} \left(\boldsymbol {x} _ {i} \in \mathcal {D} _ {- \hat {y}} \backslash \mathcal {D} _ {s}, h _ {\boldsymbol {\theta}} \left(\tilde {\boldsymbol {x}} _ {i, \epsilon_ {i}}\right) = \hat {y}\right) \right\}. \\ \end{array}
$$
which considers shared targeted adversarial risk on  $\mathcal{D}_s$  and vanilla targeted adversarial risk on  $\mathcal{D}_{-\hat{y}}\setminus \mathcal{D}_s$ .
Since the removed component is irrelevant to backdoor risk, we have  $\mathcal{R}_{bd} \leq \mathcal{R}_{tsub}$  under Assumption 1.
Proof. Under Assumption 2 and Assumption 1, for a classifier  $h_\theta$ , the following inequalities hold
$$
\mathcal {R} _ {b d} \left(h _ {\boldsymbol {\theta}}\right) \leq \mathcal {R} _ {t s u b} \left(h _ {\boldsymbol {\theta}}\right) \leq \mathcal {R} _ {t a d v} \left(h _ {\boldsymbol {\theta}}\right) \leq \mathcal {R} _ {a d v} \left(h _ {\boldsymbol {\theta}}\right).
$$
Therefore, the shared adversarial examples also benefit adversarial training with target label.
Moreover, since  $\mathcal{R}_{tsub}(h_{\pmb{\theta}})$  only considers a shared adversarial example with target label  $\hat{y}$ , the following proposition holds,
$$
\mathcal {R} _ {b d} \left(h _ {\boldsymbol {\theta}}\right) \leq \mathcal {R} _ {t s u b} \left(h _ {\boldsymbol {\theta}}\right) \leq \mathcal {R} _ {s u b} \left(h _ {\boldsymbol {\theta}}\right) \leq \mathcal {R} _ {a d v} \left(h _ {\boldsymbol {\theta}}\right).
$$
The above proposition shows that  $\mathcal{R}_{tsub}(h_{\pmb{\theta}})$  is a tighter bound of  $\mathcal{R}_{bd}(h_{\pmb{\theta}})$  than  $\mathcal{R}_{sub}(h_{\pmb{\theta}})$ .
A.6 Extension to Multi-target/Multi-trigger
In previous sections, we assume that there is only one trigger and one target for all samples. Now, we consider to a more general case where a set of triggers  $V$  and a set of target labels  $\hat{Y}$  are given. This is a more challenging case since there may be multiple triggers and/or multiple targets for one sample.
We define  $T$  to be a mapping from the sample to the corresponding set of feasible trigger-target pairs. Therefore, given a sample  $\pmb{x}$ , the set of all feasible trigger-target pairs is given by  $T(\pmb{x})$ . Then, Assumption 1 is extended to all triggers accordingly, i.e.,  $g(\pmb{x}, \Delta) - \pmb{x} \in S, \forall (\Delta, \hat{y}) \in T(\pmb{x})$ .
In this setting, we define that a classifier  $h_\theta$  is said to be attacked by a poisoned sample generated by  $\pmb{x}$  if there exists  $(\Delta, \hat{y}) \in T(\pmb{x})$  such that  $h_\theta(g(\pmb{x}, \Delta)) = \hat{y} \neq y$ .
Then, the (maximum) backdoor risk on  $\mathcal{D}_{cl}$  is defined to be
$$
\mathcal {R} _ {b d} \left(h _ {\boldsymbol {\theta}}\right) = \frac {\sum_ {i = 1} ^ {N} \max  _ {\left(\Delta_ {i} , \hat {y} _ {i}\right) \in T \left(\boldsymbol {x} _ {i}\right)} \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(g \left(\boldsymbol {x} _ {i} , \Delta_ {i}\right)\right) = \hat {y} _ {i} \neq y\right)}{\left| \mathcal {D} _ {c l} \right|}. \tag {19}
$$
Similarly, the adversarial risk on  $\mathcal{D}_{cl}$  is defined as
$$
\mathcal {R} _ {a d v} = \frac {1}{| \mathcal {D} _ {c l} |} \sum_ {i = 1} ^ {N} \max  _ {\boldsymbol {\epsilon} _ {i} \in \mathcal {S}} \left\{\mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(g \left(\boldsymbol {x} _ {i}, \boldsymbol {\epsilon} _ {i}\right)\right) \neq y\right) \right\}.
$$
We define  $\mathcal{D}_s = \{\pmb{x} \in \mathcal{D}_{cl} : h_{\pmb{\theta}_{bd}}(g(\pmb{x}, \Delta)) = \hat{y} \neq y, \forall (\Delta, \hat{y}) \in T(\pmb{x})\}$  to be the set of samples on which planting any feasible trigger can activate the corresponding backdoor. Then, the corresponding sub-adversarial risk is defined as
$$
\begin{array}{l} \mathcal {R} _ {s u b} = \frac {1}{| \mathcal {D} _ {c l} |} \sum_ {i = 1} ^ {N} \max  _ {\boldsymbol {\epsilon} _ {i} \in \mathcal {S}} \left\{\mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(g \left(\boldsymbol {x} _ {i}, \boldsymbol {\epsilon} _ {i}\right)\right) = h _ {\boldsymbol {\theta} _ {b d}} \left(g \left(\boldsymbol {x} _ {i}, \boldsymbol {\epsilon} _ {i}\right)\right) \neq y, \boldsymbol {x} _ {i} \in \mathcal {D} _ {s}\right) \right. \\ + \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(g \left(\boldsymbol {x} _ {i}, \boldsymbol {\epsilon} _ {i}\right)\right) \neq y, \boldsymbol {x} _ {i} \in \mathcal {D} _ {c l} \backslash \mathcal {D} _ {s}\right) \}. \\ \end{array}
$$
which considers shared adversarial risk on  $\mathcal{D}_s$  and vanilla targeted adversarial risk on  $\mathcal{D}_{cl} \setminus \mathcal{D}_s$ .
By definition, we have  $\mathcal{R}_{sub} \leq \mathcal{R}_{adv}$
$$
\begin{array}{l} \mathcal {R} _ {b d} (h _ {\pmb {\theta}}) = \frac {\sum_ {i = 1} ^ {N} \max _ {(\Delta_ {i} , \hat {y} _ {i}) \in T (\pmb {x} _ {i}))} \mathbb {I} (h _ {\pmb {\theta}} (g (\pmb {x} _ {i} , \Delta_ {i})) = \hat {y} _ {i} \neq y)}{| \mathcal {D} _ {c l} |} \\ = \frac {1}{\left| \mathcal {D} _ {c l} \right|} \sum_ {i = 1} ^ {N} \max  _ {\left(\Delta_ {i}, \hat {y} _ {i}\right) \in T \left(\boldsymbol {x} _ {i}\right))} \left\{\mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(g \left(\boldsymbol {x} _ {i}, \Delta_ {i}\right)\right) = h _ {\boldsymbol {\theta} _ {b d}} \left(g \left(\boldsymbol {x} _ {i}, \Delta_ {i}\right)\right), h _ {\boldsymbol {\theta} _ {b d}} \left(g \left(\boldsymbol {x} _ {i}, \Delta_ {i}\right)\right) = \hat {y} _ {i} \neq y\right) \right. \\ + \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(g \left(\boldsymbol {x} _ {i}, \Delta_ {i}\right)\right) \neq y, h _ {\boldsymbol {\theta} _ {b d}} \left(g \left(\boldsymbol {x} _ {i}, \Delta_ {i}\right)\right) \neq \hat {y} _ {i}\right) \bigg \} \\ \leq \frac {1}{| \mathcal {D} _ {c l} |} \sum_ {i = 1} ^ {N} \max  _ {(\Delta_ {i}, \dot {y} _ {i}) \in T (\boldsymbol {x} _ {i}))} \left\{\mathbb {I} (h _ {\boldsymbol {\theta}} (g (\boldsymbol {x} _ {i}, \Delta_ {i})) = h _ {\boldsymbol {\theta} _ {b d}} (g (\boldsymbol {x} _ {i}, \Delta_ {i})) \neq y, \boldsymbol {x} _ {i} \in \mathcal {D} _ {s}\right) \\ + \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(g \left(\boldsymbol {x} _ {i}, \Delta_ {i}\right)\right) \neq y, \boldsymbol {x} _ {i} \in \mathcal {D} _ {c l} \backslash \mathcal {D} _ {s}\right) \Big \} \\ \leq \frac {1}{| \mathcal {D} _ {c l} |} \sum_ {i = 1} ^ {N} \max  _ {\boldsymbol {\epsilon} _ {i} \in \mathcal {S}} \left\{\mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(g \left(\boldsymbol {x} _ {i}, \boldsymbol {\epsilon} _ {i}\right)\right) = h _ {\boldsymbol {\theta} _ {b d}} \left(g \left(\boldsymbol {x} _ {i}, \boldsymbol {\epsilon} _ {i}\right)\right) \neq y, \boldsymbol {x} _ {i} \in \mathcal {D} _ {s}\right) \right. \\ \left. + \mathbb {I} \left(h _ {\boldsymbol {\theta}} \left(g \left(\boldsymbol {x} _ {i}, \boldsymbol {\epsilon} _ {i}\right)\right) \neq y, \boldsymbol {x} _ {i} \in \mathcal {D} _ {c l} \backslash \mathcal {D} _ {s}\right) \right\} \\ = \mathcal {R} _ {s u b}. \\ \end{array}
$$
Therefore, for a backdoor attack with multiple targets and/or multiple triggers, the proposed subadversarial risk  $\mathcal{R}_{sub}$  is still a tighter upper bound of  $\mathcal{R}_{bd}$  compared to  $\mathcal{R}_{adv}$ .
A.7 Relaxation gap
To build a practical objective for defending against a backdoor attack, two relaxations are applied to  $\mathcal{R}_{sub}$ . Here, we discuss the relaxation gap in the relaxing process:
- Replace  $\mathcal{D}_s$  by  $\mathcal{D}_{-\hat{y}}$ .
Since  $\mathcal{D}_s$  is not accessible, the first relaxation is to replace  $\mathcal{D}_s$  by  $\mathcal{D}_{-\hat{y}}$  and consider shared adversarial examples on all samples in  $\mathcal{D}_{-\hat{y}}$ . Denote  $\mathcal{R}_1(h_\theta)$  to be the relaxed risk after replacing
$\mathcal{D}_s$  by  $\mathcal{D}_{-\hat{y}}$ . Since  $\mathcal{R}_{sub}$  can be decomposed as follows:
$$
\begin{array}{l} \mathcal {R} _ {s u b} (h _ {\boldsymbol {\theta}}) = \frac {1}{| \mathcal {D} _ {- \hat {y}} |} \sum_ {i = 1} ^ {N} \max _ {\boldsymbol {\epsilon} _ {i} \in \mathcal {S}} \Big \{\mathbb {I} (\boldsymbol {x} _ {i} \in \mathcal {D} _ {s}) \mathbb {I} (h _ {\boldsymbol {\theta}} (\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}) = h _ {\boldsymbol {\theta} _ {b d}} (\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}), \tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}} \in \mathcal {A} _ {\boldsymbol {\epsilon} _ {i}, \boldsymbol {\theta}}) \\ \left. + \mathbb {I} \left(\boldsymbol {x} _ {i} \in \mathcal {D} _ {- \hat {y}} \backslash \mathcal {D} _ {s}, \tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}} \in \mathcal {A} _ {\boldsymbol {\epsilon} _ {i}, \boldsymbol {\theta}}\right) \right\} \\ = \frac {1}{| \mathcal {D} _ {- \hat {y}} |} \sum_ {i = 1} ^ {N} \max _ {\boldsymbol {\epsilon} _ {i} \in \mathcal {S}} \Bigl \{\mathbb {I} (\boldsymbol {x} _ {i} \in \mathcal {D} _ {- \hat {y}}) \mathbb {I} (h _ {\boldsymbol {\theta}} (\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}) = h _ {\boldsymbol {\theta} _ {b d}} (\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}), \tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}} \in \mathcal {A} _ {\boldsymbol {\epsilon} _ {i}, \boldsymbol {\theta}}) \\ \left. + \mathbb {I} (\boldsymbol {x} _ {i} \in \mathcal {D} _ {- \hat {y}} \setminus \mathcal {D} _ {s}) \mathbb {I} (h _ {\boldsymbol {\theta}} (\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}) \neq h _ {\boldsymbol {\theta} _ {b d}} (\tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}}), \tilde {\boldsymbol {x}} _ {i, \boldsymbol {\epsilon} _ {i}} \in \mathcal {A} _ {\boldsymbol {\epsilon} _ {i}, \boldsymbol {\theta}}) \right\}. \\ \end{array}
$$
We can find that the relaxation gap is