SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This paper explains the properties of TBAs under a unified theoretical framework. We suggest that off-manifold adversarial examples play a major role in TBAs. In particular, we show that off-manifold adversarial examples are transferable when the source model is inaccurate. We also prove that the non-existence of off-manifold adversarial examples is one of the reasons why the success rates of TBAs are hard to improve.
This work is supported by the National Natural Science Foundation of China under Grant 61976161, and the Fundamental Research Funds for the Central Universities under Grant 2042022rc0016.
[1] Nicolas Papernot, Patrick D. McDaniel, and Ian J. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. CoRR, abs/1605.07277, 2016.
[2] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. In CVPR, 2019.
A Further Discussions
The following remark provides an example of the semantic information of a dataset.
Remark A.1 (The semantic information of CIFAR-10). Generally speaking, "semantic" refers to the relationship between natural data and their true label, which should be consistent with human recognition. For example, the semantic information contained in the CIFAR-10 dataset is the true labels (e.g., airplane, automobile, and bird) and their corresponding natural images (e.g., images of airliners, SUVs, and chickens). In this example, an image cannot simultaneously include an airplane and an automobile since "the classes are completely mutually exclusive" in the CIFAR-10 dataset (cf. the official website of CIFAR-10.)
It is also easy to check that the semantic information provided by CIFAR-10 is separated.
A.1 Visualization of the Non-Adversarial Region
We provide a visualization of Example 4.11 in Figure A.1.
Figure A.1: A visualization of Example 4.11. The data manifold  $\mathcal{M}$  is represented by the grid surface. Let the surface in light blue (or dark blue) be the contour surface that  $\phi_1 = 0$  (or  $\phi_2 = 0$ ). The distance between  $\mathbf{x}_0$  and the dark blue surface is  $r_3$ , which is greater than  $\delta$  and  $r_2$ . In this visualized example,  $x_0$  is robust against off-manifold adversarial examples.
A.2 The Applicability of the Manifold Attack Model
Recall that Assumption 3 asserts that the classifiers  $f \in \mathcal{F}$  can be decomposed into the product of a semantic classifier  $f_{b}$  and a concentration multiplier  $\phi$ . In this section, we empirically certify the following weaker version of Assumption 3.
Assumption 4 (Manifold attack model, weaker version). Let  $f$  be the source or target model of a TBA. Let  $\mathcal{M}_1$  be a compact smooth manifold that is not necessarily a subset of  $\mathcal{M}$ . Let  $\mathcal{N}_{\Delta_1}\mathcal{M}_1$  be a tubular neighborhood of  $\mathcal{M}_1$  for some  $\Delta_1 > 0$ . We assume that for  $\forall \mathbf{x} \in \mathcal{M}_1$  and for any non-zero  $\mathbf{u} \in N_{\mathbf{x}}\mathcal{M}_1$ , we have  $f(\mathbf{x} + r_1\mathbf{u}) > f(\mathbf{x} + r_2\mathbf{u})$  holds for  $\forall \delta_1 > r_2 > r_1 > 0$ .
By letting  $G = \mathcal{M}_1$  in Definition 4.2, the classifier in Assumption 3 would also satisfy Assumption 4. In this sense, we claim that Assumption 4 is a weaker version of Assumption 3. Next, we validate Assumption 4 on low-dimensional linearly separated data defined as follows. Given  $d > \kappa > 0$ ,
consider the  $\kappa$ -dimensional manifold (in this case, a flat plane)
$$
\mathcal {M} = \left\{\left(x _ {1}, x _ {2}, \dots , \mathbf {x} _ {k}, 0, \dots , 0\right): x _ {i} \in \mathbb {R}, \forall i = 1, 2, \dots , \kappa \right\} \subset \mathbb {R} ^ {d}. \tag {A.15}
$$
Assume that the natural data  $\mathbf{x} \in \mathcal{M}$  is drawn from some continuous distribution  $D(\mathbf{x})$ , say a mixture of Gaussian distribution. For the sake of simplicity, let the natural data be linearly separable in the following sense. Given  $\lambda > 0$  and natural data  $\mathbf{x} = (x_{1}, \dots, x_{d})$ , we assume that  $|x_{1}| > \lambda$ , and the label of  $\mathbf{x}$  is decided by  $y(\mathbf{x}) = 1$  if  $x_{1} > \lambda$ , and  $y(\mathbf{x}) = -1$  if  $x_{1} < -\lambda$ .
In order to test Assumption 4, our goal is to train a classifier that satisfies this assumption. The training setting is stated as follows. Given  $N > 0$ , let the training set  $S_{N} = \{\mathbf{x}_{1},\mathbf{x}_{2},\dots ,\mathbf{x}_{N}\} \sim D^{N}(\mathbf{x})$  be independent and identically distributed (i.i.d.) random variables. For  $\forall 1\leq n\leq N$ , denote the label of  $\mathbf{x}_n$  by  $y_{N}$ . Since  $D$  is continuous, we assume WLOG that
$$
(\exists a \in \mathbb {R} s. t. \mathbf {x} _ {i} = a \mathbf {x} _ {j}) \Longrightarrow i = j \tag {A.16}
$$
holds for  $\forall 1\leq i,j\leq N$  . The hypothesis class is defined as  $\mathcal{F}_{\mathbf{w}}\coloneqq \{f_{\mathbf{w}}:\mathbf{w}\in \mathbb{R}^{d}\}$  , where
$$
f _ {\mathbf {w}} = \left\langle \mathbf {w}, \frac {\mathbf {x}}{\| \mathbf {x} \| _ {2}} \right\rangle \tag {A.17}
$$
for  $\forall \mathbf{x} \in \mathbb{R}^d$ . Consider the logistic loss function  $l(u) \coloneqq \log(1 + \exp(-u))$ ,  $\forall u \in \mathbb{R}$ , and the gradient descent (GD) iteration
$$
\mathbf {w} (t + 1) = \mathbf {w} (t) - \eta \nabla_ {\mathbf {w}} \mathcal {L} _ {N} (\mathbf {w} (t)). \tag {A.18}
$$
We can show (both empirically and theoretically) that:
- The GD iteration converges to an accurate classifier  $\hat{f}$ ,
-  $\exists \mathcal{M}_1$  such that  $\hat{f}$  satisfies Assumption 4,
which implies that we can actually train an accurate classifier (instead of constructing one using oracle information) that satisfies our assumption. The following section briefly introduces the setting and results of the experiment. The theoretical analysis is omitted.
A.2.1 Numerical Experiments
In order to visualize our results, the manifold  $\mathcal{M}$  is set to be a 2-dimensional plane embedded in  $\mathbb{R}^3$ . The distribution  $D$  is a mixture of two Gaussian distributions with means  $(2,2,0)$  and  $(-2, -2,0)$ , and covariance  $\sigma = 0.5$ . The optimizer is stochastic gradient descent (SGD). We train a two-layer network, denoted by  $\hat{f}$ , that reaches  $100\%$  testing accuracy. We check whether  $\hat{f}$  satisfies Assumption 4. The experimental results are demonstrated in Figure A.2.
Hardware Specification and Environment Our experiments are conducted on an Ubuntu 64-bit Linux workstation, having a 10-core Intel Xeon Silver CPU (2.20 GHz) and 4 Nvidia GeForce RTX 2080 Ti GPUs with 11GB graphics memory.
B Complete Proofs
$$
f _ {b} (\mathbf {x}) = f _ {b} \left(\mathbf {x}; A _ {f}, B _ {f}\right) := \frac {d _ {p} \left(\mathbf {x} , B _ {f}\right) - d _ {p} \left(\mathbf {x} , A _ {f}\right)}{d _ {p} \left(\mathbf {x} , B _ {f}\right) + d _ {p} \left(\mathbf {x} , A _ {f}\right)}. \tag {B.19}
$$
Proof of Proposition 4.4. It is easy to check that  $f_{b}(\mathbf{x}) = 1$  when  $\mathbf{x} \in A_{f}$  and  $f_{b}(\mathbf{x}) = -1$  when  $\mathbf{x} \in B_{f}$ . By definition, we know that  $f_{b}$  is a semantic classifier.
Proposition 4.5. Take  $A_{f} = A$  and  $B_{f} = B$  in Equation (4) and denote the corresponding classifier by  $f_{b}^{*}$ . Then, for any given  $\lambda \geq \delta > 0$ , we have  $R_{\mathrm{std}}(f_b^*) = R_{\mathrm{adv}}(f_b^*,\delta) = 0$ .
Figure A.2: The experimental results can be interpreted as follows. Denote the vectors  $\mathbf{x} \in \mathbb{R}^3$  by  $\mathbf{x} = (x, y, z)$ . The bottom plane is the manifold, i.e., the  $(x, y)$ -plane. For each sub-figure, we consider the score of  $\hat{f}$  at  $(x, y, z)$  for a specific  $z$  chosen from  $\{0, 3, 5, 7\}$ . The height of the surface is the score of  $\hat{f}$ . Obviously, the score of  $\hat{f}$  decreases as  $z$  goes larger, which is in accord with the claim in Assumption 4.
Proof of Proposition 4.5. By Equation (4), we have
$$
f _ {b} ^ {*} (\mathbf {x}) = \frac {d _ {p} (\mathbf {x} , B) - d _ {p} (\mathbf {x} , A)}{d _ {p} (\mathbf {x} , B) + d _ {p} (\mathbf {x} , A)}, \forall \mathbf {x} \in \mathbb {R} ^ {d}. \tag {B.20}
$$
Clearly, we have  $f_{b}^{*}(\mathbf{x}) = 1$  when  $\mathbf{x} \in A$  and  $f_{b}^{*}(\mathbf{x}) = -1$  when  $\mathbf{x} \in B$ . Then, the standard risk of  $f_{b}^{*}$  w.r.t.  $D(\mathbf{x})$  is
$$
R _ {\mathrm {s t d}} \left(f _ {b} ^ {*}\right) = \mathbb {P} _ {D} \left[ f _ {b} ^ {*} (\mathbf {x}) y <   0 \mid \mathbf {x} \in A \right] + \mathbb {P} _ {D} \left[ f _ {b} ^ {*} (\mathbf {x}) y <   0 \mid x \in B \right] = 0 \tag {B.21}
$$
Recall that  $A$  and  $B$  are  $2\lambda$ -separated (cf. Assumption 2). For  $\forall \mathbf{x} \in A$  and  $\mathbf{x}' \in B(\mathbf{x}, \delta)$ , we have  $d_p(\mathbf{x}, B) > \delta$ , which implies that  $d_p(\mathbf{x}', B) - d_p(\mathbf{x}', A) > 0$ , and thus  $f_b^*(\mathbf{x}') f_b^*(\mathbf{x}) = f_b^*(\mathbf{x}') > 0$ . For  $\forall \mathbf{x} \in B$ , a similar deduction shows that  $f_b^*(\mathbf{x}') f_b^*(\mathbf{x}) > 0$  holds for  $\forall \mathbf{x}' \in B(\mathbf{x}, \delta)$ . Together, we have
$$
\begin{array}{l} R _ {\mathrm {a d v}} \left(f _ {b} ^ {*}, \delta\right) := \mathbb {P} \left[ \exists \mathbf {x} ^ {\prime} \in B (\mathbf {x}; \delta) s. t. f _ {b} ^ {*} \left(\mathbf {x} ^ {\prime}\right) f _ {b} ^ {*} (\mathbf {x}) <   0 \mid \mathbf {x} \in A \right] \tag {B.22} \\ + \mathbb {P} \left[ \exists \mathbf {x} ^ {\prime} \in B (\mathbf {x}; \delta) s. t. f _ {b} ^ {*} \left(\mathbf {x} ^ {\prime}\right) f _ {b} ^ {*} (\mathbf {x}) <   0 \mid \mathbf {x} \in B \right] = 0, \\ \end{array}
$$
Remark B.1. The construction of Equation (B.20) can be found in previous works [14, 45]. In particular, Li et al. [14] uses the ReLU-approximation of  $f_{b}^{*}$  to study the robust generalization of deep neural networks.
$$
\phi (\mathbf {x}) = \phi (\mathbf {x}; r, G) := \frac {r - d _ {p} (\mathbf {x} , G)}{r + d _ {p} (\mathbf {x} , G)}, \forall \mathbf {x} \in \mathbb {R} ^ {d}. \tag {B.23}
$$
Proof of Proposition 4.6. For  $\forall \mathbf{x} \in G$ , we have  $d_p(\mathbf{x}, G) = 0$ . That is,  $\phi(\mathbf{x}) = 1$  for  $\forall \mathbf{x} \in G$ . For  $\forall \mathbf{x}_1, \mathbf{x}_2$  s.t.  $d_p(\mathbf{x}_1, G) > d_p(\mathbf{x}_2, G)$ , it is easy to check that  $\phi(\mathbf{x}_1) < \phi(\mathbf{x}_2)$ .
Proof of Proposition 4.7. We first prove the first result. By definition, there are  $r > 0$  and  $G \subset \mathbb{R}^d$  such that
$$
f (\mathbf {x}) = \frac {d _ {p} (\mathbf {x} , B _ {f}) - d _ {p} (\mathbf {x} , A _ {f})}{d _ {p} (\mathbf {x} , B _ {f}) + d _ {p} (\mathbf {x} , A _ {f})} \cdot \frac {r - d _ {p} (\mathbf {x} , G)}{r + d _ {p} (\mathbf {x} , G)}. \tag {B.24}
$$
Given  $R_{\mathrm{adv}}(f; \delta) \neq 0$ , there is natural data  $\mathbf{x} \in \operatorname{supp}(D)$  and  $\mathbf{x}_0 \in B(\mathbf{x}, \delta)$  such that  $f(\mathbf{x}) f(\mathbf{x}_0) < 0$ . If  $x_0 \in \mathcal{M}^c$ , there is nothing to prove.
Otherwise, we have  $\mathbf{x}_0\in \mathcal{M}$ . Denote
$$
r _ {0} := \frac {1}{2} \min  \left\{\left| d _ {p} \left(\mathbf {x} _ {0}, A _ {f}\right) - d _ {p} \left(\mathbf {x} _ {0}, B _ {f}\right) \right|, \left| r - d _ {p} \left(\mathbf {x} _ {0}, G\right) \right|, \left| \delta - d _ {p} \left(\mathbf {x}, \mathbf {x} _ {0}\right) \right| \right\}. \tag {B.25}
$$
Consider the non-empty set
$$
B (\mathbf {x}, \delta) \cap B (\mathbf {x} _ {0}, r _ {0}) \cap \mathcal {M} ^ {c}.
$$
For  $\forall \mathbf{x}_0^{\prime}\in B(\mathbf{x}_0,r_0)$  , there is
$$
d _ {p} \left(\mathbf {x}, \mathbf {x} _ {0} ^ {\prime}\right) \leq d _ {p} \left(\mathbf {x}, \mathbf {x} _ {0}\right) + d _ {p} \left(\mathbf {x} _ {0}, \mathbf {x} _ {0} ^ {\prime}\right) <   \delta . \tag {B.26}
$$
It is also easy to check that
$$
\left(d _ {p} \left(\mathbf {x} _ {0}, B _ {f}\right) - d _ {p} \left(\mathbf {x} _ {0}, A _ {f}\right)\right) \cdot \left(d _ {p} \left(\mathbf {x} _ {0} ^ {\prime}, B _ {f}\right) - d _ {p} \left(\mathbf {x} _ {0} ^ {\prime}, A _ {f}\right)\right) > 0, \tag {B.27}
$$
$$
\left(r - d _ {p} (\mathbf {x} _ {0}, G)\right) \cdot \left(r - d _ {p} (\mathbf {x} _ {0} ^ {\prime}, G)\right) > 0,
$$
which implies that  $f(\mathbf{x})f(\mathbf{x}_0^{\prime}) < 0$  and  $\mathbf{x}_0^\prime$  is an off-manifold adversarial example.
As for the second result, since  $A \subset A_f$ ,  $B \subset B_f$ , and  $A_f \cup B_f \subset G$ , we have
$$
\frac {r - d _ {p} (\mathbf {u} , G)}{r + d _ {p} (\mathbf {u} , G)} = 1, \text {w h e r e} \mathbf {u} \in \{\mathbf {x}, \mathbf {x} ^ {\prime} \} \tag {B.28}
$$
for  $\forall \mathbf{x} \in A \cup B$  and  $\mathbf{x}' \in B(\mathbf{x}, \delta) \cap \mathcal{M}$ . We can easily obtain that  $f_{b}(\mathbf{x}) = f_{b}(\mathbf{x}')$ , which implies that  $f$  has no on-manifold adversarial examples.
Theorem 4.8. Consider TBAs with perturbation radius  $\delta \in (0,\lambda ]$ , target model  $f_{t} = f_{b}^{*}\cdot \phi_{\mathrm{off}}$  and source model  $f_{s} = f_{b}\cdot \phi_{\mathrm{off}},f_{b}\in \mathcal{F}_{b}$ . Denote the semantic information of  $f_{b}$  by  $A_{f}$  and  $B_{f}$ . Then, for  $\forall \mathbf{x}\in A\cup B$ , all adversarial examples (if exist) of  $f_{s}$  at  $\mathbf{x}$  are transferable if  $\mathbf{x}\in A_f\cup B_f$ .
Proof of Theorem 4.8. If  $f_{s}$  is robust against adversarial examples, then there is nothing to prove. Otherwise, consider  $\mathbf{x} \in A_f \cap A$  WLOG. By Equation (6), denote
$$
f _ {s} (\mathbf {x}) = f _ {b} (\mathbf {x}) \cdot \phi_ {\text {o f f}} (\mathbf {x}) = \frac {d _ {p} (\mathbf {x} , B _ {f}) - d _ {p} (\mathbf {x} , A _ {f})}{d _ {p} (\mathbf {x} , B _ {f}) + d _ {p} (\mathbf {x} , A _ {f})} \cdot \frac {\alpha \delta - d _ {p} (\mathbf {x} , \mathcal {M})}{\alpha \delta + d _ {p} (\mathbf {x} , \mathcal {M})}. \tag {B.29}
$$
We can find an adversarial example of  $f_{s}$  at  $\mathbf{x}$ . Denote this adversarial example by  $\mathbf{x}_a$ , and we have  $f_{s}(\mathbf{x}_{a}) < 0$ . It is not hard to verify that  $d_{p}(\mathbf{x}_{a},B_{f}) - d_{p}(\mathbf{x}_{a},A_{f}) > 0$  since  $\delta < \lambda$ , which implies that  $f_{b}(\mathbf{x}_{a}) > 0$ . It is easy to obtain that  $f_{s}(\mathbf{x}_{a}) < 0$  and  $\phi_{\mathrm{off}}(\mathbf{x}_a) < 0$ . We thus have  $\alpha \delta < d_p(\mathbf{x}_a,\mathcal{M})$ , which implies that  $\mathbf{x}_a$  is off the manifold and the distance between  $\mathbf{x}_a$  and  $\mathcal{M}$  is greater than  $\alpha \delta$ . In particular, we have
$$
\phi_ {\text {o f f}} (\mathbf {x}) \phi_ {\text {o f f}} \left(\mathbf {x} _ {a}\right) <   0, \tag {B.30}
$$
which is independent of the choice of  $f_{b}$ . Now consider  $f_{t}(\mathbf{x})$  and  $f_{t}(\mathbf{x}_{a})$ , where
$$
f _ {t} (\mathbf {x}) = f _ {b} ^ {*} (\mathbf {x}) \cdot \phi_ {\text {o f f}} (\mathbf {x}) = \frac {d _ {p} (\mathbf {x} , B) - d _ {p} (\mathbf {x} , A)}{d _ {p} (\mathbf {x} , B) + d _ {p} (\mathbf {x} , A)} \cdot \frac {\alpha \delta - d _ {p} (\mathbf {x} , \mathcal {M})}{\alpha \delta + d _ {p} (\mathbf {x} , \mathcal {M})}. \tag {B.31}
$$
No matter  $\mathbf{x} \in A$  or  $\mathbf{x} \in B$ , we have for  $\forall \mathbf{x}' \in B(\mathbf{x}, \delta)$ , there is  $f_b^*(\mathbf{x}) = f_b^*(\mathbf{x}')$  (by the  $2\lambda$ -separated property of  $A$  and  $B$ ). By Equation (B.30), we have
$$
f _ {t} (\mathbf {x}) f _ {t} \left(\mathbf {x} _ {a}\right) = f _ {b} ^ {*} (\mathbf {x}) f _ {b} ^ {*} \left(\mathbf {x} _ {a}\right) \cdot \phi_ {\text {o f f}} (\mathbf {x}) \phi_ {\text {o f f}} \left(\mathbf {x} _ {a}\right) <   0, \tag {B.32}
$$
i.e.,  $\mathbf{x}_a$  transfers to  $f_t$ , which completes the proof.
Proposition 4.9. Consider TBA with perturbation radius  $\delta \in (0,\lambda ]$ , target model  $f_{t} = f_{b}^{*}\cdot \phi_{\mathrm{on}}(\cdot ,f_{b}^{*})$  and source model  $f_{s} = f_{b}\cdot \phi_{\mathrm{on}}(\cdot ,f_{b}),f_{b}\in \mathcal{F}_{b}$ . Denote
$$
S _ {\mathrm {c r t}} := \left(A \cap A _ {f}\right) \cup \left(B \cap B _ {f}\right), S _ {\mathrm {w r g}} := \left(A \cap B _ {f}\right) \cup \left(B \cap A _ {f}\right). \tag {B.33}
$$
1. if  $B(\mathbf{x},\delta) \cap \mathcal{M} \subset S_{\mathrm{crit}} \cup S_{\mathrm{wrg}}$ , then  $f_{t}$  and  $f_{s}$  are both robust against adversarial examples;
2. if  $B(\mathbf{x},\delta)\cap \mathcal{M}\subset A\cup B$  and  $\mathbf{x}\in A_f\cup B_f$  , then the adversarial examples of  $f_{s}$  at  $\mathbf{x}$  (if exists) cannot transfer to  $f_{t}$
Proof of Proposition 4.9. For the first results, it is easy to check that  $\phi_{\mathrm{on}}(\mathbf{x}) = 1$  and  $\phi_{\mathrm{on}}(\mathbf{x}_a) = 1$  for  $\forall \mathbf{x}_a \in B(\mathbf{x}, \delta)$ , which implies that  $f$  is robust against adversarial examples. It remains to prove the second result. By Equation (7), denote
$$
f _ {s} (\mathbf {x}) = f _ {b} (\mathbf {x}) \cdot \phi_ {\text {o n}} (\mathbf {x}; f _ {b}) = \frac {d _ {2} (\mathbf {x} , B _ {f}) - d _ {2} (\mathbf {x} , A _ {f})}{d _ {2} (\mathbf {x} , B _ {f}) + d _ {2} (\mathbf {x} , A _ {f})} \cdot \frac {\alpha \delta - d _ {2} (\mathbf {x} , \mathcal {N} _ {\delta} (A _ {f} \cup B _ {f}))}{\alpha \delta + d _ {2} (\mathbf {x} , \mathcal {N} _ {\delta} (A _ {f} \cup B _ {f}))}. \tag {B.34}
$$
For  $\forall \mathbf{x} \in A \cup B$  such that  $B(\mathbf{x},\delta) \cap \mathcal{M} \subset A \cup B$  and  $\mathbf{x} \in A_f \cup B_f$ , denote the unspecific adversarial example (if exist) of  $f_s$  at  $\mathbf{x}$  by  $\mathbf{x}_a$ . Assume that  $\mathbf{x} \in A \cap B_f$  WLOG. By definition, we have
$$
f _ {b} ^ {*} (\mathbf {x}) = f _ {t} (\mathbf {x}) > 0, f _ {b} (\mathbf {x}) = f _ {s} (\mathbf {x}) <   0. \tag {B.35}
$$
By the  $2\lambda$ -separated assumption of  $A_{f}$  and  $B_{f}$ , we have
$$
f _ {b} \left(\mathbf {x} _ {a}\right) <   0. \tag {B.36}
$$
From  $f_{s}(\mathbf{x})f_{s}(\mathbf{x}_{a}) < 0$ , we can obtain that  $\phi_{\mathrm{on}}(\mathbf{x};f_b)\phi_{\mathrm{on}}(\mathbf{x}_a;f_b) < 0$ . Since  $\mathbf{x}\in B_f$ , we have
$$
d _ {2} (\mathbf {x}, \mathcal {N} _ {\delta} (A _ {f} \cup B _ {f})) = 0, \tag {B.37}
$$
i.e.,  $\phi_{\mathrm{on}}(\mathbf{x};f_b) = 1$ . Combine this with  $\phi_{\mathrm{on}}(\mathbf{x};f_b)\phi_{\mathrm{on}}(\mathbf{x}_a;f_b) < 0$ , we have  $\phi_{\mathrm{on}}(\mathbf{x}_a;f_b) < 0$ , i.e.,
$$
\alpha \delta <   d _ {2} \left(\mathbf {x} _ {a}, \mathcal {N} _ {\delta} \left(A _ {f} \cup B _ {f}\right)\right) <   d _ {2} \left(\mathbf {x} _ {a}, \mathbf {x}\right) \leq \delta . \tag {B.38}
$$
It remains to show that  $f_{t}(\mathbf{x}_{a}) > 0$ . By Equation (7), denote
$$
f _ {t} (\mathbf {x}) = f _ {b} ^ {*} (\mathbf {x}) \cdot \phi_ {\mathrm {o n}} (\mathbf {x}; f _ {b} ^ {*}) = \frac {d _ {2} (\mathbf {x} , B) - d _ {2} (\mathbf {x} , A)}{d _ {2} (\mathbf {x} , B) + d _ {2} (\mathbf {x} , A)} \cdot \frac {\alpha \delta - d _ {2} (\mathbf {x} , \mathcal {N} _ {\delta} (A \cup B))}{\alpha \delta + d _ {2} (\mathbf {x} , \mathcal {N} _ {\delta} (A \cup B))}. \tag {B.39}
$$
By  $\mathbf{x} \in A$  and the  $2\lambda$ -separated assumption of  $A$  and  $B$ , we have  $f_{b}^{*}(\mathbf{x}_{a}) > 0$ . By  $B(\mathbf{x}, \delta) \cap \mathcal{M} \subset A$ , we have  $\mathbf{x}_{a} \in \mathcal{N}_{\delta}(A \cup B)$ , i.e.,  $\phi_{\mathrm{on}}(\mathbf{x}_{a}; f_{b}^{*}) > 0$ . Together, we have  $f_{t}(\mathbf{x}_{a}) = f_{b}^{*}(\mathbf{x}_{a}) \cdot \phi_{\mathrm{on}}(\mathbf{x}_{a}; f_{b}^{*}) > 0$ . That is,  $\mathbf{x}_{a}$  is not an adversarial example of  $f_{t}$ , which completes the proof.
Theorem 4.13. Given perturbation radius  $\delta \in (0, \lambda]$  and target model  $f_{t} = f_{b}^{*} \cdot \phi_{\mathrm{off}}(\cdot; r, \mathcal{M})$ , Let  $\Delta$  be the constant specified in Lemma 4.12. Then, for  $\forall \mathbf{x} \in A \cup B$ , the off-manifold adversarial example of  $f_{t}$  at  $\mathbf{x}$  exists if  $r < \Delta$ .
Proof of Theorem 4.13. For  $\forall \mathbf{x} \in A \cup B$ , let  $\mathbf{u} \in N_{\mathbf{x}}(\mathcal{M})$  be the normal direction at  $\mathbf{x}$  with  $\|\mathbf{u}\|_2 = 1$ . Since  $r < \Delta$ , we can find  $r_0 > r$  such that  $r_0 < \Delta$  and  $r_0 < \delta$ . Denote
$$
\mathbf {x} _ {a} := \mathbf {x} + r _ {0} \mathbf {u}. \tag {B.40}
$$
Clearly, we have  $\mathbf{x}_a\in B(\mathbf{x},\delta)$ . Since  $\mathcal{N}_{\Delta}(\mathcal{M})$  is a tubular neighborhood of  $\mathcal{M}$ , we have
$$
d _ {2} \left(\mathbf {x} _ {a}, \mathcal {M}\right) = r _ {0} > r, \tag {B.41}
$$
which implies that  $\mathbf{x}_a$  is an off-manifold adversarial example of  $f_t$  at  $\mathbf{x}$ . Notice that Equation (B.41) not necessarily holds when  $r_0 > \Delta$ .
Corollary 5.3. Let  $f_{b}$  be a semantic classifier with semantic information  $A_{f}$  and  $B_{f}$  that satisfy a  $2\lambda$ -separated property. Given  $\epsilon > 0$ , there is a ReLU network  $\tilde{f}$  with  $O((1 / \lambda \epsilon)^d) \cdot O(d^2 + d\log (1 / \epsilon))$  parameters such that  $\| f - \tilde{f}\|_{\infty} \leq \epsilon$ .
Proof of Corollary 5.3. According to Lemma 5.2, our goal is to upper bound the Lipschitz constant  $l$  of  $f_{b}$ . By definition, it suffices to upper bound the supremum of
$$
\begin{array}{l} s := \frac {\left| f _ {b} \left(\mathbf {x} _ {1} ; A _ {f} , B _ {f}\right) - f _ {b} \left(\mathbf {x} _ {2} ; A _ {f} , B _ {f}\right) \right|}{d _ {p} \left(\mathbf {x} _ {1} , \mathbf {x} _ {2}\right)} \\ = \frac {1}{d _ {p} \left(\mathbf {x} _ {1} , \mathbf {x} _ {2}\right)} \cdot \left| \frac {d _ {p} \left(\mathbf {x} _ {1} , A _ {f}\right)}{d _ {p} \left(\mathbf {x} _ {1} , A _ {f}\right) + d _ {p} \left(\mathbf {x} _ {1} , B _ {f}\right)} - \frac {d _ {p} \left(\mathbf {x} _ {2} , A _ {f}\right)}{d _ {p} \left(\mathbf {x} _ {2} , A _ {f}\right) + d _ {p} \left(\mathbf {x} _ {2} , B _ {f}\right)} \right|. \tag {B.42} \\ \end{array}
$$
We only need to consider three cases:
1. both of  $\mathbf{x}_1, \mathbf{x}_2 \in A_f \cup B_f$ , or
2. both of  $\mathbf{x}_1, \mathbf{x}_2 \in (A_f \cup B_f)^c$ , and
3. either  $\mathbf{x}_1$  or  $\mathbf{x}_2$  is in  $A_f \cup B_f$ .
When  $\mathbf{x}_1, \mathbf{x}_2 \in A_f \cup B_f$ , a trivial verification shows that  $s \leq \frac{1}{\lambda}$ . We now turn to the second case. By symmetry, let
$$
\frac {d _ {p} \left(\mathbf {x} _ {1} , A _ {f}\right)}{d _ {p} \left(\mathbf {x} _ {1} , A _ {f}\right) + d _ {p} \left(\mathbf {x} _ {1} , B _ {f}\right)} - \frac {d _ {p} \left(\mathbf {x} _ {2} , A _ {f}\right)}{d _ {p} \left(\mathbf {x} _ {2} , A _ {f}\right) + d _ {p} \left(\mathbf {x} _ {2} , B _ {f}\right)} > 0. \tag {B.43}
$$
By simplifying Equation (B.42), we can obtain that
$$
\begin{array}{l} \frac {\left| f _ {b} \left(\mathbf {x} _ {1} ; A _ {f} , B _ {f}\right) - f _ {b} \left(\mathbf {x} _ {2} ; A _ {f} , B _ {f}\right) \right|}{d _ {p} \left(\mathbf {x} _ {1} , \mathbf {x} _ {2}\right)} \\ = \frac {1}{d _ {p} (\mathbf {x} _ {1} , \mathbf {x} _ {2})} \cdot \left(\frac {d _ {p} (\mathbf {x} _ {1} , A _ {f})}{d _ {p} (\mathbf {x} _ {1} , A _ {f}) + d _ {p} (\mathbf {x} _ {1} , B _ {f})} - \frac {d _ {p} (\mathbf {x} _ {2} , A _ {f})}{d _ {p} (\mathbf {x} _ {2} , A _ {f}) + d _ {p} (\mathbf {x} _ {2} , B _ {f})}\right) \\ \leq \frac {1}{2 \lambda} \cdot \left(\frac {d _ {p} \left(\mathbf {x} _ {1} , A _ {f}\right) - d _ {p} \left(\mathbf {x} _ {2} , A _ {f}\right)}{d _ {p} \left(\mathbf {x} _ {1} , \mathbf {x} _ {2}\right)} \cdot \frac {d _ {p} \left(\mathbf {x} _ {2} , B _ {f}\right)}{d _ {p} \left(\mathbf {x} _ {2} , A _ {f}\right) + d _ {p} \left(\mathbf {x} _ {2} , B _ {f}\right)} \right. \tag {B.44} \\ + \left. \frac {d _ {p} \left(\mathbf {x} _ {1} , B _ {f}\right) - d _ {p} \left(\mathbf {x} _ {2} , B _ {f}\right)}{d _ {p} \left(\mathbf {x} _ {1} , \mathbf {x} _ {2}\right)} \cdot \frac {d _ {p} \left(\mathbf {x} _ {2} , A _ {f}\right)}{d _ {p} \left(\mathbf {x} _ {2} , A _ {f}\right) + d _ {p} \left(\mathbf {x} _ {2} , B _ {f}\right)}\right) \\ \leq \frac {1}{2 \lambda} \cdot (1 \cdot 1 + 1 \cdot 1) = \frac {1}{\lambda}, \\ \end{array}
$$