SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: A visualization of the manifold, semantic information, and adversarial examples. The manifold  $\mathcal{M}$  is represented by the grid surface. For clarity, we only visualize the above half of  $\mathcal{M}$  and omit the below half. Left: the  $2\lambda$ -separated semantic information  $A$ ,  $B \subset \mathcal{M}$  is painted in dark blue. According to Equation (6), the adversarial examples of  $f_{t} = f_{b}^{*} \cdot \phi_{\mathrm{off}}$  lie in the shaded bodies in light blue, which is outside of the data manifold. Right: the semantic information  $A$ ,  $B$  of the natural data is represented as the light blue area on  $\mathcal{M}$ . The raised parts upon  $\mathcal{M}$  are  $\mathcal{N}_{\delta}(A_f)$  (painted in orange) and  $\mathcal{N}_{\delta}(B_f)$  (painted in dark blue). The intersections of  $\mathcal{M}$  and  $\mathcal{N}_{\delta}(A_f)$  (or  $\mathcal{N}_{\delta}(B_f)$ ) is  $A_f$  (or  $B_f$ ), which are indicated by the areas surrounded by the inner contour lines. On the outer contour line of  $A_f$  and  $B_f$ , we have  $\phi_{\mathrm{on}} = 0$  since  $d_2(\mathbf{x}, \mathcal{N}_{\delta}(A_f \cup B_f)) = \delta$ .
Proposition 4.5. Take  $A_f = A$  and  $B_f = B$  in Equation (4) and denote the corresponding classifier by  $f_b^*$ . Then, for any given  $\lambda \geq \delta > 0$ , we have  $R_{\mathrm{std}}(f_b^*) = R_{\mathrm{adv}}(f_b^*, \delta) = 0$ .
Similar to Proposition 4.4, we specify a distance-based family of concentration multipliers.
Proposition 4.6 (Concentration multiplier, binary case). For any given  $r > 0$  and  $G \subset \mathbb{R}^d$ , denote
$$
\phi (\mathbf {x}) = \phi (\mathbf {x}; r, G) := \frac {r - d _ {p} (\mathbf {x} , G)}{r + d _ {p} (\mathbf {x} , G)}, \forall \mathbf {x} \in \mathbb {R} ^ {d}. \tag {5}
$$
Then  $\phi (\mathbf{x})$  is a concentration multiplier around  $G$ .
Propositions 4.4 and 4.6 introduce a combination of  $f_{b}$  and  $\phi$  in binary case. Both  $f_{b}$  and  $\phi$  can be parameterized by the semantic and geometrical information it learns. It is also possible to extend our analysis to other scenarios by specifying other combinations of  $f_{b}$  and  $\phi$ , which is out of the scope of our paper. In the following sections, we adopt the following assumption.
Assumption 3. The source and target model are chosen from the hypothesis class given by Definition 4.3, where  $\mathcal{F}_b$  and  $\Phi$  are the function classes introduced by Propositions 4.4 and 4.6.
With slight abuse of notation, we denote the hypothesis class in Assumption 3 by  $\mathcal{F}_{\mathcal{M}}$ . Intuitively, for any  $f = f_{b} \cdot \phi \in \mathcal{F}_{\mathcal{M}}$ , the semantic classifier controls the accuracy of  $f$ . The similarity between  $A, B$  and  $A_{f}, B_{f}$  reflects how well  $f_{b}$  fits the natural data. Moreover, the overlapping area  $(A \cup B) \cap (A_{f} \cup B_{f})$  should contain the training data of  $f$  (and thus be non-empty) if  $f$  has reached zero training error. The concentration multiplier helps classification by assigning lower confidence scores on those  $\mathbf{x} \notin G$ , but it might also bring in adversarial examples. Since we only impose mild conditions on the choice of  $G$  (i.e.,  $A_{f} \cup B_{f} \subset G$ ),  $\mathcal{F}_{\mathcal{M}}$  can capture complicated geometrical information. We will come back to the expressive power of  $\mathcal{F}_{\mathcal{M}}$  at Section 5.2.
4.1 The Transferability of Adversarial Examples
As a warm-up, we study the robustness of  $f \in \mathcal{F}_{\mathcal{M}}$  against on- and off-manifold adversarial examples without specifying  $f_{b}$  and  $\phi$ . The following proposition shows that the vulnerability of  $f \in \mathcal{F}_{\mathcal{M}}$  to on- and off-manifold adversarial examples are different.
Proposition 4.7. Let  $f = f_{b} \cdot \phi$ . Given  $R_{\mathrm{adv}}(f; \delta) \neq 0$ , we can obtain (w.p. 1) that
1.  $f$  suffers from off-manifold adversarial examples.
2. if  $f_{b}$  captures the semantic information of the natural data (i.e.,  $A \subset A_{f}$  and  $B \subset B_{f}$ ), then  $f$  is robust against on-manifold adversarial examples.
Remark. The results in Proposition 4.7 hold w.p. 1 since our proof ignores some minor events (e.g.,  $f(\mathbf{x}) = 0$ ) that occur w.p. 0. For the sake of simplicity, we will not specify those events that happen almost surely in the rest of this paper.
The first result of Proposition 4.7 implies that  $f \in \mathcal{F}_{\mathcal{M}}$  suffers from adversarial examples if and only if  $f$  suffers from off-manifold adversarial examples. In Section 4.2, we will discuss the existence of transferable adversarial examples based on this result.
We explain the properties of TBAs by proving the existence of  $f \in \mathcal{F}_{\mathcal{M}}$  that satisfies desired properties. First of all, we introduce a sub-family of  $\mathcal{F}_{\mathcal{M}}$  that is robust against on-manifold adversarial examples. The following concentration multipliers force  $f_{b}$  to concentrate around the data manifold, which brings off-manifold adversarial examples to  $f_{b}$  without introducing on-manifold adversarial examples. For any unspecified  $\alpha \in (0,1)$ , let
$$
\phi_ {\text {o f f}} (\mathbf {x}) := \phi (\mathbf {x}; \alpha \delta , \mathcal {M}) = \frac {\alpha \delta - d _ {p} (\mathbf {x} , \mathcal {M})}{\alpha \delta + d _ {p} (\mathbf {x} , \mathcal {M})}. \tag {6}
$$
Consider the classifiers induced by  $\phi_{\mathrm{off}}$ , i.e.,  $f = f_{b} \cdot \phi_{\mathrm{off}}$ . It is clear from Equation (6) that  $f(\mathbf{x}) = f_{b}(\mathbf{x})$  when  $\mathbf{x} \in \mathcal{M}$ . Combining this with the separated assumptions of semantic information, we can see that  $\phi_{\mathrm{off}}$  blocks the on-manifold adversarial examples. Figure 1 demonstrates that the target model  $f_{t} = f_{b}^{*} \cdot \phi_{\mathrm{off}}$  is only vulnerable to off-manifold perturbations. The following proposition shows that off-manifold adversarial examples are transferable even if the source model is inaccurate, which explains the phenomenon in Papernot et al. [7].
Theorem 4.8. Consider TBAs with perturbation radius  $\delta \in (0, \lambda]$ , target model  $f_{t} = f_{b}^{*} \cdot \phi_{\mathrm{off}}$  and source model  $f_{s} = f_{b} \cdot \phi_{\mathrm{off}}$ ,  $f_{b} \in \mathcal{F}_{b}$ . Denote the semantic information of  $f_{b}$  by  $A_{f}$  and  $B_{f}$ . Then, for  $\forall \mathbf{x} \in A \cup B$ , all adversarial examples (if exist) of  $f_{s}$  at  $\mathbf{x}$  are transferable if  $\mathbf{x} \in A_{f} \cup B_{f}$ .
Theorem 4.8 proves that all of the adversarial examples of  $f_{s}$  are transferable if the concentration multipliers of  $f_{t}$  and  $f_{s}$  are identical, given that  $(A \cup B) \cap (A_{f} \cup B_{f}) \neq \emptyset$ . In particular,  $f_{s}$  is not necessarily accurate, which is consistent with the empirical results in Papernot et al. [7]. Theorem 4.8 also implies that the vulnerability of ML models might be due to the non-robust geometrical information. Such vulnerability is transferable between models that learn similar geometrical information.
Next, we turn to the on-manifold adversarial examples. Proposition 4.7 shows that we cannot find an  $f \in \mathcal{F}_{\mathcal{M}}$  such that  $f$  suffers from only on-manifold adversarial examples. Instead, we specify a family of concentration multipliers that blocks out off-manifold adversarial examples that is "directly above" the manifold. In this case, we consider Euclidean space  $\mathbb{R}^d$  with  $l_{2}$ -norm and inner product  $\langle \mathbf{x}_1, \mathbf{x}_2 \rangle \coloneqq \mathbf{x}_1^T \mathbf{x}_2$ . For  $\forall \mathbf{x} \in \mathcal{M}$ , let  $T_{\mathbf{x}} \mathcal{M} \subset \mathbb{R}^d$  be the tangent space of  $\mathcal{M}$  at  $\mathbf{x}$ , i.e., the space spanned by the possible tangent directions that pass through  $\mathbf{x}$ . Now that we are considering an inner product space, let  $N_{\mathbf{x}} \mathcal{M} \subset \mathbb{R}^d$  be the normal space at  $\mathbf{x}$  such that  $\forall u \in N_{\mathbf{x}} \mathcal{M}$  and  $v \in T_{\mathbf{x}} \mathcal{M}$ , we have  $\langle u, v \rangle = 0$ .
Denote  $\mathcal{N}_r(S) := \{\mathbf{x}' \in \mathbb{R}^d : \exists \mathbf{x} \in S \text{ s.t. } d_2(\mathbf{x}, \mathbf{x}') < r, \mathbf{x}' - \mathbf{x} \in N_{\mathbf{x}}\mathcal{M}\}$  for any given  $r > 0$  and  $S \subset \mathcal{M}$ . We call  $\mathcal{N}_r(S)$  a tubular neighborhood of  $S$  if for  $\forall \mathbf{x}' \in \mathcal{N}_r(S)$ , there is an unique  $\mathbf{x} \in S$  such that  $\mathbf{x}' - \mathbf{x} \in N_{\mathbf{x}}\mathcal{M}$ . For any semantic classifier  $f_b$  with semantic information  $A_f$  and  $B_f$ , define
$$
\phi_ {\text {o n}} (\mathbf {x}) = \phi_ {\text {o n}} (\mathbf {x}; f _ {b}) := \phi (\mathbf {x}; \delta , \mathcal {N} _ {\delta} (A _ {f} \cup B _ {f}) \tag {7}
$$
and consider  $f = f_{b} \cdot \phi_{\mathrm{on}}(\cdot ;f_{b})$ . According to Equation (7),  $\phi_{\mathrm{on}}$  blocks the off-manifold adversarial examples of  $f$  that is "directly above"  $A_{f}$  and  $B_{f}$ . We use Figure 1 to visualize our idea. The following proposition provides a sufficient condition for adversarial examples that are not transferable.
Proposition 4.9. Consider TBA with perturbation radius  $\delta \in (0,\lambda ]$ , target model  $f_{t} = f_{b}^{*}\cdot \phi_{\mathrm{on}}(\cdot ,f_{b}^{*})$  and source model  $f_{s} = f_{b}\cdot \phi_{\mathrm{on}}(\cdot ;f_{b}),f_{b}\in \mathcal{F}_{b}$ . Denote
$$
S _ {\mathrm {c r t}} := \left(A \cap A _ {f}\right) \cup \left(B \cap B _ {f}\right), S _ {\mathrm {w r g}} := \left(A \cap B _ {f}\right) \cup \left(B \cap A _ {f}\right). \tag {8}
$$
Then, for  $\forall \mathbf{x} \in A \cup B$ , we have
1. if  $B(\mathbf{x},\delta)\cap \mathcal{M}\subset S_{\mathrm{crit}}\cup S_{\mathrm{wrg}}$  , then  $f_{t}$  and  $f_{s}$  are both robust against adversarial examples;
2. if  $B(\mathbf{x},\delta)\cap \mathcal{M}\subset A\cup B$  and  $\mathbf{x}\in A_f\cup B_f$  , then the adversarial examples of  $f_{s}$  at  $\mathbf{x}$  (if exists) cannot transfer to  $f_{t}$
We interpret  $S_{\mathrm{crit}}$  (or  $S_{\mathrm{wrg}}$ ) as the "correct (or wrong) semantic information" captured by  $f_b$ . When  $S_{\mathrm{crit}} = \operatorname{supp}(D)$  and  $S_{\mathrm{wrg}} = \emptyset$  (i.e.,  $f_b$  captures the semantic information of the natural data), the
first result of Proposition 4.9 implies that a large part of  $\mathbf{x} \in \mathrm{supp}(D)$  is robust against adversarial examples if we block the off-manifold adversarial examples that are "directly above" the semantic information. The second result of Proposition 4.9 implies that the potentially transferable adversarial examples are mostly located inside of the following set
$$
(A \cup B) \cap \left(S _ {\text {w r g}} \cup S _ {\text {c r t}}\right) ^ {c} = (A \cup B) \cap \left(A _ {f} \cup B _ {f}\right) ^ {c}, \tag {9}
$$
or at least close to its boundary. Here, we interpret  $(A\cup B)\cap (A_f\cup B_f)^c$  as the semantic information not contained in the training data.
In summary, the results in this section together provide a general view of the role that on- and off-manifold adversarial examples play in TBAs. Although both on- and off-manifold adversarial exist, ML models seem to be more vulnerable to off-manifold adversarial examples, and the off-manifold adversarial examples seem to play a more important role in TBAs.
4.2 The Existence of Transferable Adversarial Examples
Dong et al. [2] argue that the low success rates of TBAs are possibly due to the adversarial examples of  $f_{s}$  falling into a "non-adversarial region" of  $f_{t}$ . In this section, we try to formalize this explanation and study the properties of such non-adversarial regions.
According to Proposition 4.7,  $f \in \mathcal{F}_{\mathcal{M}}$  suffers from adversarial examples if and only if  $f$  suffers from off-manifold adversarial examples. This section is devoted to discussing the possible non-existence of off-manifold examples. Before we delve into this problem, let us introduce the notion of robust radius that is initially studied in the certified robustness problems [38, 39].
Definition 4.10 (Robust Radius, binary case). For any classifier  $f$ , the robust radius of  $f$  is defined as the minimum  $r \geq 0$  such that for  $\forall \mathbf{x} \in A \cup B$  and  $\mathbf{x}_1, \mathbf{x}_2 \in B(\mathbf{x}, r)$ , we have  $f(\mathbf{x}_1)f(\mathbf{x}_2) \geq 0$ .
Denote the robust radius of  $f$  by  $r_{\delta}(f)$ . In our model, the robust radius of  $f \in \mathcal{F}_{\mathcal{M}}$  is controlled by  $\phi$ . Based on this notion, the following example demonstrates how the existence of off-manifold adversarial examples depends on the shape of the data manifold  $\mathcal{M}$ .
Example 4.11. We use the same setting as Proposition 4.9 and consider the target model  $f_{t} = f_{b}^{*}\cdot \phi_{\mathrm{off}}$ . Given  $0 < r_1 < r_2 < \delta < r_3$ , denote  $\phi_{1} = \phi_{\mathrm{off}}(\cdot ;r_{1},\mathcal{M})$  and  $\phi_{2} = \phi_{\mathrm{off}}(\cdot ;r_{2},\mathcal{M})$ . Using Equation (6), we obtain that  $r_\delta (f_b^*\cdot \phi_1) = r_1 < \delta$  and  $r_\delta (f_b^*\cdot \phi_2) = r_2 < \delta$ . However,  $r_\delta (f) < \delta$  does not imply that all  $\mathbf{x}\in A\cup B$  are vulnerable to adversarial examples.
We demonstrated our idea in Figure A.1. Denote the dark blue surface in Figure A.1 by  $\mathcal{M}_{r_2} \coloneqq \{\mathbf{x} : d_2(\mathbf{x}, \mathcal{M}) = r_2\}$ . Observe that  $r_2$  is of the same magnitude as the "curvature" of  $\mathcal{M}$ , we can find a  $\mathbf{x}_0 \in A \cup B$  such that  $d_2(\mathbf{x}, \mathcal{M}_{r_2}) = r_3 > \delta$ , i.e.,  $\mathbf{x}_0$  is not vulnerable to adversarial examples. Note that all of such  $\mathbf{x}_0$  together form the non-adversarial region in Dong et al. [2].
Example 4.11 shows that the existence of the adversarial examples of  $f = f_{b} \cdot \phi_{\mathrm{off}}$  depends on the "curvature" of  $\mathcal{M}$ , which seems to be a rather agnostic result. However, we can quantify the "curvature" of  $\mathcal{M}$  by the following lemma.
Lemma 4.12 (Bredon [40]). Let  $\mathcal{M} \subset \mathbb{R}^d$  be a compact smooth manifold, then there is a  $\Delta > 0$  such that  $\mathcal{N}_{\Delta}\mathcal{M}$  is a tubular neighborhood of  $\mathcal{M}$ .
Lemma 4.12 can be viewed as a variant of the tubular neighborhood theorem (cf. Bredon [40]). The constant  $\Delta$  is decided by  $\mathcal{M}$  and can be used to evaluate the "curvature" of  $\mathcal{M}$  in Example 4.11. More specifically, the following proposition provides a sufficient condition for the existence of the off-manifold adversarial examples.
Theorem 4.13. Given perturbation radius  $\delta \in (0, \lambda]$  and target model  $f_{t} = f_{b}^{*} \cdot \phi_{\mathrm{off}}(\cdot; r, \mathcal{M})$ . Let  $\Delta$  be the constant specified in Lemma 4.12. Then, for  $\forall \mathbf{x} \in A \cup B$ , the off-manifold adversarial example of  $f_{t}$  at  $\mathbf{x}$  exists if  $r < \Delta$ .
Theorem 4.13 establishes a quantitative relationship between the existence of off-manifold examples and the "curvature" of the manifold.
5.1 Approximation with ReLU Networks
In applications, the target and source models of TBAs are mostly neural networks, e.g., ReLU networks. In this subsection, we use ReLU networks (cf. Definition 5.1) to approximate the classifiers  $f \in \mathcal{F}_{\mathcal{M}}$ . Moreover, we recover some of the results in Section 4 based on the approximated classifier.
We start by introducing the family of ReLU networks. Denote the rectified linear unit (ReLU) by  $\rho (\mathbf{x})\coloneqq \max \{\mathbf{x},\mathbf{0}\}$ , where  $\mathbf{0}$  is the zero vector in  $\mathbb{R}^d$  and  $\max \{\cdot ,\cdot \}$  takes the entry-wise maximum of  $\mathbf{x}$  and  $\mathbf{0}$ . The family of deep ReLU networks defined as follows.
Definition 5.1 (Deep ReLU networks). For any bias vector  $\pmb{b} = (b_{1},\dots ,b_{L})$  and sequence of weight matrices  $W = (W_{1},\dots ,W_{L})$ , we call function  $\tilde{f}:\mathbb{R}^d\to \mathbb{R}$  a (deep) ReLU networks parametrized by  $\pmb{b}$  and  $W$  if
$$
\tilde {f} (\mathbf {x}) = b _ {L} + \boldsymbol {W} _ {L} \rho \left(b _ {L - 1} + \boldsymbol {W} _ {L - 1} \rho \left(\dots \rho \left(b _ {1} + \boldsymbol {W} _ {1} \mathbf {x}\right) \dots\right)\right) \tag {10}
$$
for  $\forall \mathbf{x} \in \mathbb{R}^d$ . Let  $W_i$  be a matrix with  $m_i$  rows and  $n_i$  columns. We have  $n_1 = d$ ,  $m_L = 1$ , and  $m_i = n_{i+1}$  ( $1 \leq i \leq L-1$ ).
The non-constant entries in  $\pmb{b}$  and  $\pmb{W}$  are called the parameters of  $\tilde{f}$ . We say that a function is induced by ReLU networks if it can be expressed by the linear combination of finite many ReLU networks.
It has been proven [14, 41-43] that ReLU networks can approximate (w.r.t. the sup-norm) continuous functions with a bounded domain to any precision. Here, we denote the sup-norm of a function  $f: S \subset \mathbb{R}^d \to \mathbb{R}$  by  $\|f\|_{\infty} := \sup_{\mathbf{x} \in S} |f(\mathbf{x})|$ . Specifically, we have:
Lemma 5.2 (Li et al. [14]). Given  $l$ -Lipschitz function  $f:[0,1]^d \to [-1,1]$  and precision  $\epsilon > 0$ , there is a ReLU network  $\tilde{f}$  with  $\mathcal{O}((l / \epsilon)^d) \cdot \mathcal{O}(d^2 + d\log(1 / \epsilon))$  parameters that satisfies  $\| f - \tilde{f} \|_{\infty} \leq \epsilon$ .
Lemma 5.2 provides a useful tool to approximate the Lipschitz functions from  $[0,1]^d$  to  $[-1,1]$ . It is straightforward to check the Lipschitzness of  $f\in \mathcal{F}_M$ . As a result, given  $\epsilon >0$  and  $f\in \mathcal{F}_M$ , we can find an ReLU network  $\tilde{f}$  such that  $\| f - \tilde{f}\|_{\infty} < \epsilon$ . Unfortunately,  $\tilde{f}$  can no longer be decomposed into the product of  $f_{b}$  and  $\phi$ , which invalidates most of the results in Section 4. We have to make a detour to approximate  $f\in \mathcal{F}_M$ . As a first step, we approximate  $f_{b}$  and  $\phi$  by  $\tilde{f}_{b}$  and  $\tilde{\phi}$ , respectively.
Corollary 5.3. Let  $f_{b}$  be a semantic classifier with semantic information  $A_{f}$  and  $B_{f}$  that satisfy a  $2\lambda$ -separated property. Given  $\epsilon > 0$ , there is a ReLU network  $\tilde{f}_{b}$  with  $O((1 / \lambda \epsilon)^{d}) \cdot O(d^{2} + d\log (1 / \epsilon))$  parameters such that  $\| f_{b} - \tilde{f}_{b}\|_{\infty} \leq \epsilon$ .
Corollary 5.4. Given  $\epsilon >0$ ,  $r > 0$  and  $S\subset [0,1]^d$ , there is a ReLU network  $\tilde{\phi}$  with  $O((1 / r\epsilon)^{d})\cdot O(d^{2} + d\log (1 / \epsilon))$  parameters that can approximate  $\phi (\cdot ;r,S)$  to precision  $\epsilon$ .
Corollaries 5.3 and 5.4 approximate  $f_{b}$  and  $\phi$  with ReLU networks. It remains to approximate the product of two ReLU networks using the following lemma.
Lemma 5.5 (Yarotsky [41]). There is a function  $\tilde{\times} : [-1,1]^2 \to [-1,1]$  induced by ReLU network with  $O(\log^2(\epsilon^{-1}))$  parameters such that  $\tilde{\times}(x,y) = 0$  if  $xy = 0$ , and
$$
\sup  _ {x, y \in [ 0, 1 ]} | \tilde {x} (x, y) - x y | \leq \epsilon . \tag {11}
$$
The above lemma constructs a ReLU network that efficiently approximates the multiplying operator in  $\mathbb{R}^2$ . The following proposition shows that we can approximate  $f \in \mathcal{F}_M$  with ReLU networks to any precision.
Proposition 5.6. Given  $\epsilon, \lambda, \delta, r > 0$ , for any  $f \in \mathcal{F}_{\mathcal{M}}$ , there is a ReLU network  $\tilde{f}$  with
$$
O \left(\max  \left\{\frac {1}{\lambda \epsilon}, \frac {1}{r \epsilon} \right\} ^ {d}\right) \cdot O \left(d ^ {2} + d \log \left(\frac {1}{\epsilon}\right)\right) + O \left(\log^ {2} \left(\frac {1}{\epsilon}\right)\right) \tag {12}
$$
parameters that satisfies  $\| f - \tilde{f}\|_{\infty}\leq \epsilon$
The following theorem partially recovers the results in Theorem 4.8 with ReLU networks.
Theorem 5.7. Consider TBAs with perturbation radius  $\delta \in (0, \lambda/2]$ , target model  $f_t = f_b^* \cdot \phi_{\mathrm{off}}$  and source model  $f_s = f_b \cdot \phi_{\mathrm{off}}$ ,  $f_b \in \mathcal{F}_b$ . Denote the semantic information of  $f_b$  by  $A_f$  and  $B_f$ . Given  $\epsilon \leq 0.1$ , let  $\tilde{f}_t$  and  $\tilde{f}_s$  be ReLU networks that satisfy
$$
\left\| \tilde {f} _ {t} - f _ {t} \right\| _ {\infty} \leq \epsilon , \left\| \tilde {f} _ {s} - f _ {s} \right\| _ {\infty} \leq \epsilon . \tag {13}
$$
Then, for  $\forall \mathbf{x} \in (A \cup B) \cap (A_f \cup B_f)$ , the adversarial examples  $\mathbf{x}_a$  (if exist) of  $\tilde{f}_s$  satisfies
$$
\tilde {f} _ {t} (\mathbf {x}) \cdot \tilde {f} _ {t} \left(\mathbf {x} _ {a}\right) \leq 2 \epsilon (1 + \epsilon) ^ {2} + 2 \epsilon^ {2}. \tag {14}
$$
Notice that Equation (14) does not imply that the adversarial examples are transferable. Instead, it can only reduce the confidence in the decision made by  $\tilde{f}_t$ . The results in Theorem 4.8 cannot be fully recovered. When  $f$  is close to zero, whether the approximated ReLU network  $\tilde{f}$  is greater or less than zero is hard to decide.
The following theorem recovers Proposition 4.9 after modifying some parameters.
Theorem 5.8. Consider TBAs with perturbation radius  $\delta \in (0, \lambda/2]$ , target model  $f_t = f_b^* \cdot \phi_{\mathrm{off}}$  and source model  $f_s = f_b \cdot \phi_{\mathrm{off}}$ ,  $f_b \in \mathcal{F}_b$ . Denote the semantic information of  $f_b$  by  $A_f$  and  $B_f$ . Given  $\epsilon \leq 0.1$ , let  $\tilde{f}_t$  and  $\tilde{f}_s$  be ReLU networks that satisfy Equation (13). Then, for  $\forall \mathbf{x} \in A \cup B$ , we have
1. if  $B(\mathbf{x},\delta)\cap \mathcal{M}\subset S_{\mathrm{crit}}\cup S_{\mathrm{wrg}}$  , then  $\tilde{f}_t$  and  $\tilde{f}_s$  are both robust against adversarial examples;
2. if  $B(\mathbf{x},\delta) \cap \mathcal{M} \subset A \cup B$  and  $\mathbf{x} \in A_f \cup B_f$ , then the adversarial examples of  $\tilde{f}_s$  at  $\mathbf{x}$  (if exists) cannot transfer to  $\tilde{f}_t$ .
By approximating the classifiers in  $\mathcal{F}_{\mathcal{M}}$ , we build a bridge between the less-developed theory behind TBAs and the huge amount of theoretical works analyzing ReLU networks.
5.2 The Expressiveness of the Manifold Attack Model
In learning theory, the expressive power of a hypothesis class  $\mathcal{F}$  evaluates the performance of  $f\in \mathcal{F}$  in fitting natural data; see [44] for a comprehensive overview. Previous works have tried to explain the transferability based on less expressive hypothesis classes, e.g., linear classifiers [32] and the linear combination of "features" [17]. In this subsection, we study the expressive power of our model.
The main goal of TBAs is not to fit the natural data. Instead, for any well-trained classifier  $f^{*}:[0,1]^{d}\to \{-1,1\}$ , i.e.,  $R_{\mathrm{std}}(f^{*}) = 0$ , the goal of our model is to find  $f\in \mathcal{F}_{\mathcal{M}}$  such that the adversarial examples of  $f$  fits those of  $f^{*}$ . More specifically, we have
Proposition 5.9. For any classifier  $f^{*}$  with  $R_{\mathrm{std}}(f^{*}) = 0$  and perturbation radius  $\delta \in (r_{\delta}(f^{*}), \lambda)$ , there is  $f \in \mathcal{F}_{M}$  such that 1)  $R_{\mathrm{std}}(f) = R_{\mathrm{std}}(f^{*})$ , and 2) for  $\forall \mathbf{x} \in A \cup B$ , if  $\mathbf{x}_a$  is an adversarial example of  $f^{*}$  at  $\mathbf{x}$ , then exists  $\mathbf{x}_a' \in B(\mathbf{x}_a, r_{\delta}(f^{*}) / 4)$  such that  $\mathbf{x}_a'$  is an adversarial example of  $f$ .
The above proposition implies that our proposed model can cover a wide range of classifiers and is thus sufficient for the study of TBAs on manifolds.