SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yanbo Chen, Weiwei Liu*
{yanbo.acad, liuweiwei863}@gmail.com
Transfer-based attacks [1] are a practical method of black-box adversarial attacks in which the attacker aims to craft adversarial examples from a source model that is transferable to the target model. Many empirical works [2-6] have tried to explain the transferability of adversarial examples from different angles. However, these works only provide ad hoc explanations without quantitative analyses. The theory behind transfer-based attacks remains a mystery.
This paper studies transfer-based attacks under a unified theoretical framework. We propose an explanatory model, called the manifold attack model, that formalizes popular beliefs and explains the existing empirical results. Our model explains why adversarial examples are transferable even when the source model is inaccurate as observed in Papernot et al. [7]. Moreover, our model implies that the existence of transferable adversarial examples depends on the "curvature" of the data manifold, which further explains why the success rates of transfer-based attacks are hard to improve. We also discuss our model's expressive power and applicability.
Machine learning (ML) models are vulnerable to adversarial examples [8, 9]. It has been observed that adversarial examples of one model can fool another model [9, 10], i.e., adversarial examples are transferable. Utilizing this transferability, researchers have developed some practical black-box attack methods, in which they first obtain a source model and then use the adversarial examples of the source model to attack the target model [5, 11]. This type of attack is referred to in the literature as transfer-based attacks (TBA).
TBAs have many intriguing properties. For instance, Papernot et al. [7] find that the adversarial examples of an inaccurate source model can transfer to the target model with a non-negligible success rate. Besides, the success rates of TBAs are constantly lower than other methods of black-box attacks [5, 6, 12]; the low success rate seems to be an intrinsic property of TBAs.
Previous works have tried to explain these properties from different perspectives [2-6]. Unfortunately, these works only provide ad hoc explanations for the properties of TBAs without quantitative analyses under a unified framework. For example, Dong et al. [6] suggest that the low success rates of TBAs are due to the adversarial examples of the source model falling into a "non-adversarial region" of the target model, while the properties of such regions are not discussed quantitatively. As for the cause of transferability, a self-evident explanation is that the source and target models have similar decision boundaries [10]. However, these similarities are hard to characterize, and it is even harder to associate
the decision boundary similarity with the properties of TBAs. More importantly, this explanation contradicts the abovementioned experimental results in Papernot et al. [7], i.e., adversarial examples are also transferable when the source and target model does not have similar decision boundaries. We refer the readers to Section 2 for more related works. In summary, existing analyses make good sense under their respective settings, but it is unclear whether they are in accord with each other. It is natural to ask the following question:
Can we find a unified theoretical framework that explains the properties of TBAs?
This paper gives a positive answer to this question. More specifically, we propose an explanatory model, called the manifold attack model, that formalizes popular beliefs and explains the existing empirical results in a unified theoretical framework. Our model assumes that the natural data lies on a low-dimensional manifold [13-16] denoted by  $\mathcal{M}$ ; see Section 2 for more related works on this assumption. As the central part of our model, we specify a hypothesis class  $\mathcal{F}_{\mathcal{M}}$  and assume that both the source and the target model come from  $\mathcal{F}_{\mathcal{M}}$ . The construction of  $\mathcal{F}_{\mathcal{M}}$  is motivated by the following two widespread beliefs. In order to classify unseen data correctly, ML models are believed to have extracted semantic information (i.e., the ground truth) of the natural data after training. On the other hand, ML models are also able to learn non-robust features that generalize well in test phrases [17, 18]. These features reflect the geometrical information of the training data [17].
In our model, the hypothesis class  $\mathcal{F}_{\mathcal{M}}$  is designed to capture both semantic and geometric information of the data. As we will define in Section 4, the classifiers in  $\mathcal{F}_{\mathcal{M}}$  can be decomposed into the product of a basis classifier  $f_{b}$  and a multiplier  $\phi$ , i.e.,  $f = f_{b} \cdot \phi$  for  $\forall f \in \mathcal{F}_{\mathcal{M}}$ . We characterize the semantic information as separated sets  $A^{1}, A^{2}, \dots, A^{k} \subset \mathcal{M}$  (for a  $k$ -class classification task) and capture this information by letting  $f_{b}$  take different values on these sets. The geometrical information  $G$  is interpreted as the "approximated shape" of the data manifold, and we let  $\phi$  concentrate around  $G$ . By multiplying  $f_{b}$  and  $\phi$  together, we obtain a classifier that captures the semantic and geometry information of the training data. In brief, our model assumes that both the source and target models come from a specified hypothesis class and the natural data is drawn from a low-dimensional manifold.
Despite its simple form, the manifold attack model provides us with powerful tools to theoretically analyze the properties of TBAs. More specifically, Theorem 4.8 proves that the off-manifold adversarial examples are transferable when the source model is inaccurate, which explains the empirical results from Papernot et al. [7]. By further discussing the existence of off-manifold adversarial examples, we theoretically explain why the success rates of TBAs are hard to improve. Moreover, Theorem 4.13 quantitatively discusses the relationship between the existence of transferable adversarial examples and the "curvature" of the data manifold  $\mathcal{M}$ , which formalizes the explanation in Dong et al. [6].
In addition to the explanatory results, we further discuss the expressive power and the possible extensions of our model in general applications. Section 5.1 proves that our model is extensible, i.e., we can replace the hypothesis class  $\mathcal{F}_M$  by the family of ReLU networks while proving similar results. Our model builds a bridge between the less developed theory behind TBAs and the huge amount of theoretical works analyzing ReLU networks. Furthermore, Section 5.2 demonstrate that the expressive power of our model is strong enough for the study of TBAs. We also provide a detailed discussion on the applicability of our model in Appendix A.
In summary, we propose a model that is theoretically tractable and consistent with existing results. It formalizes widespread beliefs and explains the properties of TBAs under a unified framework. The remainder of this paper is structured as follows. Sections 2 and 3 introduce the related works and terminologies, respectively. We propose our model and present the explanatory results in Section 4. Section 5 makes further discussions on our model. Section 6 summarizes this paper. Some remarks and the omitted proofs can be found in the appendix.
The related works of TBAs and the low-dimensional manifold assumption are summarized as follows.
2.1 Transfer-Based Attacks
The research on the adversarial robustness of ML can be roughly divided into adversarial attacks [8, 9, 19], defenses [20-22], and the analyses of the robustness of existing methods [23-26]. Adversarial
attacks can be divided into two classes based on whether the attackers have the gradient information of the target model, i.e., white-box [27] and black-box [7] attacks. TBAs are one of the two main approaches to performing black-box attacks on ML models. Apart from TBAs, the other main approach is the optimization-based attack that approximates the gradient of the target model and performs white-box attacks thereafter [28-30]. While TBAs require much less information from the target model (and thus more practical) than optimize-based attacks, the success rates of TBAs are constantly lower, even when the source models are almost accurate [2, 3, 5, 6, 11].
There are a few theoretical works that have tried to explain the properties of TBAs. However, these works rely heavily on either simple models or strong assumptions. Apart from those mentioned in Section 1, the seminal work of Goodfellow et al. [9] tries to explain the transferability of adversarial examples using linear models. Charles et al. [31] theoretically prove, in the context of linear classifiers and two-layer ReLU networks, that transferable adversarial examples exist. However, [31] does not explain those properties of TBAs mentioned in Section 1. Gilmer et al. [32] assume that the natural data is drawn from a "concentric spheres dataset", which can be viewed as a special form of the low-dimensional data manifold. In comparison to previous works, our model requires milder assumptions on both data distribution and hypothesis class, while providing a more detailed discussion of the properties of TBAs.
2.2 The Low-Dimensional Manifold Assumption
The low-dimensional manifold assumption is commonly seen in many areas of research, ranging from classic approximation theory [13, 14] to computer vision [15, 16]. Under this assumption, it is intuitive to divide the adversarial examples into two groups based on whether they lie on or off the data manifold. Both on-manifold [15, 33] and off-manifold [32, 34] adversarial examples have been studied in many previous works. In most cases, a classifier is simultaneously vulnerable to both on- and off-manifold adversarial examples [33].
In this paper, we discuss the transferability of both on- and off-manifold adversarial examples. We are mainly motivated by the techniques from Zhang et al. [35] and Li et al. [14]. The main results of Zhang et al. [35] decompose the adversarial risk based on the position of the adversarial example and discuss the existence of different types of adversarial examples; Zhang et al. [35] provides us with useful tools to analyze the existence of transferable adversarial examples. Li et al. [14] consider robust generalization and approximate continuous classifiers by ReLU networks. In this paper, we prove that the hypothesis class  $\mathcal{F}_M$  in our model can also be approximated by ReLU networks.
3 Problem Setup
This section introduces the basic setups and terminologies. We use a black triangle sign  $(\triangle)$  to indicate the end of assumptions, definitions, or remarks.
3.1 Notations
Let  $\mathbb{R}^d$  be the real vector space equipped with the  $l_{p}$ -norm, where  $p \in [1, +\infty]$ . In this paper, we consider the  $k$ -class classification problems  $(k \geq 2)$  on  $\mathbb{R}^d$ . Denote the data-label pairs by  $(\mathbf{x}, y)$ , in which  $y$  is chosen from some label space  $\mathcal{Y}$ . By convention, we let  $\mathcal{Y} = \{-1, 1\}$  in binary classification and  $\mathcal{Y} = \{1, 2, \dots, k\}$  when  $k > 2$ . Unless otherwise specified, we assume that the natural data lie on a low-dimensional manifold  $\mathcal{M} \subset \mathbb{R}^d$ . More specifically, we adopt the following assumption.
Assumption 1 (Low-dimensional Manifold). Let  $\mathcal{M} \subset [0,1]^d$  be a compact smooth manifold² and assume that the dimension of  $\mathcal{M}$  is less than  $d$ . Let  $D$  be some given continuous distribution that is supported on  $\mathcal{M}$ . We say that a random variable  $\mathbf{x}$  is natural data if  $\mathbf{x} \sim D$ .
Denote the support set of  $D$  by  $\operatorname{supp}(D)$ . With slight abuse of notation, let  $y$  also be a mapping from  $\operatorname{supp}(D)$  to  $\mathcal{Y}$  that assigns each natural data  $\mathbf{x} \sim D$  with a true label  $y = y(\mathbf{x})$ . Notice that only natural data is endowed with a true label. Since  $\operatorname{supp}(D) \subset [0,1]^d$ , we will make no distinction between  $\mathbb{R}^d$  and  $[0,1]^d$  in the remainder of this paper without loss of generality (WLOG).
2Some realistic datasets are supported on  $[0,1]^d$  , e.g., MNIST and Fashion-MNIST.
Let  $d_p$  be the metric induced by the  $l_{p}$ -norm and  $B(\mathbf{x};r)$  the  $l_{p}$ -ball around  $\mathbf{x}$  with radius  $r$ . For any subset  $S\subset \mathbb{R}^d$ , let  $d_p(\mathbf{x},S)\coloneqq \inf_{\mathbf{x}'\in S}d_p(\mathbf{x},\mathbf{x}')$  be the distance between  $S$  and  $\mathbf{x}$ . Let the classifiers  $f$  be functions defined on  $\mathbb{R}^d$ . When  $k > 2$ , we consider  $f(\mathbf{x}) = (f^{(1)}(\mathbf{x}),f^{(2)}(\mathbf{x}),\dots ,f^{(k)}(\mathbf{x}))^T$  that maps  $\mathbf{x}\in \mathbb{R}^d$  to  $f(\mathbf{x})\in \mathbb{R}^k$ . For ease of notation, let  $y(f,\mathbf{x})\coloneqq \arg \max_{1\leq i\leq k}f^{(i)}(\mathbf{x})$  be the output class of  $f$  for  $\forall \mathbf{x}\in \mathbb{R}^d$ . Since  $D$  is continuous,  $y(f,\mathbf{x})$  take unique value w.p. (with probability) 1 when  $\mathbf{x}\sim D$ . In binary classification problems, we consider  $f:\mathbb{R}^d\to \mathbb{R}$  and  $y(f,\mathbf{x}) = \mathrm{sign}(f(\mathbf{x}))$ .
Given perturbation radius  $\delta$  and natural data  $\mathbf{x} \sim D$ , we call  $\mathbf{x}_a \in B(\mathbf{x}; \delta)$  an adversarial example of  $f$  at  $\mathbf{x}$  if  $y(f, \mathbf{x}) \neq y(f, \mathbf{x}_a)$ . Given classifier  $f$ , let the standard risk of  $f$  be  $R_{\mathrm{std}}(f) \coloneqq \mathbb{P}_{\mathbf{x} \sim D}[y(f, \mathbf{x}) \neq y(\mathbf{x})]$ , and the adversarial risk of  $f$  with regard to (w.r.t.)  $\delta$  is defined as
$$
R _ {\mathrm {a d v}} (f; \delta) := \mathbb {P} _ {\mathbf {x} \sim D} [ \exists \mathbf {x} _ {a} \in B (\mathbf {x}; \delta) s. t. y (f, \mathbf {x}) \neq y (f, \mathbf {x} _ {a}) ]. \tag {1}
$$
Since TBA is often against accurate target models in practice, our paper assumes that the standard risk of the target model  $f_{t}$  is  $R_{\mathrm{std}}(f_t) = 0$  for the sake of simplicity. Given such accurate  $f_{t}$ , the goal of a TBA is to find adversarial examples of  $f_{t}$ . Specifically, the attacker needs to obtain a source model  $f_{s}$  and craft adversarial examples of  $f_{s}$  using white-box attacks. Given natural data  $\mathbf{x} \sim D$  and an adversarial example  $\mathbf{x}_a \in B(\mathbf{x};\delta)$  of  $f_{s}$  at  $\mathbf{x}$ , we say that  $\mathbf{x}_a$  transfers to  $f_{t}$  if  $y(f_{t},\mathbf{x}) \neq y(f_{t},\mathbf{x}_{a})$ .
3.2 The On- and Off-Manifold Adversarial Examples
In this paper, the on- and off-manifold adversarial examples are defined as follows.
Definition 3.1 (On- and off-manifold adversarial examples). For any given classifier  $f$ , perturbation radius  $\delta$  and natural data  $\mathbf{x} \sim D$ , let  $\mathbf{x}_a \in B(\mathbf{x};\delta)$  be an adversarial example of  $f$  at  $\mathbf{x}$ . We call  $\mathbf{x}_a$  an on-manifold adversarial example  $\mathbf{x}_a \in \mathcal{M}$ , or an off-manifold adversarial example if  $\mathbf{x}_a \notin \mathcal{M}$ .
We say that a classifier  $f$  suffers from, or is vulnerable to (on-, off-manifold) adversarial examples if  $\exists \mathbf{x} \sim D, \mathbf{x}_a \in B(\mathbf{x}; \delta)$  such that  $\mathbf{x}_a$  is an (on-, off-manifold) adversarial examples of  $f$  at  $\mathbf{x}$ . If  $f$  does not suffer from (on-, off-manifold) adversarial examples, then we consider it to be robust against (on-, off-manifold) adversarial examples.
In most cases, the adversarial examples of  $f_{t}$  are regarded as imperceptible to humans (e.g., in a cat-dog classification task, the adversarial examples of cat images still look like cats), which implies that adversarial examples are not naturally generated data. Otherwise, the adversarial examples would be endowed with the same label as their corresponding natural data, which leads to a contradiction since  $R_{\mathrm{std}}(f_t) = 0$ . In other words, adversarial examples should not lie in the support of  $D$ . To ensure this, we additionally assume that the natural data with different true labels are separated from each other. Notice that there is an intrinsic connection between the natural data  $\mathbf{x}$  and its true label  $y(\mathbf{x})$  which is often referred to as "semantic information" of the natural data. In this paper, we formalize the semantic information of the natural data as a family of sets  $\{A^j : j \in \mathcal{Y}\}$ , where
$$
A ^ {j} := \left\{\mathbf {x} \in \operatorname {s u p p} (D): y (\mathbf {x}) = j \right\}, \forall j \in \mathcal {Y}. \tag {2}
$$
In practice, the semantic information  $A^i$  and  $A^j$  is often separated if  $i \neq j$  (e.g., Remark A.1). In this paper, we assume that the semantic information is separated in the following sense.
Assumption 2 (Separated Semantic Information). Given  $\lambda >0$ , for  $\forall i\neq j\in \mathcal{Y}$ , we assume that  $A^i$  and  $A^j$  are  $2\lambda$ -separated. Here, for any given subsets  $E,F\subset \mathbb{R}^{d}$  and  $r > 0$ , we say that  $E$  and  $F$  are  $r$ -separated if  $d_{p}(\mathbf{x}_{1},\mathbf{x}_{2})\geq r$  for  $\forall \mathbf{x}_1\in E$  and  $\mathbf{x}_2\in F$ .
With the help of Assumption 2, it is easy to check that the adversarial examples would not lie in the support set of  $D$  if  $\delta < \lambda$ . In the rest of this paper, we will treat  $\delta$  and  $\lambda$  as fixed parameters and always assume that  $\delta < \lambda$ .
4 The Manifold Attack Model
Our proposed model specifies a hypothesis class that has two components. The first one is a semantic classifier that tries to capture the semantic information of the natural data. Similar to Equation (2), we call  $\{A_f^j : j \in \mathcal{Y}\}$  the semantic information learned by a classifier  $f$  if  $y(f, \mathbf{x}) = j$  for  $\forall \mathbf{x} \in A_f^j$  and  $\forall j \in \mathcal{Y}$ . To avoid trivial discussion, we assume that  $A_f^j \neq \emptyset$  for  $\forall j \in \mathcal{Y}$ . The following definition specifies a family of classifiers that learn separated semantic information.
Definition 4.1 (Semantic classifier). We call  $f_{b}$  a semantic classifier if there is a family of pairwise  $2\lambda$ -separated set  $\{A_{f}^{j}: j \in \mathcal{Y}\}$  such that  $\{A_{f}^{j}: j \in \mathcal{Y}\}$  is the semantic information learned by  $f_{b}$ .
It is easy to check that  $R_{\mathrm{std}}(f_b) = 0$  if  $A^j \subset A_f^j$  for  $\forall j \in \mathcal{Y}$ . Intuitively, the accuracy of a classifier depends on how well it learns the semantic information of the natural data.
The second component of our model is a function that captures the geometric structure of the data.
Definition 4.2 (Concentration multiplier). Given  $G \subset \mathbb{R}^d$ , we call  $\phi$  a concentration multiplier around  $G$  if  $\phi(\mathbf{x}) = 1$  for  $\forall \mathbf{x} \in G$ , and  $\phi(\mathbf{x}_1) < \phi(\mathbf{x}_2)$  for  $\forall \mathbf{x}_1, \mathbf{x}_2 \in \mathbb{R}^d$  with  $d_p(\mathbf{x}_1, G) > d_p(\mathbf{x}_2, G)$ .
Intuitively, the geometric information  $G$  can be interpreted as the "approximated shape" of the natural data. The concentration multipliers would assign a much lower confidence score to those data points that are outside of or far away from  $G$ . In the ideal case,  $G$  precisely captures the shape of  $\mathcal{M}$ , which enables  $\phi$  to detect the off-manifold data points (i.e., the out-of-distribution (OD) data [36, 37]).
As mentioned in Section 1, ML models often capture both the semantic and geometrical information of the natural data. Therefore, we assume that the source and target models can be decomposed into the product of a semantic classifier and a concentration multiplier. More specifically, we consider the hypothesis class defined as follows.
Definition 4.3. Consider the  $k$ -class classification problems  $(k \geq 2)$  on  $[0,1]^d$ . Let  $\mathcal{F}_b$  and  $\Phi$  be a collection of semantic classifiers and concentration multipliers, respectively. When  $k > 2$ , let  $\mathcal{F}_M$  be the family of all classifiers  $f: [0,1]^d \to \mathbb{R}^k$  that satisfy
$$
f ^ {(j)} (\mathbf {x}) = f _ {b} ^ {(j)} (\mathbf {x}) \cdot \phi (\mathbf {x}) \text {f o r} \forall \mathbf {x} \in [ 0, 1 ] ^ {d} \text {a n d} \forall j \in \mathcal {Y}. \tag {3}
$$
In the binary classification case, we let  $f(\mathbf{x}) = f_{b}(\mathbf{x}) \cdot \phi(\mathbf{x})$  for  $\forall \mathbf{x} \in [0,1]^{d}$ . Here,  $f_{b}$  and  $\phi$  are chosen from  $\mathcal{F}_{b}$  and  $\Phi$ , respectively. Since the semantic information is contented in the manifold, we additionally assume that  $\cup_{j \in \mathcal{Y}} A_{f}^{j} \subset G$ , in which  $\{A_{f}^{j} : j \in \mathcal{Y}\}$  is the semantic information learned by  $f_{b}$  and  $G$  is the approximated shape of  $\mathcal{M}$  learned by  $\phi$ .
Definitions 4.1 to 4.3 together with Assumptions 1 and 2 establish the abstract framework for the manifold attack model. This model formalizes how a classifier captures the semantic and geometric information from natural data. It is worth noticing that choices of  $f \in \mathcal{F}_M$  in Definition 4.3 remains largely arbitrary if we choose  $\mathcal{F}_b$  (and  $\Phi$ ) to be the family of all possible semantic classifiers (and concentration multiplier) introduced in Definition 4.1 (and Definition 4.2). In fact, for any  $f$  that learns  $2\lambda$ -separated semantic information  $\{A_f^j : j \in \mathcal{Y}\}$  and any  $\phi$  such that  $(f(\mathbf{x}) = 0) \Rightarrow (\phi(\mathbf{x}) = 0)$  for  $\forall \mathbf{x} \in \mathbb{R}^d$ , we can simply let  $f_b^{(j)}(\mathbf{x}) = f^{(j)}(\mathbf{x}) \cdot \phi^{-1}(\mathbf{x})$  for  $\forall \mathbf{x} \neq 0$ . It is easy to check that  $f_b$  is a semantic classifier as long as  $A_f^j \cap G \neq \emptyset$  for  $\forall j \in \mathcal{Y}$ .
The generality in Definition 4.3 comes at the expense of theoretical intractability. Take the semantic classifier for example. In Definition 4.1,  $f_{b}$  cannot be determined by the semantic information it learns. Moreover,  $f_{b}$  cannot even be parameterized according to this definition. There is an implicit tradeoff between generality and theoretical tractability: quantitative analysis of specific scenarios might need more delicate definitions. Next, we introduce one specific combination of  $f_{b}$  and  $\phi$  to carry on analyzing the properties of TBAs.
For the sake of simplicity, we only present analyses and results in the context of binary classification in Sections 4 and 5. For ease of notation, let  $A \coloneqq \{\mathbf{x} : y(\mathbf{x}) = 1\}$  and  $B \coloneqq \{\mathbf{x} : y(\mathbf{x}) = -1\}$ , i.e., the semantic information of binary natural data. The following proposition specifies a sub-family of semantic classifiers:
Proposition 4.4 (semantic classifier, binary case). Given  $2\lambda$ -separated sets  $A_{f}, B_{f} \subset \mathcal{M}$ . Define:
$$
f _ {b} (\mathbf {x}) = f _ {b} (\mathbf {x}; A _ {f}, B _ {f}) := \frac {d _ {p} (\mathbf {x} , B _ {f}) - d _ {p} (\mathbf {x} , A _ {f})}{d _ {p} (\mathbf {x} , B _ {f}) + d _ {p} (\mathbf {x} , A _ {f})}. \tag {4}
$$
Then,  $f_{b}$  is a semantic classifier. In particular, we can obtain from Equation (4) that  $f_{b}(\mathbf{x}) > 0$  if  $\mathbf{x}$  is closer (w.r.t.  $d_{p}$ ) to  $A_{f}$  than  $B_{f}$  and  $f_{b}(\mathbf{x}) < 0$  otherwise.
Let  $\mathcal{F}_{\mathrm{bin}}$  the collection of all semantic classifiers defined by Equation (4) for all possible  $2\lambda$ -separated  $A_{f}, B_{f} \subset \mathcal{M}$ . The following proposition shows that a semantic classifier can be accurate and robust.