Title: A Theory of Transfer-Based Black-Box Attacks: Explanation and Implications
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning Theory
Abstract: Transfer-based attacks are a practical method of black-box adversarial attacks, in which the attacker aims to craft adversarial examples from a source (surrogate) model that is transferable to the target model. A wide range of empirical works has tried to explain the transferability of adversarial examples from different angles. However, these works only provide ad hoc explanations without quantitative analyses. The theory behind transfer-based attacks remains a mystery.
This paper studies transfer-based attacks under a unified theoretical framework. We propose an explanatory model, called the manifold attack model, that formalizes popular beliefs and explains the existing empirical results. Our model explains why adversarial examples are transferable even when the source model is inaccurate. Moreover, our model implies that the existence of transferable adversarial examples depends on the “curvature” of the data manifold, which quantitatively explains why the success rates of transfer-based attacks are hard to improve. We also discuss the expressive power and the possible extensions of our model in general applications.

A Theory of Transfer-Based Black-Box Attacks: Explanation and Implications

Abstract

Transfer-based attacks [1] are a practical method of black-box adversarial attacks in which the attacker aims to craft adversarial examples from a source model that is transferable to the target model. Many empirical works [2-6] have tried to explain the transferability of adversarial examples from different angles. However, these works only provide ad hoc explanations without quantitative analyses. The theory behind transfer-based attacks remains a mystery.

This paper studies transfer-based attacks under a unified theoretical framework. We propose an explanatory model, called the manifold attack model , that formalizes popular beliefs and explains the existing empirical results. Our model explains why adversarial examples are transferable even when the source model is inaccurate as observed in Papernot et al. [7]. Moreover, our model implies that the existence of transferable adversarial examples depends on the 'curvature' of the data manifold, which further explains why the success rates of transfer-based attacks are hard to improve. We also discuss our model's expressive power and applicability.

1 Introduction

Machine learning (ML) models are vulnerable to adversarial examples [8, 9]. It has been observed that adversarial examples of one model can fool another model [9, 10], i.e., adversarial examples are transferable. Utilizing this transferability, researchers have developed some practical black-box attack methods, in which they first obtain a source model and then use the adversarial examples of the source model to attack the target model [5, 11]. This type of attack is referred to in the literature as transfer-based attacks (TBA).

TBAs have many intriguing properties. For instance, Papernot et al. [7] find that the adversarial examples of an inaccurate source model can transfer to the target model with a non-negligible success rate. Besides, the success rates of TBAs are constantly lower than other methods of black-box attacks [5, 6, 12]; the low success rate seems to be an intrinsic property of TBAs.

Previous works have tried to explain these properties from different perspectives [2-6]. Unfortunately, these works only provide ad hoc explanations for the properties of TBAs without quantitative analyses under a unified framework. For example, Dong et al. [6] suggest that the low success rates of TBAs are due to the adversarial examples of the source model falling into a 'non-adversarial region' of the target model, while the properties of such regions are not discussed quantitatively. As for the cause of transferability, a self-evident explanation is that the source and target models have similar decision boundaries [10]. However, these similarities are hard to characterize, and it is even harder to associate

the decision boundary similarity with the properties of TBAs. More importantly, this explanation contradicts the abovementioned experimental results in Papernot et al. [7], i.e., adversarial examples are also transferable when the source and target model does not have similar decision boundaries. We refer the readers to Section 2 for more related works. In summary, existing analyses make good sense under their respective settings, but it is unclear whether they are in accord with each other. It is natural to ask the following question:

Can we find a unified theoretical framework that explains the properties of TBAs?

This paper gives a positive answer to this question. More specifically, we propose an explanatory model, called the manifold attack model , that formalizes popular beliefs and explains the existing empirical results in a unified theoretical framework. Our model assumes that the natural data lies on a low-dimensional manifold [13-16] denoted by M ; see Section 2 for more related works on this assumption. As the central part of our model, we specify a hypothesis class FM and assume that both the source and the target model come from FM . The construction of FM is motivated by the following two widespread beliefs. In order to classify unseen data correctly, ML models are believed to have extracted semantic information (i.e., the ground truth) of the natural data after training. On the other hand, ML models are also able to learn non-robust features that generalize well in test phrases [17, 18]. These features reflect the geometrical information of the training data [17].

In our model, the hypothesis class FM is designed to capture both semantic and geometric information of the data. As we will define in Section 4, the classifiers in FM can be decomposed into the product of a basis classifier f b and a multiplier ϕ , i.e., f = f b · ϕ for ∀ f ∈ FM . We characterize the semantic information as separated sets A 1 , A 2 , · · · , A k ⊂ M (for a k -class classification task) and capture this information by letting f b take different values on these sets. The geometrical information G is interpreted as the 'approximated shape' of the data manifold, and we let ϕ concentrate around G . By multiplying f b and ϕ together, we obtain a classifier that captures the semantic and geometry information of the training data. In brief, our model assumes that both the source and target models come from a specified hypothesis class and the natural data is drawn from a low-dimensional manifold.

Despite its simple form, the manifold attack model provides us with powerful tools to theoretically analyze the properties of TBAs. More specifically, Theorem 4.8 proves that the off-manifold adversarial examples are transferable when the source model is inaccurate, which explains the empirical results from Papernot et al. [7]. By further discussing the existence of off-manifold adversarial examples, we theoretically explain why the success rates of TBAs are hard to improve. Moreover, Theorem 4.13 quantitatively discusses the relationship between the existence of transferable adversarial examples and the 'curvature' of the data manifold M , which formalizes the explanation in Dong et al. [6].

In addition to the explanatory results, we further discuss the expressive power and the possible extensions of our model in general applications. Section 5.1 proves that our model is extensible, i.e., we can replace the hypothesis class FM by the family of ReLU networks while proving similar results. Our model builds a bridge between the less developed theory behind TBAs and the huge amount of theoretical works analyzing ReLU networks. Furthermore, Section 5.2 demonstrate that the expressive power of our model is strong enough for the study of TBAs. We also provide a detailed discussion on the applicability of our model in Appendix A.

In summary, we propose a model that is theoretically tractable and consistent with existing results. It formalizes widespread beliefs and explains the properties of TBAs under a unified framework. The remainder of this paper is structured as follows. Sections 2 and 3 introduce the related works and terminologies, respectively. We propose our model and present the explanatory results in Section 4. Section 5 makes further discussions on our model. Section 6 summarizes this paper. Some remarks and the omitted proofs can be found in the appendix.

2 Related Works

The related works of TBAs and the low-dimensional manifold assumption are summarized as follows.

2.1 Transfer-Based Attacks

The research on the adversarial robustness of ML can be roughly divided into adversarial attacks [8, 9, 19], defenses [20-22], and the analyses of the robustness of existing methods [23-26]. Adversarial

attacks can be divided into two classes based on whether the attackers have the gradient information of the target model, i.e., white-box [27] and black-box [7] attacks. TBAs are one of the two main approaches to performing black-box attacks on ML models. Apart from TBAs, the other main approach is the optimization-based attack that approximates the gradient of the target model and performs white-box attacks thereafter [28-30]. While TBAs require much less information from the target model (and thus more practical) than optimize-based attacks, the success rates of TBAs are constantly lower, even when the source models are almost accurate [2, 3, 5, 6, 11].

There are a few theoretical works that have tried to explain the properties of TBAs. However, these works rely heavily on either simple models or strong assumptions. Apart from those mentioned in Section 1, the seminal work of Goodfellow et al. [9] tries to explain the transferability of adversarial examples using linear models. Charles et al. [31] theoretically prove, in the context of linear classifiers and two-layer ReLU networks, that transferable adversarial examples exist. However, [31] does not explain those properties of TBAs mentioned in Section 1. Gilmer et al. [32] assume that the natural data is drawn from a 'concentric spheres dataset', which can be viewed as a special form of the lowdimensional data manifold. In comparison to previous works, our model requires milder assumptions on both data distribution and hypothesis class, while providing a more detailed discussion of the properties of TBAs.

2.2 The Low-Dimensional Manifold Assumption

The low-dimensional manifold assumption is commonly seen in many areas of research, ranging from classic approximation theory [13, 14] to computer vision [15, 16]. Under this assumption, it is intuitive to divide the adversarial examples into two groups based on whether they lie on or off the data manifold. Both on-manifold [15, 33] and off-manifold [32, 34] adversarial examples have been studied in many previous works. In most cases, a classifier is simultaneously vulnerable to both onand off-manifold adversarial examples [33].

In this paper, we discuss the transferability of both on- and off-manifold adversarial examples. We are mainly motivated by the techniques from Zhang et al. [35] and Li et al. [14]. The main results of Zhang et al. [35] decompose the adversarial risk based on the position of the adversarial example and discuss the existence of different types of adversarial examples; Zhang et al. [35] provides us with useful tools to analyze the existence of transferable adversarial examples. Li et al. [14] consider robust generalization and approximate continuous classifiers by ReLU networks. In this paper, we prove that the hypothesis class FM in our model can also be approximated by ReLU networks.

3 Problem Setup

This section introduces the basic setups and terminologies. We use a black triangle sign ( ▲ ) to indicate the end of assumptions, definitions, or remarks.

3.1 Notations

Let R d be the real vector space equipped with the l p -norm, where p ∈ [1 , + ∞ ]. In this paper, we consider the k -class classification problems ( k ≥ 2) on R d . Denote the data-label pairs by ( x , y ), in which y is chosen from some label space Y . By convention, we let Y = {-1 , 1 } in binary classification and Y = { 1 , 2 , · · · , k } when k > 2. Unless otherwise specified, we assume that the natural data lie on a low-dimensional manifold M⊂ R d . More specifically, we adopt the following assumption.

Assumption 1 (Low-dimensional Manifold) . Let M⊂ [0 , 1] d be a compact smooth manifold 2 and assume that the dimension of M is less than d . Let D be some given continuous distribution that is supported on M . We say that a random variable x is natural data if x ∼ D . ▲

Denote the support set of D by supp( D ). With slight abuse of notation, let y also be a mapping from supp( D ) to Y that assigns each natural data x ∼ D with a true label y = y ( x ). Notice that only natural data is endowed with a true label. Since supp( D ) ⊂ [0 , 1] d , we will make no distinction between R d and [0 , 1] d in the remainder of this paper without loss of generality (WLOG).

Let dp be the metric induced by the l p -norm and B ( x ; r ) the l p -ball around x with radius r . For any subset S ⊂ R d , let dp ( x , S ) : = inf x ′ ∈ S dp ( x , x ′ ) be the distance between S and x . Let the classifiers f be functions defined on R d . When k > 2, we consider f ( x ) = ( f (1) ( x ) , f (2) ( x ) , · · · , f ( k ) ( x )) T that maps x ∈ R d to f ( x ) ∈ R k . For ease of notation, let y ( f , x ) : = arg max 1 ≤ i ≤ k f ( i ) ( x ) be the output class of f for ∀ x ∈ R d . Since D is continuous, y ( f , x ) take unique value w.p. (with probability) 1 when x ∼ D . In binary classification problems, we consider f : R d → R and y ( f , x ) = sign( f ( x )).

Given perturbation radius δ and natural data x ∼ D , we call x a ∈ B ( x ; δ ) an adversarial example of f at x if y ( f , x ) , y ( f , x a ). Given classifier f , let the standard risk of f be R std ( f ) : = P x ∼ D GLYPH<2> y ( f , x ) , y ( x ) GLYPH<3> , and the adversarial risk of f with regard to (w.r.t.) δ is defined as

Since TBA is often against accurate target models in practice, our paper assumes that the standard risk of the target model f t is R std ( f t ) = 0 for the sake of simplicity. Given such accurate f t , the goal of a TBA is to find adversarial examples of f t . Specifically, the attacker needs to obtain a source model f s and craft adversarial examples of f s using white-box attacks. Given natural data x ∼ D and an adversarial example x a ∈ B ( x ; δ ) of f s at x , we say that x a transfers to f t if y ( f t , x ) , y ( f t , x a ).

3.2 The On- and Off-Manifold Adversarial Examples

In this paper, the on- and off-manifold adversarial examples are defined as follows.

Definition 3.1 (On- and off-manifold adversarial examples) . For any given classifier f , perturbation radius δ and natural data x ∼ D , let x a ∈ B ( x ; δ ) be an adversarial example of f at x . We call x a an on-manifold adversarial example x a ∈ M , or an off-manifold adversarial example if x a < M . ▲

We say that a classifier f suffers from , or is vulnerable to (on-, off-manifold) adversarial examples if ∃ x ∼ D , x a ∈ B ( x ; δ ) such that x a is an (on-, off-manifold) adversarial examples of f at x . If f does not suffer from (on-, off-manifold) adversarial examples, then we consider it to be robust against (on-, off-manifold) adversarial examples.

In most cases, the adversarial examples of f t are regarded as imperceptible to humans (e.g., in a cator-dog classification task, the adversarial examples of cat images still look like cats), which implies that adversarial examples are not naturally generated data. Otherwise, the adversarial examples would be endowed with the same label as their corresponding natural data, which leads to a contradiction since R std ( f t ) = 0. In other words, adversarial examples should not lie in the support of D . To ensure this, we additionally assume that the natural data with different true labels are separated from each other. Notice that there is an intrinsic connection between the natural data x and its true label y ( x ), which is often referred to as 'semantic information' of the natural data. In this paper, we formalize the semantic information of the natural data as a family of sets { A j : j ∈ Y} , where

In practice, the semantic information A i and A j is often separated if i , j (e.g., Remark A.1). In this paper, we assume that the semantic information is separated in the following sense.

Assumption 2 (Separated Semantic Information) . Given λ > 0, for ∀ i , j ∈ Y , we assume that A i and A j are 2 λ -separated. Here, for any given subsets E , F ⊂ R d and r > 0, we say that E and F are r-separated if dp ( x 1 , x 2 ) ≥ r for ∀ x 1 ∈ E and x 2 ∈ F . ▲

With the help of Assumption 2, it is easy to check that the adversarial examples would not lie in the support set of D if δ < λ . In the rest of this paper, we will treat δ and λ as fixed parameters and always assume that δ < λ .

4 The Manifold Attack Model

Our proposed model specifies a hypothesis class that has two components. The first one is a semantic classifier that tries to capture the semantic information of the natural data. Similar to Equation (2), we call { A j f : j ∈ Y} the semantic information learned by a classifier f if y ( f , x ) = j for ∀ x ∈ A j f and ∀ j ∈ Y . To avoid trivial discussion, we assume that A j f , ∅ for ∀ j ∈ Y . The following definition specifies a family of classifiers that learn separated semantic information.

Definition 4.1 (Semantic classifier) . We call f b a semantic classifier if there is a family of pairwise 2 λ -separated set { A j f : j ∈ Y} such that { A j f : j ∈ Y} is the semantic information learned by f b . ▲

It is easy to check that R std ( f b ) = 0 if A j ⊂ A j f for ∀ j ∈ Y . Intuitively, the accuracy of a classifier depends on how well it learns the semantic information of the natural data.

The second component of our model is a function that captures the geometric structure of the data.

Definition 4.2 (Concentration multiplier) . Given G ⊂ R d , we call ϕ a concentration multiplier around G if ϕ ( x ) = 1 for ∀ x ∈ G , and ϕ ( x 1 ) < ϕ ( x 2 ) for ∀ x 1 , x 2 ∈ R d with dp ( x 1 , G ) > dp ( x 2 , G ). ▲

Intuitively, the geometric information G can be interpreted as the 'approximated shape' of the natural data. The concentration multipliers would assign a much lower confidence score to those data points that are outside of or far away from G . In the ideal case, G precisely captures the shape of M , which enables ϕ to detect the off-manifold data points (i.e., the out-of-distribution (OOD) data [36, 37]).

As mentioned in Section 1, ML models often capture both the semantic and geometrical information of the natural data. Therefore, we assume that the source and target models can be decomposed into the product of a semantic classifier and a concentration multiplier. More specifically, we consider the hypothesis class defined as follows.

Definition 4.3. Consider the k -class classification problems ( k ≥ 2) on [0 , 1] d . Let F b and Φ be a collection of semantic classifiers and concentration multipliers, respectively. When k > 2, let FM be the family of all classifiers f : [0 , 1] d → R k that satisfy

In the binary classification case, we let f ( x ) = f b ( x ) · ϕ ( x ) for ∀ x ∈ [0 , 1] d . Here, f b and ϕ are chosen from F b and Φ , respectively. Since the semantic information is contented in the manifold, we additionally assume that ∪ j ∈Y A j f ⊂ G , in which { A j f : j ∈ Y} is the semantic information learned by f b and G is the approximated shape of M learned by ϕ . ▲

Definitions 4.1 to 4.3 together with Assumptions 1 and 2 establish the abstract framework for the manifold attack model. This model formalizes how a classifier captures the semantic and geometric information from natural data. It is worth noticing that choices of f ∈ FM in Definition 4.3 remains largely arbitrary if we choose F b (and Φ ) to be the family of all possible semantic classifiers (and concentration multiplier) introduced in Definition 4.1 (and Definition 4.2). In fact, for any f that learns 2 λ -separated semantic information { A j f : j ∈ Y} and any ϕ such that ( f ( x ) = 0) = ⇒ ( ϕ ( x ) = 0) for ∀ x ∈ R d , we can simply let f ( j ) b ( x ) = f ( j ) ( x ) · ϕ -1 ( x ) for ∀ x , 0. It is easy to check that f b is a semantic classifier as long as A j ∩ G , ∅ for ∀ j ∈ Y .

f

The generality in Definition 4.3 comes at the expense of theoretical intractability. Take the semantic classifier for example. In Definition 4.1, f b cannot be determined by the semantic information it learns. Moreover, f b cannot even be parameterized according to this definition. There is an implicit tradeoff between generality and theoretical tractability: quantitative analysis of specific scenarios might need more delicate definitions. Next, we introduce one specific combination of f b and ϕ to carry on analyzing the properties of TBAs.

For the sake of simplicity, we only present analyses and results in the context of binary classification in Sections 4 and 5. For ease of notation, let A : = { x : y ( x ) = 1 } and B : = { x : y ( x ) = -1 } , i.e., the semantic information of binary natural data. The following proposition specifies a sub-family of semantic classifiers:

Proposition 4.4 (semantic classifier, binary case) . Given 2 λ -separated sets A f , Bf ⊂ M . Define:

Then, fb is a semantic classifier. In particular, we can obtain from Equation (4) that f b ( x ) > 0 if x is closer (w.r.t. dp) to A f than Bf and fb ( x ) < 0 otherwise.

Let F bin the collection of all semantic classifiers defined by Equation (4) for all possible 2 λ -separated Af , Bf ⊂ M . The following proposition shows that a semantic classifier can be accurate and robust.

r

δ

>

2λ

A

B

M

B

A

Af

Bf

M

δ

δ

>

2λ

Proposition 4.5. Take Af = A and Bf = B in Equation (4) and denote the corresponding classifier by f ∗ b . Then, for any given λ ≥ δ > 0 , we have R std ( f ∗ b ) = R adv ( f ∗ b , δ ) = 0 .

Similar to Proposition 4.4, we specify a distance-based family of concentration multipliers.

Proposition 4.6 (Concentration multiplier, binary case) . For any given r > 0 and G ⊂ R d , denote

Then ϕ ( x ) is a concentration multiplier around G.

Propositions 4.4 and 4.6 introduce a combination of f b and ϕ in binary case. Both f b and ϕ can be parameterized by the semantic and geometrical information it learns. It is also possible to extend our analysis to other scenarios by specifying other combinations of f b and ϕ , which is out of the scope of our paper. In the following sections, we adopt the following assumption.

Assumption 3. The source and target model are chosen from the hypothesis class given by Definition 4.3, where F b and Φ are the function classes introduced by Propositions 4.4 and 4.6. ▲

With slight abuse of notation, we denote the hypothesis class in Assumption 3 by FM . Intuitively, for any f = f b · ϕ ∈ FM , the semantic classifier controls the accuracy of f . The similarity between A , B and Af , Bf reflects how well f b fits the natural data. Moreover, the overlapping area ( A ∪ B ) ∩ ( Af ∪ Bf ) should contain the training data of f (and thus be non-empty) if f has reached zero training error. The concentration multiplier helps classification by assigning lower confidence scores on those x < G , but it might also bring in adversarial examples. Since we only impose mild conditions on the choice of G (i.e., Af ∪ Bf ⊂ G ), FM can capture complicated geometrical information. We will come back to the expressive power of FM at Section 5.2.

4.1 The Transferability of Adversarial Examples

As a warm-up, we study the robustness of f ∈ FM against on- and off-manifold adversarial examples without specifying f b and ϕ . The following proposition shows that the vulnerability of f ∈ FM to onand off-manifold adversarial examples are different.

Proposition 4.7. Let f = f b · ϕ . Given R adv ( f ; δ ) , 0 , we can obtain (w.p. 1 ) that

f suffers from off-manifold adversarial examples.

if f b captures the semantic information of the natural data (i.e., A ⊂ Af and B ⊂ Bf ), then f is robust against on-manifold adversarial examples.

Remark. The results in Proposition 4.7 hold w.p. 1 since our proof ignores some minor events (e.g., f ( x ) = 0) that occur w.p. 0. For the sake of simplicity, we will not specify those events that happen almost surely in the rest of this paper.

The first result of Proposition 4.7 implies that f ∈ FM suffers from adversarial examples if and only if f suffers from off-manifold adversarial examples. In Section 4.2, we will discuss the existence of transferable adversarial examples based on this result.

We explain the properties of TBAs by proving the existence of f ∈ FM that satisfies desired properties. First of all, we introduce a sub-family of FM that is robust against on-manifold adversarial examples. The following concentration multipliers force f b to concentrate around the data manifold, which brings off-manifold adversarial examples to f b without introducing on-manifold adversarial examples. For any unspecified α ∈ (0 , 1), let

Consider the classifiers induced by ϕ o ff , i.e., f = f b · ϕ o ff . It is clear from Equation (6) that f ( x ) = f b ( x ) when x ∈ M . Combining this with the separated assumptions of semantic information, we can see that ϕ o ff blocks the on-manifold adversarial examples. Figure 1 demonstrates that the target model f t = f ∗ b · ϕ o ff is only vulnerable to off-manifold perturbations. The following proposition shows that off-manifold adversarial examples are transferable even if the source model is inaccurate, which explains the phenomenon in Papernot et al. [7].

Theorem 4.8. Consider TBAs with perturbation radius δ ∈ (0 , λ ] , target model ft = f ∗ b · ϕ o ff and source model fs = f b · ϕ o ff , f b ∈ F b. Denote the semantic information of fb by Af and Bf . Then, for ∀ x ∈ A ∪ B, all adversarial examples (if exist) of f s at x are transferable if x ∈ Af ∪ Bf .

Theorem 4.8 proves that all of the adversarial examples of f s are transferable if the concentration multipliers of f t and f s are identical, given that ( A ∪ B ) ∩ ( Af ∪ Bf ) , ∅ . In particular, f s is not necessarily accurate, which is consistent with the empirical results in Papernot et al. [7]. Theorem 4.8 also implies that the vulnerability of ML models might be due to the non-robust geometrical information. Such vulnerability is transferable between models that learn similar geometrical information.

Next, we turn to the on-manifold adversarial examples. Proposition 4.7 shows that we cannot find an f ∈ FM such that f suffers from only on-manifold adversarial examples. Instead, we specify a family of concentration multipliers that blocks out off-manifold adversarial examples that is 'directly above' the manifold. In this case, we consider Euclidean space R d with l 2-norm and inner product ⟨ x 1 , x 2 ⟩ : = x T 1 x 2. For ∀ x ∈ M , let T x M⊂ R d be the tangent space of M at x , i.e., the space spanned by the possible tangent directions that pass through x . Now that we are considering an inner product space, let N x M⊂ R d be the normal space at x such that ∀ u ∈ N x M and v ∈ T x M , we have ⟨ u , v ⟩ = 0.

Denote N r ( S ) : = { x ′ ∈ R d : ∃ x ∈ S s . t . d 2 ( x , x ′ ) < r , x ′ -x ∈ N x M} for any given r > 0 and S ⊂ M . We call N r ( S ) a tubular neighborhood of S if for ∀ x ′ ∈ N r ( S ), there is an unique x ∈ S such that x ′ -x ∈ N x M . For any semantic classifier f b with semantic information Af and Bf , define

and consider f = f b · ϕ on ( · ; f b ). According to Equation (7), ϕ on blocks the off-manifold adversarial examples of f that is 'directly above' Af and Bf . Weuse Figure 1 to visualize our idea. The following proposition provides a sufficient condition for adversarial examples that are not transferable.

Proposition 4.9. Consider TBA with perturbation radius δ ∈ (0 , λ ] , target model f t = f ∗ b · ϕ on ( · , f ∗ b ) and source model fs = f b · ϕ on ( · ; f b ) , f b ∈ F b. Denote

Then, for ∀ x ∈ A ∪ B, we have

if B ( x , δ ) ∩ M ⊂ S crt ∪ S wrg , then ft and fs are both robust against adversarial examples;

if B ( x , δ ) ∩ M ⊂ A ∪ B and x ∈ Af ∪ Bf , then the adversarial examples of f s at x (if exists) cannot transfer to f t .

We interpret S crt (or S wrg) as the ' correct (or wrong) semantic information ' captured by f b . When S crt = supp( D ) and S wrg = ∅ (i.e., f b captures the semantic information of the natural data), the

first result of Proposition 4.9 implies that a large part of x ∈ supp( D ) is robust against adversarial examples if we block the off-manifold adversarial examples that are 'directly above' the semantic information. The second result of Proposition 4.9 implies that the potentially transferable adversarial examples are mostly located inside of the following set

or at least close to its boundary. Here, we interpret ( A ∪ B ) ∩ ( Af ∪ Bf ) c as the semantic information not contained in the training data.

In summary, the results in this section together provide a general view of the role that on- and offmanifold adversarial examples play in TBAs. Although both on- and off-manifold adversarial exist, ML models seem to be more vulnerable to off-manifold adversarial examples, and the off-manifold adversarial examples seem to play a more important role in TBAs.

4.2 The Existence of Transferable Adversarial Examples

Dong et al. [2] argue that the low success rates of TBAs are possibly due to the adversarial examples of f s falling into a 'non-adversarial region' of f t . In this section, we try to formalize this explanation and study the properties of such non-adversarial regions.

According to Proposition 4.7, f ∈ FM suffers from adversarial examples if and only if f suffers from off-manifold adversarial examples. This section is devoted to discussing the possible non-existence of off-manifold examples. Before we delve into this problem, let us introduce the notion of robust radius that is initially studied in the certified robustness problems [38, 39].

Definition 4.10 (Robust Radius, binary case) . For any classifier f , the robust radius of f is defined as the minimum r ≥ 0 such that for ∀ x ∈ A ∪ B and x 1 , x 2 ∈ B ( x , r ), we have f ( x 1 ) f ( x 2 ) ≥ 0.

Denote the robust radius of f by r δ ( f ). In our model, the robust radius of f ∈ FM is controlled by ϕ . Based on this notion, the following example demonstrates how the existence of off-manifold adversarial examples depends on the shape of the data manifold M .

Example 4.11. We use the same setting as Proposition 4.9 and consider the target model f t = f ∗ b · ϕ o ff . Given 0 < r 1 < r 2 < δ < r 3, denote ϕ 1 = ϕ o ff ( · ; r 1 , M ) and ϕ 2 = ϕ o ff ( · ; r 2 , M ). Using Equation (6), we obtain that r δ ( f ∗ b · ϕ 1 ) = r 1 < δ and r δ ( f ∗ b · ϕ 2 ) = r 2 < δ . However, r δ ( f ) < δ does not imply that all x ∈ A ∪ B are vulnerable to adversarial examples.

We demonstrated our idea in Figure A.1. Denote the dark blue surface in Figure A.1 by M r 2 : = { x : d 2 ( x , M ) = r 2 } . Observe that r 2 is of the same magnitude as the 'curvature' of M , we can find a x 0 ∈ A ∪ B such that d 2 ( x , M r 2 ) = r 3 > δ , i.e., x 0 is not vulnerable to adversarial examples. Note that all of such x 0 together form the non-adversarial region in Dong et al. [2]. ▲

Example 4.11 shows that the existence of the adversarial examples of f = f b · ϕ o ff depends on the 'curvature' of M , which seems to be a rather agnostic result. However, we can quantify the 'curvature' of M by the following lemma.

Lemma 4.12 (Bredon [40]) . Let M⊂ R d be a compact smooth manifold, then there is a ∆ > 0 such that N ∆ M is a tubular neighborhood of M .

Lemma 4.12 can be viewed as a variant of the tubular neighborhood theorem (cf. Bredon [40]). The constant ∆ is decided by M and can be used to evaluate the 'curvature' of M in Example 4.11. More specifically, the following proposition provides a sufficient condition for the existence of the off-manifold adversarial examples.

Theorem 4.13. Given perturbation radius δ ∈ (0 , λ ] and target model ft = f ∗ b · ϕ o ff ( · ; r , M ) . Let ∆ be the constant specified in Lemma 4.12. Then, for ∀ x ∈ A ∪ B, the off-manifold adversarial example of f t at x exists if r < ∆ .

Theorem 4.13 establishes a quantitative relationship between the existence of off-manifold examples and the 'curvature' of the manifold.

5 Discussion

5.1 Approximation with ReLU Networks

In applications, the target and source models of TBAs are mostly neural networks, e.g., ReLU networks. In this subsection, we use ReLU networks (cf. Definition 5.1) to approximate the classifiers f ∈ FM . Moreover, we recover some of the results in Section 4 based on the approximated classifier.

We start by introducing the family of ReLU networks. Denote the rectified linear unit (ReLU) by ρ ( x ) : = max { x , 0 } , where 0 is the zero vector in R d and max {· , ·} takes the entry-wise maximum of x and 0 . The family of deep ReLU networks defined as follows.

Definition 5.1 (Deep ReLU networks) . For any bias vector b = ( b 1 , · · · , bL ) and sequence of weight matrices W = ( W 1 , · · · , W L ), we call function ˜ f : R d → R a (deep) ReLU networks parametrized by b and W if

for ∀ x ∈ R d . Let Wi be a matrix with mi rows and ni columns. We have n 1 = d , mL = 1, and mi = ni + 1 (1 ≤ i ≤ L -1).

The non-constant entries in b and W are called the parameters of ˜ f . We say that a function is induced by ReLU networks if it can be expressed by the linear combination of finite many ReLU networks.

It has been proven [14, 41-43] that ReLU networks can approximate (w.r.t. the sup-norm) continuous functions with a bounded domain to any precision. Here, we denote the sup-norm of a function f : S ⊂ R d → R by ∥ f ∥∞ : = sup x ∈ S | f ( x ) | . Specifically, we have:

Lemma 5.2 (Li et al. [14]) . Given l-Lipschitz function f : [0 , 1] d → [ -1 , 1] and precision ϵ > 0 , there is a ReLU network ˜ f with O (( l /ϵ ) d ) · O ( d 2 + d log(1 /ϵ )) parameters that satisfies ∥ f -˜ f ∥∞ ≤ ϵ .

Lemma 5.2 provides a useful tool to approximate the Lipschitz functions from [0 , 1] d to [ -1 , 1]. It is straightforward to check the Lipschitzness of f ∈ FM . As a result, given ϵ > 0 and f ∈ FM , we can find an ReLU network ˜ f such that ∥ f -˜ f ∥∞ < ϵ . Unfortunately, ˜ f can no longer be decomposed into the product of f b and ϕ , which invalidates most of the results in Section 4. We have to make a detour to approximate f ∈ FM . As a first step, we approximate f b and ϕ by ˜ f b and ˜ ϕ , respectively.

Corollary 5.3. Let fb be a semantic classifier with semantic information A f and Bf that satisfy a 2 λ -separated property. Given ϵ > 0 , there is a ReLU network ˜ f b with O ((1 /λϵ ) d ) · O ( d 2 + d log(1 /ϵ )) parameters such that ∥ f b -˜ f b ∥∞ ≤ ϵ .

Corollary 5.4. Given ϵ > 0 , r > 0 and S ⊂ [0 , 1] d , there is a ReLU network ˜ ϕ with O ((1 / r ϵ ) d ) · O ( d 2 + d log(1 /ϵ )) parameters that can approximate ϕ ( · ; r , S ) to precision ϵ .

Corollaries 5.3 and 5.4 approximate f b and ϕ with ReLU networks. It remains to approximate the product of two ReLU networks using the following lemma.

Lemma 5.5 (Yarotsky [41]) . There is a function ˜ × : [ -1 , 1] 2 → [ -1 , 1] induced by ReLU network with O (log 2 ( ϵ -1 )) parameters such that ˜ × ( x , y ) = 0 if xy = 0 , and

The above lemma constructs a ReLU network that efficiently approximates the multiplying operator in R 2 . The following proposition shows that we can approximate f ∈ FM with ReLU networks to any precision.

Proposition 5.6. Given ϵ, λ, δ, r > 0 , for any f ∈ FM , there is a ReLU network ˜ f with

parameters that satisfies ∥ f -˜ f ∥∞ ≤ ϵ .

The following theorem partially recovers the results in Theorem 4.8 with ReLU networks.

Theorem 5.7. Consider TBAs with perturbation radius δ ∈ (0 , λ/ 2] , target model f t = f ∗ b · ϕ o ff and source model fs = f b · ϕ o ff , f b ∈ F b. Denote the semantic information of f b by Af and Bf . Given ϵ ≤ 0 . 1 , let ˜ ft and ˜ f s be ReLU networks that satisfy

˜

Notice that Equation (14) does not imply that the adversarial examples are transferable. Instead, it can only reduce the confidence in the decision made by ˜ f t . The results in Theorem 4.8 cannot be fully recovered. When f is close to zero, whether the approximated ReLU network ˜ f is greater or less than zero is hard to decide.

The following theorem recovers Proposition 4.9 after modifying some parameters.

Theorem 5.8. Consider TBAs with perturbation radius δ ∈ (0 , λ/ 2] , target model f t = f ∗ b · ϕ o ff and source model fs = f b · ϕ o ff , f b ∈ F b. Denote the semantic information of f b by Af and Bf . Given ϵ ≤ 0 . 1 , let ˜ ft and ˜ f s be ReLU networks that satisfy Equation (13) . Then, for ∀ x ∈ A ∪ B, we have

if B ( x , δ ) ∩ M ⊂ S crt ∪ S wrg , then ˜ ft and ˜ f s are both robust against adversarial examples;

if B ( x , δ ) ∩ M ⊂ A ∪ B and x ∈ Af ∪ Bf , then the adversarial examples of ˜ f s at x (if exists) cannot transfer to ˜ f t .

By approximating the classifiers in FM , we build a bridge between the less-developed theory behind TBAs and the huge amount of theoretical works analyzing ReLU networks.

5.2 The Expressiveness of the Manifold Attack Model

In learning theory, the expressive power of a hypothesis class F evaluates the performance of f ∈ F in fitting natural data; see [44] for a comprehensive overview. Previous works have tried to explain the transferability based on less expressive hypothesis classes, e.g., linear classifiers [32] and the linear combination of 'features' [17]. In this subsection, we study the expressive power of our model.

The main goal of TBAs is not to fit the natural data. Instead, for any well-trained classifier f ∗ : [0 , 1] d →{-1 , 1 } , i.e., R std ( f ∗ ) = 0, the goal of our model is to find f ∈ FM such that the adversarial examples of f fits those of f ∗ . More specifically, we have

Proposition 5.9. For any classifier f ∗ with R std ( f ∗ ) = 0 and perturbation radius δ ∈ ( r δ ( f ∗ ) , λ ) , there is f ∈ FM such that 1) R std ( f ) = R std ( f ∗ ) , and 2) for ∀ x ∈ A ∪ B, if x a is an adversarial example of f ∗ at x , then exists x ′ a ∈ B ( x a , r δ ( f ∗ ) / 4) such that x ′ a is an adversarial example of f .

The above proposition implies that our proposed model can cover a wide range of classifiers and is thus sufficient for the study of TBAs on manifolds.

6 Conclusion

This paper explains the properties of TBAs under a unified theoretical framework. We suggest that off-manifold adversarial examples play a major role in TBAs. In particular, we show that off-manifold adversarial examples are transferable when the source model is inaccurate. We also prove that the non-existence of off-manifold adversarial examples is one of the reasons why the success rates of TBAs are hard to improve.

Acknowledgments and Disclosure of Funding

This work is supported by the National Natural Science Foundation of China under Grant 61976161, and the Fundamental Research Funds for the Central Universities under Grant 2042022rc0016.

References

Nicolas Papernot, Patrick D. McDaniel, and Ian J. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. CoRR , abs/1605.07277, 2016.

Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. In CVPR , 2019.

Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L. Yuille. Improving transferability of adversarial examples with input diversity. In CVPR , 2019.

Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In CVPR , 2018.

Shuman Fang, Jie Li, Xianming Lin, and Rongrong Ji. Learning to learn transferable attack. In AAAI , 2022.

Yinpeng Dong, Shuyu Cheng, Tianyu Pang, Hang Su, and Jun Zhu. Query-efficient black-box adversarial attacks guided by a transfer-based prior. IEEE Trans. Pattern Anal. Mach. Intell. , 2022.

Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In AsiaCCS , 2017.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR , 2014.

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR , 2015.

Florian Tramèr, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, and Patrick D. McDaniel. The space of transferable adversarial examples. CoRR , abs/1704.03453, 2017.

Zhipeng Wei, Jingjing Chen, Micah Goldblum, Zuxuan Wu, Tom Goldstein, and Yu-Gang Jiang. Towards transferable adversarial attacks on vision transformers. In AAAI , 2022.

Zhichao Huang and Tong Zhang. Black-box adversarial attack with transferable model-based embedding. In ICLR , 2020.

Minshuo Chen, Haoming Jiang, Wenjing Liao, and Tuo Zhao. Efficient approximation of deep relu networks for functions on low dimensional manifolds. In NeurIPS , pages 8172-8182, 2019.

Binghui Li, Jikai Jin, Han Zhong, John E. Hopcroft, and Liwei Wang. Why robust generalization in deep learning is difficult: Perspective of expressive power. CoRR , abs/2205.13863, 2022.

Wei-An Lin, Chun Pong Lau, Alexander Levine, Rama Chellappa, and Soheil Feizi. Dual manifold adversarial robustness: Defense against lp and non-lp adversarial attacks. In NeurIPS , 2020.

Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In ICLR , 2021.

Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurIPS , 2019.

Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. In FOCS , 2021.

Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. Adversarial attacks and defences: A survey. CoRR , abs/1810.00069, 2018.

Boqi Li and Weiwei Liu. WAT: improve the worst-class robustness in adversarial training. In AAAI , 2023.

Xiyuan Li, Xin Zou, and Weiwei Liu. Defending against adversarial attacks via neural dynamic system. In NeurIPS , 2022.

Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In ICML , 2023.

Xinsong Ma, Zekai Wang, and Weiwei Liu. On the tradeoff between robustness and fairness. In NeurIPS , 2022.

Zhengyu Zhou and Weiwei Liu. Sample complexity for distributionally robust learning under chi-square divergence. Journal of Machine Learning Research , 2023.

Jingyuan Xu and Weiwei Liu. On robust multiclass learnability. In NeurIPS , 2022.

Xin Zou and Weiwei Liu. Generalization bounds for adversarial contrastive learning. Journal ofMachine Learning Research , 2023.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018.

Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. ZOO: zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In AISec , pages 15-26, 2017.

Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In ICML , 2018.

Shuyu Cheng, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Improving black-box adversarial attacks with a transfer-based prior. In NeurIPS , 2019.

Zachary Charles, Harrison Rosenberg, and Dimitris S. Papailiopoulos. A geometric perspective on the transferability of adversarial directions. In AISTATS , 2019.

Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian J. Goodfellow. Adversarial spheres. In ICLR , 2018.

David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generalization. In CVPR , 2019.

Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi N. R. Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E. Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR , 2018.

Wenjia Zhang, Yikai Zhang, Xiaoling Hu, Mayank Goswami, Chao Chen, and Dimitris N. Metaxas. A manifold view of adversarial risk. In AISTATS , 2022.

Sachin Vernekar, Ashish Gaurav, Vahdat Abdelzad, Taylor Denouden, Rick Salay, and Krzysztof Czarnecki. Out-of-distribution detection in classifiers via generation. CoRR , abs/1910.04241, 2019.

Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Label smoothed embedding hypothesis for out-of-distribution detection. CoRR , abs/2102.05131, 2021.

Linyi Li, Xiangyu Qi, Tao Xie, and Bo Li. Sok: Certified robustness for deep neural networks. CoRR , abs/2009.04131, 2020.

Zekai Wang and Weiwei Liu. Robustness verification for contrastive learning. In ICML , 2022.

Glen E. Bredon. Topology and Geometry . Springer, 1993.

Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks , 2017.

Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation characterized by number of neurons. CoRR , abs/1906.05497, 2019.

Zuowei Shen, Haizhao Yang, and Shijun Zhang. Optimal approximation rate of relu networks in terms of width and depth. CoRR , abs/2103.00502, 2021.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning - From Theory to Algorithms . Cambridge University Press, 2014.

Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Ruslan Salakhutdinov, and Kamalika Chaudhuri. A closer look at accuracy vs. robustness. In NeurIPS , 2020.

A Further Discussions

The following remark provides an example of the semantic information of a dataset.

Remark A.1 (The semantic information of CIFAR-10) . Generally speaking, "semantic" refers to the relationship between natural data and their true label, which should be consistent with human recognition. For example, the semantic information contained in the CIFAR-10 dataset is the true labels (e.g., airplane, automobile, and bird) and their corresponding natural images (e.g., images of airliners, SUVs, and chickens). In this example, an image cannot simultaneously include an airplane and an automobile since "the classes are completely mutually exclusive" in the CIFAR-10 dataset (cf. the official website of CIFAR-10.) ▲

It is also easy to check that the semantic information provided by CIFAR-10 is separated.

A.1 Visualization of the Non-Adversarial Region

We provide a visualization of Example 4.11 in Figure A.1.

r1

r2

M

A

r3

A.2 The Applicability of the Manifold Attack Model

Recall that Assumption 3 asserts that the classifiers f ∈ F can be decomposed into the product of a semantic classifier f b and a concentration multiplier ϕ . In this section, we empirically certify the following weaker version of Assumption 3.

Assumption 4 (Manifold attack model, weaker version) . Let f be the source or target model of a TBA. Let M 1 be a compact smooth manifold that is not necessarily a subset of M . Let N ∆ 1 M 1 be a tubular neighborhood of M 1 for some ∆ 1 > 0. We assume that for ∀ x ∈ M 1 and for any non-zero u ∈ N x M 1, we have f ( x + r 1 u ) > f ( x + r 2 u ) holds for ∀ δ 1 > r 2 > r 1 > 0.

By letting G = M 1 in Definition 4.2, the classifier in Assumption 3 would also satisfy Assumption 4. In this sense, we claim that Assumption 4 is a weaker version of Assumption 3. Next, we validate Assumption 4 on low-dimensional linearly separated data defined as follows. Given d > κ > 0,

consider the κ -dimensional manifold (in this case, a flat plane)

Assume that the natural data x ∈ M is drawn from some continuous distribution D ( x ), say a mixture of Gaussian distribution. For the sake of simplicity, let the natural data be linearly separable in the following sense. Given λ > 0 and natural data x = ( x 1 , · · · , xd ), we assume that | x 1 | > λ , and the label of x is decided by y ( x ) = 1 if x 1 > λ , and y ( x ) = -1 if x 1 < -λ .

In order to test Assumption 4, our goal is to train a classifier that satisfies this assumption. The training setting is stated as follows. Given N > 0, let the training set S N = { x 1 , x 2 , · · · , x N } ∼ D N ( x ) be independent and identically distributed (i.i.d.) random variables. For ∀ 1 ≤ n ≤ N , denote the label of x n by yN . Since D is continuous, we assume WLOG that

holds for ∀ 1 ≤ i , j ≤ N . The hypothesis class is defined as F w : = { f w : w ∈ R d } , where

for ∀ x ∈ R d . Consider the logistic loss function l ( u ) : = log(1 + exp( -u )) , ∀ u ∈ R , and the gradient descent (GD) iteration

We can show (both empirically and theoretically) that:

The GD iteration converges to an accurate classifier ˆ f ,

∃M 1 such that ˆ f satisfies Assumption 4,

which implies that we can actually train an accurate classifier (instead of constructing one using oracle information) that satisfies our assumption. The following section briefly introduces the setting and results of the experiment. The theoretical analysis is omitted.

A.2.1 Numerical Experiments

In order to visualize our results, the manifold M is set to be a 2-dimensional plane embedded in R 3 . The distribution D is a mixture of two Gaussian distributions with means (2 , 2 , 0) and ( -2 , -2 , 0), and covariance σ = 0 . 5. The optimizer is stochastic gradient descent (SGD). We train a two-layer network, denoted by ˆ f , that reaches 100% testing accuracy. We check whether ˆ f satisfies Assumption 4. The experimental results are demonstrated in Figure A.2.

Hardware Specification and Environment Our experiments are conducted on an Ubuntu 64-bit Linux workstation, having a 10-core Intel Xeon Silver CPU (2.20 GHz) and 4 Nvidia GeForce RTX 2080 Ti GPUs with 11GB graphics memory.

B Complete Proofs

Proposition 4.4 (semantic classifier, binary case) . Given 2 λ -separated sets A f , Bf ⊂ M . Define:

Then, fb is a semantic classifier. In particular, we can obtain from Equation (4) that f b ( x ) > 0 if x is closer (w.r.t. dp) to A f than Bf and fb ( x ) < 0 otherwise.

Proof of Proposition 4.4. It is easy to check that f b ( x ) = 1 when x ∈ Af and f b ( x ) = -1 when x ∈ Bf . By definition, we know that f b is a semantic classifier. □

Proposition 4.5. Take Af = A and Bf = B in Equation (4) and denote the corresponding classifier by f ∗ b . Then, for any given λ ≥ δ > 0 , we have R std ( f ∗ b ) = R adv ( f ∗ b , δ ) = 0 .

1

2

3

4

5

1

2

3

4

5

0.0

0.2

0.4

0.6

0.8

1.0

z=0

1

2

3

4

5

1

2

3

4

5

0.0

0.2

0.4

0.6

0.8

1.0

z=3

1

2

3

4

5

1

2

3

4

5

0.0

0.2

0.4

0.6

0.8

1.0

z=5

1

2

3

4

5

1

2

3

4

5

0.0

0.2

0.4

0.6

0.8

1.0

z=7

Proof of Proposition 4.5. By Equation (4), we have

Clearly, we have f ∗ b ( x ) = 1 when x ∈ A and f ∗ b ( x ) = -1 when x ∈ B . Then, the standard risk of f ∗ b w.r.t. D ( x ) is

Recall that A and B are 2 λ -separated (cf. Assumption 2). For ∀ x ∈ A and x ′ ∈ B ( x , δ ), we have dp ( x , B ) > δ , which implies that dp ( x ′ , B ) -dp ( x ′ , A ) > 0, and thus f ∗ b ( x ′ ) f ∗ b ( x ) = f ∗ b ( x ′ ) > 0. For ∀ x ∈ B , a similar deduction shows that f ∗ b ( x ′ ) f ∗ b ( x ) > 0 holds for ∀ x ′ ∈ B ( x , δ ). Together, we have

which completes the proof.

□

Remark B.1 . The construction of Equation (B.20) can be found in previous works [14, 45]. In particular, Li et al. [14] uses the ReLU-approximation of f ∗ b to study the robust generalization of deep neural networks.

Proposition 4.6 (Concentration multiplier, binary case) . For any given r > 0 and G ⊂ R d , denote

Then ϕ ( x ) is a concentration multiplier around G.

Proof of Proposition 4.6. For ∀ x ∈ G , we have dp ( x , G ) = 0. That is, ϕ ( x ) = 1 for ∀ x ∈ G . For ∀ x 1 , x 2 s.t. dp ( x 1 , G ) > dp ( x 2 , G ), it is easy to check that ϕ ( x 1 ) < ϕ ( x 2 ). □

Proposition 4.7. Let f = f b · ϕ . Given R adv ( f ; δ ) , 0 , we can obtain (w.p. 1 ) that

f suffers from off-manifold adversarial examples.

if f b captures the semantic information of the natural data (i.e., A ⊂ Af and B ⊂ Bf ), then f is robust against on-manifold adversarial examples.

Proof of Proposition 4.7. We first prove the first result. By definition, there are r > 0 and G ⊂ R d such that

Given R adv ( f ; δ ) , 0, there is natural data x ∈ supp( D ) and x 0 ∈ B ( x , δ ) such that f ( x ) f ( x 0 ) < 0. If x 0 ∈ M c , there is nothing to prove.

Otherwise, we have x 0 ∈ M . Denote

Consider the non-empty set

For ∀ x ′ 0 ∈ B ( x 0 , r 0 ), there is

It is also easy to check that

which implies that f ( x ) f ( x ′ 0 ) < 0 and x ′ 0 is an off-manifold adversarial example.

As for the second result, since A ⊂ Af , B ⊂ Bf , and Af ∪ Bf ⊂ G , we have

for ∀ x ∈ A ∪ B and x ′ ∈ B ( x , δ ) ∩ M . We can easily obtain that f b ( x ) = f b ( x ′ ), which implies that f has no on-manifold adversarial examples. □

Theorem 4.8. Consider TBAs with perturbation radius δ ∈ (0 , λ ] , target model ft = f ∗ b · ϕ o ff and source model fs = f b · ϕ o ff , f b ∈ F b. Denote the semantic information of fb by Af and Bf . Then, for ∀ x ∈ A ∪ B, all adversarial examples (if exist) of f s at x are transferable if x ∈ Af ∪ Bf .

Proof of Theorem 4.8. If f s is robust against adversarial examples, then there is nothing to prove. Otherwise, consider x ∈ Af ∩ A WLOG. By Equation (6), denote

We can find an adversarial example of f s at x . Denote this adversarial example by x a , and we have f s ( x a ) < 0. It is not hard to verify that dp ( x a , Bf ) -dp ( x a , Af ) > 0 since δ < λ , which implies that f b ( x a ) > 0. It is easy to obtain that f s ( x a ) < 0 and ϕ o ff ( x a ) < 0. We thus have αδ < dp ( x a , M ), which implies that x a is off the manifold and the distance between x a and M is greater than αδ . In particular, we have

which is independent of the choice of f b . Now consider f t ( x ) and f t ( x a ), where

No matter x ∈ A or x ∈ B , we have for ∀ x ′ ∈ B ( x , δ ), there is f ∗ b ( x ) = f ∗ b ( x ′ ) (by the 2 λ -separated property of A and B ). By Equation (B.30), we have

i.e., x a transfers to f t , which completes the proof.

□

Proposition 4.9. Consider TBA with perturbation radius δ ∈ (0 , λ ] , target model f t = f ∗ b · ϕ on ( · , f ∗ b ) and source model fs = f b · ϕ on ( · , f b ) , f b ∈ F b. Denote

Then, for ∀ x ∈ A ∪ B, we have

if B ( x , δ ) ∩ M ⊂ S crt ∪ S wrg , then ft and fs are both robust against adversarial examples;

if B ( x , δ ) ∩ M ⊂ A ∪ B and x ∈ Af ∪ Bf , then the adversarial examples of f s at x (if exists) cannot transfer to f t .

Proof of Proposition 4.9. For the first results, it is easy to check that ϕ on ( x ) = 1 and ϕ on ( x a ) = 1 for ∀ x a ∈ B ( x , δ ), which implies that f is robust against adversarial examples. It remains to prove the second result. By Equation (7), denote

For ∀ x ∈ A ∪ B such that B ( x , δ ) ∩ M ⊂ A ∪ B and x ∈ Af ∪ Bf , denote the unspecific adversarial example (if exist) of f s at x by x a . Assume that x ∈ A ∩ Bf WLOG. By definition, we have

By the 2 λ -separated assumption of Af and Bf , we have

From f s ( x ) f s ( x a ) < 0, we can obtain that ϕ on ( x ; f b ) ϕ on ( x a ; f b ) < 0. Since x ∈ Bf , we have

i.e., ϕ on ( x ; f b ) = 1. Combine this with ϕ on ( x ; f b ) ϕ on ( x a ; f b ) < 0, we have ϕ on ( x a ; f b ) < 0, i.e.,

It remains to show that f t ( x a ) > 0. By Equation (7), denote

By x ∈ A and the 2 λ -separated assumption of A and B , we have f ∗ b ( x a ) > 0. By B ( x , δ ) ∩ M ⊂ A , we have x a ∈ N δ ( A ∪ B )), i.e., ϕ on ( x a ; f ∗ b ) > 0. Together, we have f t ( x a ) = f ∗ b ( x a ) · ϕ on ( x a ; f ∗ b ) > 0. That is, x a is not an adversarial example of f t , which completes the proof. □

Theorem 4.13. Given perturbation radius δ ∈ (0 , λ ] and target model ft = f ∗ b · ϕ o ff ( · ; r , M ) . Let ∆ be the constant specified in Lemma 4.12. Then, for ∀ x ∈ A ∪ B, the off-manifold adversarial example of f t at x exists if r < ∆ .

Proof of Theorem 4.13. For ∀ x ∈ A ∪ B , let u ∈ N x ( M ) be the normal direction at x with ∥ u ∥ 2 = 1. Since r < ∆ , we can find r 0 > r such that r 0 < ∆ and r 0 < δ . Denote

Clearly, we have x a ∈ B ( x , δ ). Since N ∆ ( M ) is a tubular neighborhood of M , we have

which implies that x a is an off-manifold adversarial example of f t at x . Notice that Equation (B.41) not necessarily holds when r 0 > ∆ . □

Corollary 5.3. Let fb be a semantic classifier with semantic information A f and Bf that satisfy a 2 λ -separated property. Given ϵ > 0 , there is a ReLU network ˜ f with O ((1 /λϵ ) d ) · O ( d 2 + d log(1 /ϵ )) parameters such that ∥ f -˜ f ∥∞ ≤ ϵ .

Proof of Corollary 5.3. According to Lemma 5.2, our goal is to upper bound the Lipschitz constant l of f b . By definition, it suffices to upper bound the supremum of

We only need to consider three cases:

both of x 1 , x 2 ∈ Af ∪ Bf , or

both of x 1 , x 2 ∈ ( Af ∪ Bf ) c , and

either x 1 or x 2 is in Af ∪ Bf .

When x 1 , x 2 ∈ Af ∪ Bf , a trivial verification shows that s ≤ 1 λ . We now turn to the second case. By symmetry, let

By simplifying Equation (B.42), we can obtain that

which implies that s ≤ 1 λ in this case. Finally, we consider the third case. We assume WLOG that x 1 ∈ Af and x 2 ∈ ( Af ∪ Bf ) c . Substitute into Equation (B.42), we have

To sum up above, we have sup x 1 , x 2 s = 1 λ , which implies that f b is 1 λ -Lipschitz continuous, as is required. □

Corollary 5.3. Given ϵ > 0 , r > 0 and S ⊂ [0 , 1] d , there is a ReLU network ˜ ϕ with O ((1 / r ϵ ) d ) · O ( d 2 + d log(1 /ϵ )) parameters that can approximate ϕ ( · ; r , S ) to precision ϵ .

Proof of Corollary 5.3. We prove this corollary in a similar manner as Corollary 5.3, i.e., we upper bound the supremum of

We also consider three cases in this proof:

both of x 1 , x 2 ∈ S , or

both of x 1 , x 2 ∈ S c , and

either x 1 or x 2 is in S .

In case 1, we see at once that s = 0. When both of x 1 , x 2 ∈ S c , we assume WLOG that dp ( x 1 , S ) > dp ( x 2 , S ). By Equation (B.46), we have

Analysis similar to Equation (B.47) shows that

To sum up above, we have sup x 1 , x 2 s = 1 λ , which implies that ϕ is 1 r -Lipschitz continuous, as is required. □

Proposition 5.6. Given ϵ, λ, δ, r > 0 , for any f ∈ FM , there is a ReLU network ˜ f with

parameters that satisfies ∥ f -˜ f ∥∞ ≤ ϵ .

Proof of Proposition 5.6. This proposition can be derived directly from Lemma 5.5, Corollary 5.3, and Corollary 5.4. □

Theorem 5.7. Consider TBAs with perturbation radius δ ∈ (0 , λ/ 2] , target model f t = f ∗ b · ϕ o ff and source model fs = f b · ϕ o ff , f b ∈ F b. Denote the semantic information of f b by Af and Bf . Given ϵ ≤ 0 . 1 , let ˜ ft and ˜ f s be ReLU networks that satisfy

Then, for ∀ x ∈ ( A ∪ B ) ∩ ( Af ∪ Bf ) , the adversarial examples x a (if exist) of ˜ f s satisfies

Proof of Theorem 5.7. Consider x ∈ Af WLOG. If f s is robust against adversarial examples at x ∈ B ( x , δ ) ∩M⊂ Af , then there is nothing to prove. If not, denote the adversarial example of f s at x by x a . Since x ∈ Af ⊂ M , there is

and we thus have ˜ f s ( x a ) < 0. By x a ∈ B ( x ; δ ) and the assumption δ < λ 2 , we have

To obtain ˜ × ( ˜ f b , ˜ ϕ o ff )( x a ) < 0, there must be ˜ f b ( x a ) · ˜ ϕ o ff ( x a ) < ϵ , which implies that

Now consider ˜ f t ( x ) and ˜ f t ( x a ). By definition, we have ˜ f ∗ b ( x ) ∈ [1 -ϵ, 1 + ϵ ], ˜ ϕ o ff ( x ) ∈ [1 -ϵ, 1 + ϵ ], and thus

Similar to Equation (B.53), there is

Combining Equations (B.54) to (B.56) together, we have

□

as is required.

Theorem 5.8. Consider TBAs with perturbation radius δ ∈ (0 , λ/ 2] , target model f t = f ∗ b · ϕ o ff and source model fs = f b · ϕ o ff , f b ∈ F b. Denote the semantic information of f b by Af and Bf . Given ϵ ≤ 0 . 1 , let ˜ ft and ˜ f s be ReLU networks that satisfy Equation (13). Then, for ∀ x ∈ A ∪ B, we have

if B ( x , δ ) ∩ M ⊂ S crt ∪ S wrg , then ˜ ft and ˜ f s are both robust against adversarial examples;

if B ( x , δ ) ∩ M ⊂ A ∪ B and x ∈ Af ∪ Bf , then the adversarial examples of ˜ f s at x (if exists) cannot transfer to ˜ f t .

Proof of Theorem 5.8. The proof of the first result is also straightforward, which is omitted here. It remains to prove the second result. By Equation (7), denote

For ∀ x ∈ A ∪ B such that B ( x , δ ) ∩ M ⊂ A ∪ B and x ∈ Af ∪ Bf , denote the unspecific adversarial example (if exist) of f s at x by x a . Assume that x ∈ A ∩ Af WLOG. By definition, we have

By the 2 λ -separated assumption of Af and Bf , we have and

Since x ∈ Af ∪ Bf , we have x ∈ N δ ( Af ∪ Bf ) and

From ˜ f s ( x ) ˜ f s ( x a ) < 0, we can obtain that ˜ f s ( x a ) < 0, which implies that

By x ∈ A and the 2 λ -separated assumption of A and B , we have

By B ( x , δ ) ∩ M ⊂ A , we have x a ∈ N δ ( A ∪ B )), i.e.,

□

Together, we have which completes the proof.

Proposition 5.9. For any classifier f ∗ with R std ( f ∗ ) = 0 and perturbation radius δ ∈ ( r δ ( f ∗ ) , λ ) , there is f ∈ FM such that

R std ( f ) = R std ( f ∗ ) , and

for ∀ x ∈ A ∪ B, if x a is an adversarial example of f ∗ at x , then exists x ′ a ∈ B ( x a , r δ ( f ∗ ) / 4) such that x ′ a is an adversarial example of f .

which implies that which implies that

By definition, we have

Proof of Proposition 5.9. Define the following set

and let

By definition, A ∪ B ∈ G . Consider

Since A ∪ B ∈ G , we have R std ( f ) = R std ( f ∗ b ) = 0 = R std ( f ) ∗ . For ∀ x ∈ A ∪ B , if x a is an adversarial example of f ∗ at x , then

which implies that exists x ′ a ∈ B ( x a , r δ ( f ∗ ) / 4) such that x ′ a is an adversarial example of f . □

Peer reviews:
comments: SUMMARY:
This paper proposes an explanatory model for transfer-based attacks (TBAs). This model formalizes two popular intuitions and specify a hypothesis class $F_M$. By assuming that the source and target models come from $F_M$, this paper explains the properties of TBA. This paper also discusses some properties of the model itself.

STRENGTHS:
+ Technically solid paper with rigorous mathematical proofs.
+ The way that this paper formalizes semantic/geometric information is very interesting. The proposed model greatly simplifies the analysis while not making too much assumptions. This technique might be useful in other fields of research like transfer learning and domain adaptation.

WEAKNESSES:
+ The main results (Section 4) of this paper are not well-organized. It seems that the authors have put all related results in just one section, which would confuse the readers that are not familiar with this topic.

QUESTIONS:
+ Are there any similar definitions about the semantic/geometrical information in previous works?
+ Whether the manifold attack model can be used to derive probabilistic bound on the success rate of TBAs?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

comments: SUMMARY:
The paper focuses on a black-box adversarial attack method called the transfer-based attacks (TBA). The paper establishes a unified framework that explains the properties of TBAs, which has never been done by previous work. More specifically, the paper explains 
1. why adversarial examples are transferable when the source model is inaccurate
2. why TBA’s success rate is hard to improve.
The paper also discuss
1. how to approximate their model with ReLU networks
2. the expressive power of their model

STRENGTHS:
The paper provides theoretical explanation for the properties of TBA. Comparing to previous works in this area, the paper is based on milder assumptions, provides more delicate results, and can view some previous works as special cases. The paper also discusses the proposed model’s expressive power and approximates the model with ReLU networks.

WEAKNESSES:
1. The paper does not quantitatively discuss how $f_b$ and $\phi$ control the semantic and geometrical information of the classifier. The intuition behind these two functions are not clear enough.

QUESTIONS:
1. The authors claim (l. 26-26, l. 90-91) that previous works “rely heavily on either simple models or strong assumptions”, but the references seem to be out of date. The authors should include more related work in the past few (2-3) years.
2. It is not proper to claim that the proposed model “explains that properties of TBAs” (l. 39). It seems that this paper only explains those properties of TBA that is mentioned in l.26-35 and l. 82-89. If there are any other properties of TBA that cannot be explained by the proposed model?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new framework, manifold attack models, to explain the transferability of adversarial examples. The model divides adversarial examples into on-manifold examples and off-manifold examples, where the former one lies in the low dimensional manifold of natural data. The paper show that off-manifold adversarial examples are transferable when the source model is inaccurate. It also prove that the non-existence of off-manifold adversarial examples is one of the reasons why the success rates of TBAs are hard to improve.\n\nSTRENGTHS:\n1. The paper takes a large step towards an very important and interesting problem, why adversarial examples are transferable across different models. It conducts  in-depth theoretical analysis into the problem, which previous works did not perform.\n2. The theoretical analysis successfully explain find several practical properties of the adversarial examples: (1) an inaccurate source model can transfer to the target model with a non-negligible success rate; (2) the success rates of TBAs are constantly lower than other methods of black-box attacks.\n\nI think the paper makes great contribution towards understanding adversarial examples if the statements are correct. But I cannot judge their correctness based on the current version (see weakness).\n\nWEAKNESSES:\nThe paper is very hard to read. I suggest the authors to format their presentation of the work in the following aspects.\n\n1. Clearly states the assumption of each proposition and theorem. For example, I think one assumption of propositions in Section 4 is the specific forms of the classifiers, which are not clearly stated.\n2. Formally states their propositions and theorems using mathematical languages. For example, replace \"suffers from off-manifold adversarial examples\" in Proposition 4.6. with formal mathematical equations and explain the meaning of the equations after the theorem.\n3. Reduce the number of propositions and give one main theorem for one conclusion with one main theorem.\n\nIn addition, it is hard to evaluate the correctness of the these theorems without mathematical languages in current version.\n\nAnd I think the paper also make strong assumption that the classifier need to have certain forms, which limits the contribution of the paper.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nSee Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an explanatory model for the transferability phenomenon of adversarial examples. This model involves a specific hypothesis class that can be decomposed into a “semantic classifier” and a “geometric multiplier.” The authors then use a specific choice of these classifiers to explain two interesting observations in the literature that lacks any theoretical understanding. Finally, the paper also tries to establish a relationship between the proposed model to ReLU neural networks. \n\n---\n\nAfter reading the rebuttal and the other reviews, I decide to raise the score from 4 to 5.\n\nSTRENGTHS:\n1. This paper provides an easy-to-follow and intuitive model for explaining the transferability of adversarial examples. The decomposition into on- and off-manifold components makes a lot of sense here and is also fairly novel.\n2. The paper provides a lot of clear definitions and background which makes it self-contained. I really appreciate the efforts that go into making this paper comprehensive and easy to read.\n3. The problem of transferability is indeed not well-explored theoretically. This paper fills in the gap in my understanding fairly well by formalizing many concepts that have been floating around in the literature.\n\nWEAKNESSES:\nMy main concern about this work is the applicability of the model. The results depend on rather strong assumptions about a specific model, and I am not fully convinced that this model accurately describes a neural network. I list more detailed comments below:\n\n1. All of the results in the paper are based on a specific form of semantic classifier and concentration multiplier from Proposition 4.3 and 4.5, respectively. I wonder if all the results also hold for all combinations of semantic classifiers (Definition 4.1) and concentration multipliers (Definition 4.3).\n2. Proposition 4.7, 4.8, and Example 4.10 are shown for a specific choice of the concentration multiplier (i.e., $\\phi_{\\mathrm{on}}$ or $\\phi_{\\mathrm{off}}$). This further limits the applicability of these statements. Can the authors justify why these specific choices are reasonable?\n3. Proposition 4.6 also seems to hold generally. We can see off-manifold adversarial examples as an $\\epsilon$-expansion of the data manifold $\\mathcal M$ so if an adversarial example exists in $\\mathcal M$, then there must also exist its off-manifold version. Therefore, in my opinion, the major results of the paper start from Proposition 4.7 onward.\n4. Section 5.1 shows that the hypothesis class $\\mathcal F_{\\mathcal M}$ can be approximated with ReLU neural networks, but I believe a reverse statement would be much more powerful. The current result is perhaps not surprising given that it is well-known that a neural network is a universal function approximator. In other words, it is more important for the results in this paper to be relevant in practice if a neural network can be approximated by some $f \\in \\mathcal F_{\\mathcal M}$. \n5. I can see that Section 5.2 and Proposition 5.9 attempt to show the universality of the model which partially addresses my first concern. However, I would hope to see a stronger result showing that typical (or even toy-ish) neural networks can indeed be decomposed into the very specific form of semantic classifier and concentration multiplier used in the proofs. It would be powerful evidence for convincing the community that the proposed explanatory model is truly applicable to neural networks. My suggestion is to rely on an empirical experiment on synthetic toy datasets, which have well-defined on- and off-manifold subspaces.\n6. It will also be a big plus if the authors can use the developed model to predict some undiscovered empirical behavior of the transferability phenomenon.\n\n### Minor typo\n\n1. Missing parenthesis in Equation (7).\n\nI am happy to reconsider my rating if the authors can address my concerns during the rebuttal period.\n\nQUESTIONS:\n1. Is the following statement from L145-146 contradictory?\n    \n    > In most cases, the adversarial perturbations are regarded as imperceptible to humans, which implies that adversarial examples are **not** naturally generated data.\n    \n    I can potentially agree that adversarial examples are not in support of $D(\\mathbf x)$ because we don’t simply find adversarial examples by sampling from $D(\\mathbf x)$. However, the reasoning above does seem contradictory to me.\n    \n2. Can the authors justify the need to differentiate the support of $D(\\mathbf x)$ from the natural image manifold $\\mathcal M$? I am wondering if we can simply unify the two. Here’s my reasoning: based on the paper, my understanding is that $D(\\mathbf x)$ represents train/test data of the given classification task (e.g., cats vs dogs), and $\\mathcal M$ represents a space of all possible natural images. So the support of $D(\\mathbf x)$ is a subset of $\\mathcal M$ (please correct me if my understanding is incorrect). Coming back to the cats vs. dogs example, I can see that there is the “noisy” version of the adversarial examples which I’d regard as in **neither** $\\mathcal M$ nor the support of $D(\\mathbf x)$. Another type of adversarial example might be the **imperceptible** one which I see as something in **both** $\\mathcal M$ and support of $D(\\mathbf x)$. Now this makes me wonder what are the adversarial examples that are in $\\mathcal M$ but **not** on the support of $D(\\mathbf x)$? In other words, as far as I know, there are no adversarial examples that look like a snake (or any other objects) in the cats vs. dogs task, for example. So if there are no adversarial examples that fit this last category, can we just unify $\\mathcal M$ and the support of $D(\\mathbf x)$? What would be the implication in this case?\n3. An issue with the assumption on L222 that states\n    \n    > we let the semantic classifier of $f_t$ be the classifier $f^*_b$ in Proposition 4.4.\n    \n    I interpret this assumption as that the semantic classifier is perfect, and the adversarial examples simply exist because of the geometric multiplier. If I understand correctly, this also implies that $f_t$ does not have on-manifold adversarial examples which I think is not very accurate as the previous work has shown both on- and off-manifold adversarial examples on typical neural networks. Can the authors discuss more the implications of this assumption (e.g., Why is it reasonable? What would fail if we don’t assume this?)?\n\nLIMITATIONS:\nI do not see an explicitly listed limitation section. Please refer to the Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper provides a analysis of transfer-based attacks within a unified theoretical framework. The authors introduce the manifold attack model, which serves as an explanatory tool, formalizing commonly held beliefs and shedding light on previously observed empirical results. Through their model, they elucidate the phenomenon of transferability in adversarial examples, even when the source model is inaccurate, by highlighting the role of the data manifold's curvature.

STRENGTHS:
The paper addresses the crucial problem of transfer-based attacks, which are a significant threat to the security and robustness of machine learning models.

WEAKNESSES:
One major concern with this paper is the lack of sufficient quantitative analyses. While the proposed manifold attack model offers a theoretical framework for understanding transfer-based attacks, it would greatly benefit from concrete quantitative analyses to support its claims. One suggestion is to fit the model using low-dimensional data and perform a thorough verification. This would not only provide empirical evidence for the effectiveness of the proposed model but also enhance the paper's credibility by demonstrating its applicability in practical scenarios.

QUESTIONS:
None

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12