DOES ADVERSARIAL TRANSFERABILITY INDICATE KNOWLEDGE TRANSFERABILITY?
Year: 2020

Abstract:
Despite the immense success that deep neural networks (DNNs) have achieved, adversarial examples, which are perturbed inputs that aim to mislead DNNs to make mistakes, have recently led to great concerns. On the other hand, adversarial examples exhibit interesting phenomena, such as adversarial transferability. DNNs also exhibit knowledge transfer, which is critical to improving learning efficiency and learning in domains that lack high-quality training data. To uncover the fundamental connections between these phenomena, we investigate and give an affirmative answer to the question: does adversarial transferability indicate knowledge transferability? We theoretically analyze the relationship between adversarial transferability and knowledge transferability, and outline easily checkable sufficient conditions that identify when adversarial transferability indicates knowledge transferability. In particular, we show that composition with an affine function is sufficient to reduce the difference between the two models when they possess high adversarial transferability. Furthermore, we provide empirical evaluation for different transfer learning scenarios on diverse datasets, showing a strong positive correlation between the adversarial transferability and knowledge transferability, thus illustrating that our theoretical insights are predictive of practice.

text: Despite the immense success that deep neural networks (DNNs) have achieved, adversarial examples, which are perturbed inputs that aim to mislead DNNs to make mistakes, have recently led to great concerns. On the other hand, adversarial examples exhibit interesting phenomena, such as adversarial transferability. DNNs also exhibit knowledge transfer, which is critical to improving learning efficiency and learning in domains that lack high-quality training data. To uncover the fundamental connections between these phenomena, we investigate and give an affirmative answer to the question: does adversarial transferability indicate knowledge transferability? We theoretically analyze the relationship between adversarial transferability and knowledge transferability, and outline easily checkable sufficient conditions that identify when adversarial transferability indicates knowledge transferability. In particular, we show that composition with an affine function is sufficient to reduce the difference between the two models when they possess high adversarial transferability. Furthermore, we provide empirical evaluation for different transfer learning scenarios on diverse datasets, showing a strong positive correlation between the adversarial transferability and knowledge transferability, thus illustrating that our theoretical insights are predictive of practice.

heading: 1 INTRODUCTION
text: Knowledge transferability and adversarial transferability are two fundamental properties when a learned model transfers to other domains. Knowledge transferability, also known as learning transferability, has attracted extensive studies in machine learning. Long before it was formally defined, the computer vision community has exploited it to perform important visual manipulations (Johnson et al., 2016), such as style transfer and super-resolution, where pretrained VGG networks (Simonyan & Zisserman, 2014) are utilized to encode images into semantically meaningful features. After the release of ImageNet (Russakovsky et al., 2015), pretrained ImageNet models (e.g., on TensorFlow Hub or PyTorch-Hub) has quickly become the default option for the transfer source, because of its broad coverage of visual concepts and compatibility with various visual tasks (Huh et al., 2016). Adversarial transferability, on the other hand, is a phenomenon that adversarial examples can not only attack the model they are generated against, but also affect other models (Goodfellow et al., 2014; Papernot et al., 2016). Thus, adversarial transferability is extensively exploited to inspire black-box attacks (Ilyas et al., 2018; Liu et al., 2016). Many theoretical analyses have been conducted to establish sufficient conditions of adversarial transferability (Demontis et al., 2019; Ma et al., 2018).
Knowledge transferability and adversarial transferability both reveal some nature of machine learning models and the corresponding data distributions. Particularly, the relation between these two phenomena interests us the most. We begin by showing that adversarial transferability can indicate knowledge transferability. This tie can potentially provide a similarity measure between data distributions, an identifier of important features focused by a complex model, and an affinity map between complicated tasks. Thus, we believe our results have further implications in model interpretability and verification, fairness, robust and efficient transfer learning, and etc.
To the best of our knowledge, this is the first work studying the fundamental relationship between adversarial transferability and knowledge transferability both theoretically and empirically. Our main contributions are as follows.
• We formally define two quantities, τ1 and τ2, to measure adversarial transferability from different aspects, which enables in-depth understanding of adversarial transferability from a geometric point of view in the feature representation space. • We derive an upper bound for knowledge transferability with respect to adversarial transfer-
ability. We rigorously depict their underlying relation and show that adversarial transferability can indicate knowledge transferability. • We conduct thorough controlled experiments for diverse knowledge transfer scenarios (e.g.
knowledge transfer among data distributions, attributes, and tasks) on benchmark datasets including STL-10, CIFAR-10, CelebA, Taskonomy-data, and four language datasets. Our empirical results show strong positive correlation between adversarial and knowledge transferability, which validates our theoretical prediction.

heading: 2 RELATED WORK
text: Knowledge transferability has been widely applied in scenarios where the available data for certain domain is limited, and has achieved great success (Van Opbroek et al., 2014; Wurm et al., 2019; Wang et al., 2017; Kim & Park, 2017; Maqueda et al., 2018; Devlin et al., 2018). Several studies have been conducted to understand the factors that affect knowledge transferability (Yosinski et al., 2014; Long et al., 2015b; Wang et al., 2019; Xu et al., 2019; Shinya et al., 2019). Empirical observations show that the correlation between learning tasks (Achille et al., 2019; Zamir et al., 2018), the similarity of model architectures, and data distribution are all correlated with different knowledge transfer effects.
Adversarial Transferability has been observed by several works (Papernot et al., 2016; Goodfellow et al., 2014; Joon Oh et al., 2017). Since the early work, a lot of studies have been conducted, aiming to further understand the phenomenon and design more transferable adversarial attacks. Regardless of the threat model, a lot of attack methods have been proposed to boost adversarial transferability (Zhou et al., 2018; Demontis et al., 2019; Dong et al., 2019; Xie et al., 2019). Naseer et al. (2019) propose to produce adversarial examples that transfer cross-domain via a generative adversarial network. In addition to the efficacy, efficiency (Ilyas et al., 2018) and practicality (Papernot et al., 2017) are also optimized. Beyond the above empirical studies, there is some work dedicated to analyzing this phenomenon, showing different conditions that may enhance adversarial transferability (Athalye et al., 2018; Tramèr et al., 2017; Ma et al., 2018; Demontis et al., 2019). Building upon these observations, it is clear that there exist certain connections between adversarial transferability and other knowledge transfer scenarios, and here we aim to provide the first theoretic justification to verify it and design systematic empirical studies to measure such correlation.

heading: 3 ADVERSARIAL TRANSFERABILITY VS. KNOWLEDGE TRANSFERABILITY
text: In this section, we establish connections between adversarial examples and knowledge transferability rigorously. We first formally state the problem studied in this section. Then, we move on to subsection 3.1 to introduce two metrics that encode information about adversarial attacks. Finally, we present our theoretical results about the relationship between adversarial and knowledge transferability in subsection 3.2.
Notations. We use blackboard bold to denote sets, e.g., R. We use calligraphy to denote distributions, e.g., D. The support of a distribution D is denoted as supp(D). We use bold lower case letters to denote vectors, e.g., x ∈ Rn. We use bold uppercase letter to denote a matrix, e.g.,A. We useA† to denote the Moore–Penrose inverse of matrix A. We use ◦ to denote the composition of functions, i.e., g ◦ f(x) = g(f(x)). We use ‖ · ‖2 to denote Euclidean norm induced by standard inner product 〈·, ·〉. Given a function f , we use f(x) to denote its evaluated value at x, and we use f to represent this function in function space. We use 〈·, ·〉D to denote inner product induced by distribution D, i.e., 〈f1, f2〉D = Ex∼D〈f1(x), f2(x)〉. Accordingly, we use ‖ · ‖D to denote a norm induced by inner product 〈·, ·〉D, i.e., ‖f‖D = √ 〈f, f〉D. For a matrix function F : supp(D) → Rd×m, we
define its L2(D)-norm in accordance with matrix 2-norm as ‖F‖D,2 = √ Ex∼D‖F (x)‖22. We define projection operator proj(·, r) to project a matrix to a hyperball of spectral norm radius r, i.e.,
proj(A, r) = { A, if ‖A‖2 ≤ r rA/‖A‖2 if ‖A‖2 > r .
Setting. Assume we are given a target problem defined by data distribution x ∼ D, where x ∈ Rn, and y : Rn → Rd represent the ground truth labeling function. As a first try, a reference model fT : Rn → Rd trained on the target dataset is obtained through optimizing over a function class fT ∈ FT . Now suppose we have a source model fS : Rn → Rm pretrained on source data, and we are curious how would fS transfer to the target data D? Knowledge transferability. Given a trainable function g : Rm → Rd, where g ∈ G is from a small function class for efficiency purpose, we care about whether fS can achieve low loss L(·; y,D), e.g., mean squared error, after stacking with a trainable function g comparing with fT , i.e.,
min g∈G
L(g ◦ fS ; y,D) compare with L(fT ; y,D).
Clearly, the solution to this optimization problem depends on the choice of G. Observing that in practice it is common to stack and fine-tune a linear layer given a pretrained feature extractor, we consider the class of affine functions. Formally, the problem that is studied in our theory is stated as follows. Problem 1. Given a reference model fT trained on target distribution D, and a source model fS pre-trained on source data. Can we predict the best possible performance of the composite function g◦fS onD, where g is from a bounded affine function class, given adversarial transferability between fS and fT ?

heading: 3.1 ADVERSARIAL TRANSFERABILITY
text: We use the `2-norm to characterize the effectiveness of an attack. Definition 1 (Virtual Adversarial Attack (Miyato et al., 2018)). Given a model f : Rn → Rd, the attack on point x within -ball is defined as argmax‖δ‖≤ ‖f(x)− f(x+ δ)‖2. As this is intractable in practice, we consider the use of the tangent function to approximate the difference:
δf, (x) = arg max ‖δ‖≤
‖∇f(x)>δ‖2,
where ∇f(x) ∈ Rn×d is the Jacobian matrix. The will be dropped in clear context or when it is irrelevant.
To provide a quantitative view of adversarial transferability, we define two metrics τ1 and τ2. Both the metrics are in the range of [0, 1], where higher values indicate more adversarial transferability. Definition 2 (Adversarial Transferability (Angle)). Given two function f1, f2, we assume they have the same input dimension, and may have different output dimensions. The Adversarial Transferability (Angle) of f1 and f2 at point x is defined as the squared cosine value of the angle between the two attacks, i.e.,
τ1(x) = 〈δf1(x), δf2(x)〉2
‖δf1(x)‖22 · ‖δf2(x)‖22 .
We denote its expected value as τ1 = Ex∼D[τ1(x)].
Intuitively, τ1 characterizes the similarity of the two attacks. The higher the cosine similarity, the better they can be attacked together. Noting that we are suggesting to use the square of their cosine values, which means that cosine value being either 1 or−1 has the same indication of high knowledge transferability. This is because fine-tuning the last layer can rectify such difference by changing the sign of the last linear layer. However, it is not sufficient to fully characterize how good fS will perform only knowing the angle of two attack directions. For example, it is not difficult to construct two functions with highest τ1 = 1, but not transferable with affine functions. Moreover, it is also oberserved in our experiments that only τ1 is not sufficient.
Therefore, in addition to the information of attacks δf captured by τ1, we also need information about deviation of a function given attacks. We denote the deviation of a function f , given attack δ(x), as f(x+ δ(x))− f(x), and we define its approximation as
∆f,δ(x) = ∇f(x)>δ(x). (1) Accordingly, we define another metric to answer the following question: applying f1’s adversarial attacks on both the models, how much can the deviation of their function value be aligned by affine transformations?
Definition 3 (Adversarial Transferability (Deviation)). Given two functions f1, f2 with the same input dimensions and potentially different output dimensions, the Adversarial Transferability (Deviation) of adversarial attacks from f1 to f2 given data distribution D is defined as
τf1→f22 = 〈2∆f2,δf1 −A∆f1,δf1 ,A∆f1,δf1 〉D
‖∆f2,δf1‖ 2 D
,
whereA is a constant matrix defined as
A = proj(Ex∼D[∆f2,δf1 (x)∆f1,δf1 (x) >] ( Ex∼D[∆f1,δf1 (x)∆f1,δf1 (x) >] )† , ‖∆f2,δf1‖D ‖∆f1,δf1‖D ).
We note that A is the best linear map trying to align the two deviations (∆f2,δf1 and ∆f1,δf1 ) in the function space. It serves as a guess on the best linear map to align f1 and f2, using only the information from adversarial attacks. To have better sense of τ2 and the relationships with other quantities, we present an example for visual illustration in Figure 1. Note that high τ2 does not necessarily require ∆f1,δf1 and ∆f2,δf1 to be similar, but they can be well aligned by the constant linear transformationA. We refer to the proof of Proposition 1 at section B in appendix for detailed explanation of τ2. Proposition 1. Both τ1 and τ2 are in [0, 1].

heading: 3.2 ADVERSARIAL TRANSFERABILITY INDICATES KNOWLEDGE TRANSFERABILITY
text: In this subsection, we will provide our theoretical results. First, to have a better intuition, we will show a special case where the theorems are simplified, i.e., where fS and fT are both Rn → R. Then, we present the general case where fS and fT are multi-dimensional. Note that their output dimensions are not necessarily the same.
When fS and fT are both Rn → R, the τ1 and τ2 come out in a surprisingly elegant form. Let
us show what the two metrics are to have further intuition on what τ1 and τ2 characterize.
First, let us see what the attack is in this case. As function f has one-dimensional output, its gradient is a vector∇f ∈ Rn. Thus,
δf, (x) = arg max ‖δ‖≤ ‖∇f(x)>δ‖2 = ∇f(x) ‖∇f(x)‖2
is simply the gradient with its scale normalized. Then, the τ1 becomes
τ1(x) = 〈∇fS(x),∇fT (x)〉2
‖∇fS(x)‖22 · ‖∇fT (x)‖22 ,
which is the squared cosine (angle) between two gradients. For τ2, the matrix A degenerates to a scalar constant, which makes τ2 simpler as well, i.e.,
A = 〈∆fT ,δfS ,∆fS ,δfS 〉D ‖∆fS ,δfS ‖ 2 D , and τfS→fT2 = 〈∆fS ,δfS ,∆fT ,δfS 〉 2 D ‖∆fS ,δfS ‖ 2 D · ‖∆fT ,δfS ‖ 2 D .
We can see, in this case τ2 is interestingly in the same form of the first metric τ1. We will simply use τ2 to denote τ fS→fT 2 afterwards.
Accordingly, when fS and fT are both Rn → R, the result also comes out in an elegant form. In this case, adversarial attacks reflect all the information of the gradients of the two models, enabling τ1 and τ2 to encode all the information we need to prove the following theorem. Theorem 1. For two functions fS and fT that both are Rn → R, there is an affine function g : R→ R, such that
‖∇fT −∇(g ◦ fS)‖2D = Ex∼D [ (1− τ1(x)τ2)‖∇fT (x)‖22 ] ,
where g(x) = Ax + Const. Moreover, though not necessarily, if assuming that fT is L-Lipschitz continuous, i.e.,‖∇fT (x)‖2 ≤ L for ∀x ∈ supp(D), we have a more elegant statement:
‖∇fT −∇(g ◦ fS)‖2D ≤ (1− τ1τ2)L2.
The theorem suggests that, if adversarial transferability is high, there exists an affine transformation with bounded norm, such that g ◦ fS is close to fT . As an intuition of the proof, the difference between two gradients can be represented by the angle between them, which can be characterized by τ1; and the norm difference between them, which can be characterized by τ2.
As for the general case, we consider when the output dimensions of both functions are multidimensional and not necessarily the same. In this scenario, adversarial attacks correspond to the largest singular value of the Jacobian matrix. Therefore, we need to introduce the following definition to capture other information that is not revealed by adversarial attacks.
Definition 4 (Singular Value Ratio). For any function f , the Singular Value Ratio for the function gradient at x is defined as λf (x) = σ2(x)σ1(x) , where σ1(x), σ2(x) are the largest and the second largest singular value in absolute value of∇f(x), respectively. In addition, we define the worst-case singular value ratio as λf = maxx∈supp(D) λf (x).
Theorem 2. For two functions fS : Rn → Rm, and fT : Rn → Rd, assuming that fT is L-Lipschitz continuous, i.e., ‖∇fT (x)‖2 ≤ L for ∀x ∈ supp(D), there is an affine function g : Rm → Rd, such that
‖∇fT −∇(g ◦ fS)‖2D ≤ ( (1− τ1τ2) + (1− τ1)(1− τ2)λ2fT + (λfT + λfS ) 2 ) 5L2,
where g is defined as g(z) = Az +Const.
We note that this theorem also has a statement offering tighter bound where we do not assume Lipschitz continuous. The full version of this theorem is provided in appendix. Theorem 2 suggests that big τ1 and τ2 indicate potentially small differences of gradients between the target model and the transferred model. Based on this, intuitively, given the right constant value shift, minimal difference in gradients implies minimal difference in function value, which should result in bounded loss. Indeed, we prove in Theorem 3 that the squared loss of the transferred model g ◦ fS is bounded by the loss of fT and their gradient difference, by assuming the β-smoothness of both the functions.
Definition 5 (β-smoothness). A function f is β-smooth if for all x,y,
‖∇f(x)−∇f(y)‖2 ≤ β‖x− y‖2.
For the target data distribution D, and its ground truth labeling function y, the mean squared loss of the transferred model is Ex∼D‖g ◦ fS(x)− y(x)‖22 = ‖g ◦ fS − y‖2D. Therefore, the following theorem presents upper bound on the mean squared loss of the transferred model.
Theorem 3. Without loss of generality we assume ‖x‖2 ≤ 1 for ∀x ∈ supp(D). Consider functions fS : Rn → Rm, fT : Rn → Rd, and an affine function g : Rm → Rd, suggested by Theorem 1 or Theorem 2, with the constant set to let g(fS(0)) = fT (0). If both fT , fS are β-smooth, then
‖g ◦ fS − y‖2D ≤ ( ‖fT − y‖D + ‖∇fT −∇g ◦ fS‖D + ( 1 + ‖∇fT ‖D,2 ‖∇fS‖D,2 ) β )2 .

heading: 3.3 PRACTICAL MEASUREMENT OF ADVERSARIAL TRANSFERABILITY
text: Existing studies have shown that similar models share high adversarial transferability (Liu et al., 2016; Papernot et al., 2016; Tramèr et al., 2017). In previous work, it is common to use cross adversarial loss as an indication of adversarial transferability, e.g., the loss of fT with attacks generated on fS . It is intuitive to consider that the higher cross adversarial loss, the higher adversarial transferability. However, it may have a drawback comparing to the τ1, τ2 defined in this work.
Definition 6 (Cross Adversarial Loss). Given a loss function `T (·, y) on the target domain, where y is ground truth, the adversarial loss of fT with attack δfS generated against source model fS is
Ladv(fT , δfS ; y,D) = Ex∼D `T (fT (x+ δfS (x)), y(x)).
The cross adversarial loss depends on the choice of loss function, the output dimension, etc. Thus, it can be incomparable when we want to test adversarial transferability among different fT , unlike that τ1, τ2 are always between [0, 1]. To investigate the relationship between the adversarial loss and the adversarial transferability we defined, we show in the following proposition that the cross adversarial loss is similar to τ1. In the next section, we verify the theoretical predictions through thorough experiments.
Proposition 2. If `T is mean squared loss and fT achieves zero loss on D, then the adversarial loss defined in Definition 6 is approximately upper and lower bounded by
Ladv(fT , δfS ; y,D) ≥ 2Ex∼D [ τ1(x) ‖∇fT (x)‖22 ] +O( 3),
Ladv(fT , δfS ; y,D) ≤ 2Ex∼D [( λ2fT + (1− λ 2 fT )τ1(x) ) ‖∇fT (x)‖22 ] +O( 3),
where O( 3) denotes a cubic error term.

heading: 4 EXPERIMENTAL EVALUATION
text: The empirical evaluation of the relationship between adversarial transferability and knowledge transferability is done by four different sets of experiment. First we present a set of synthetic experiment that verifies our theoretical study, and then we present our empirical study on realworld datasets with models widely used in practice, described in three knowledge transfer scenarios: knowledge transfer on data distributions, attributes, and tasks. Details regarding the three scenarios are elaborated below, and all training details are deferred to the Appendix.
Knowledge-transfer among data distributions is the most common setting of transfer learning. It transfers the knowledge of a model trained/gained from one data domain to the other data domains. For instance, Shie et al. (2015) manage to use pre-trained ImageNet representations to achieve state-of-the-art accuracy for medical data analysis. The relation between adversarial and knowledge transferability can not only determine the best pretrained models to use, but also detect distribution shifts, which is crucial in learning agents deployed in continual setting (Diethe et al., 2019). Knowledge-transfer among attributes is a popular method to handle zero-shot and few-shot learning (Jayaraman & Grauman, 2014; Romera-Paredes & Torr, 2015). It transfers the knowledge learned from the attributes of the source problem to a new target problem Russakovsky & Fei-Fei (2010). The relation between adversarial and knowledge transferability can be used as a probe to deployed classification models to verify attributes that their decisions are based on. This will have profound implications on fairness and interpretability. Knowledge-transfer among tasks is widely applied across various vision tasks, such as super resolution (Johnson et al., 2016), style transfer (Gatys et al., 2016), semantic and instance segmentation (Girshick, 2015; He et al., 2017; Long et al., 2015a). It involves transferring the knowledge the model gains by learning to do one task to another novel task. The relation between adversarial and knowledge transferability, as many recent works (Achille et al., 2019; Standley et al., 2019; Zamir et al., 2018), can be used to charting the affinity map between tasks, aiming to guide potential transfer.

heading: 4.1 SYNTHETIC EXPERIMENT ON RADIAL BASIS FUNCTIONS REGRESSION
text: In the synthetic experiment, we compute quantities that are otherwise inefficient to compute to verify our theoretical results. We also try different settings to see how other factors affect the results. Details follow.
Models. Both the source model fS and the target model fT are one-hidden-layer neural networks with sigmoid activation.
Overall Steps. First, sample D = {(xi,yi)}Ni=1 from a distribution (details later), where x is n-dimensional, y is d-dimensional, and there are N samples. Then we train a target model fT on D. Denoting the weights of fT asW , we randomly sample a direction V where each entry of V is sampled from U(−0.5, 0.5), and choose a scale t ∈ [0, 1]. To derive the source model, we perturb the target model as W ′ := W + tV . Define the source model fS to be a one-hidden-layer neural network with weightsW ′. Then, we compute each of the quantities we care about, including τ1, τ2, cross adversarial loss (Definition 6), the upper bound in theorem 2 on the difference of gradients, etc.
Noting that we reported the cross adversarial loss normalized by its own adversarial loss, defined as α = ‖∆fT ,δfS ‖ 2 D/‖∆fT ,δfT ‖ 2 D ≈ Ladv(fT , δfS ; y,D)/Ladv(fT , δfT ; y,D) when fT achieves low error. Note that α ∈ [0, 1]. Finally, we fine-tune the last layer of fS , and get the true transferred loss.
Dataset. Denote a radial basis function as φi(x) = e−‖x−µi‖ 2 2/(2σi) 2
, and we set the target ground truth function to be the sum of M = 100 basis functions as f = ∑M i=1 φi, where each entry of the parameters are sampled once from U(−0.5, 0.5). We set the dimension of x to be 30, and the dimension of y to be 10. We generate N = 1200 samples of x from a Gaussian mixture formed by three Gaussian with different centers but the same covariance matrix Σ = I . The centers are sampled randomly from U(−0.5, 0.5)n. We use the ground truth regressor f to derive the corresponding y for each x. That is, we want our neural networks to approximate f on the Gaussian mixture.
Results. We present two sets of experiment in Figure 2. The correlations between adversarial transferabilities (τ1, τ2, α) and the knowledge transferability (transferred loss) are observed. The upper bound for the difference of gradients (Theorem 2) basically tracks its true value. Although the absolute value of the upper bound on the transferred loss (Theorem 3) can be big compared to the true transferred loss, their trends are similar. We note the big difference in absolute value is due to the use of β-smoothness, which considers the worst case scenario. It is also observed that τ1 tracks the normalized adversarial cross loss α, as Proposition 2 suggests.

heading: 4.2 ADVERSARIAL TRANSFERABILITY INDICATES KNOWLEDGE-TRANSFER AMONG DATA
text: DISTRIBUTIONS
In this experiment, we show that the closer the source data distribution is to the target data distribution, the more adversarially transferable the source model to the reference model, thus we observe that the source model is more knowledge transferable to the target dataset. We demonstrate this on both image and natural language domains. Dataset. Image: 5 source datasets (5 source models) are constructed based on CIFAR-10 (Hinton et al., 2012) and a single target dataset (1 reference model) based on STL-10 (Coates et al., 2011). Each of the source datasets consists of 4 classes from CIFAR10 and the target dataset also consists of 4 classes from STL10. Natural Language: We select 4 diverse natural language datasets: AG’s News (AG), Fake News Detection (Fake), IMDB, Yelp Polarity (Yelp). Then we pick IMDB as the target and the rest as sources. Adversarial Transferability. Image: We take 1000 images (STL-10) from the target dataset and generate 1000 adversarial examples on each of the five source models. We run 10 step PGD L∞ attack with = 0.1. Then we measure the effectiveness of the adversarial examples by the cross-entropy loss on the reference model. Natural Language: We take 100 sample sentences from target dataset(IMDB) and generate adversarial sentences on each of the source models(AG, Fake, Yelp) with TextFooler(Jin et al., 2019). The ratio of changed words is constrained to less or equal to 0.1. Then, we measure their adversarial transferability against the reference model(IMDB). Knowledge Transferability. To measure the knowledge transferability, we fine-tune a new linear layer on the target dataset to replace the last layer of each source model to generate the corresponding transferred models. Then we measure the performance of the transferred models on the target dataset based on the standard accuracy and cross-entropy loss. Results From Figure 4.2, it’s clear that if the source models that has highest adversarial transferability, its corresponding transferred model achieves the highest transferred accuracy. This phenomenon is prominent in both image and natural language domains. The results in Figure 4.2 (b) could verify the implication by our theory that only τ1 is not sufficient for indicating knowledge transferability.

heading: 4.3 ADVERSARIAL TRANSFERABILITY INDICATING KNOWLEDGE-TRANSFER AMONG ATTRIBUTES
text: In addition to the data distributions, we validate our theory on another dimension, attributes. This experiment suggests that the more adversarially transferable the source model of certain attributes is to the reference model, the better the model performs on the target task aiming to learn target attributes. Dataset CelebA (Liu et al., 2018) consists of 202,599 face images from 10,177 identities. A reference facial recognition model is trained on this identities. Each image also comes with 40 binary attributes, on which we train 40 source models. Our goal is to test whether source models of source attributes, can transfer to perform facial recognition. Adversarial Transferability We sample 1000 images from CelebA and perform a virtual adversarial attack as described in section 3 on each of the 40 attribute classifiers. Then we measure the adversarial transfer effectiveness of these adversarial examples on the reference facial recognition model. Knowledge Transferability To fairly assess the knowledge transferability, we test the 40 transferred models on 7 well-known facial recognition benchmarks, LFW (Huang et al., 2007), CFP-FF, CFP-FP (S. Sengupta, 2016), AgeDB (Moschoglou et al., 2017), CALFW, CPLFW (Zheng et al., 2017) and VGG2-FP (Cao et al., 2018). We report the average classification accuracy target datasets. Result In Table 1, we list the top-5 attribute source models that share the highest adversarial transferability and the performance of their transferred models on the 7 target facial recognition benchmarks. We observe that the attribute "Young" has the highest adversarial transferability; as a result, it also achieves highest classification average performance across the 7 benchmarks.

heading: 4.4 ADVERSARIAL TRANSFERABILITY INDICATING KNOWLEDGE-TRANSFER AMONG TASKS

heading: 5 CONCLUSION
text: We theoretically analyze the relationship between adversarial transferability and knowledge transferability, along with thorough experimental justifications in diverse scenarios. Both our theoretical and empirical results show that adversarial transferability can indicate knowledge transferability, which reveal important properties of machine learning models. We hope our discovery can inspire and facilitate further investigations, including model interpretability, fairness, robust and efficient transfer learning, and etc.

heading: A DISCUSSION ABOUT VALIDNESS OF THE NOTATIONS
text: Before starting proving our theory, it is necessary to show that our mathematical tools are indeed valid. It is easy to verify that 〈·, ·〉D is a valid inner product inherited form standard Euclidean inner product. Therefore, the norm ‖ · ‖D, induced by the inner product, is also a valid norm. What does not come directly is the validness of the norm ‖ · ‖D,2. Particularly, whether it satisfies the triangle inequality. Recall that, for a function of matrix output F : supp(D) → Rd×m, its L2(D)-norm in accordance with matrix 2-norm is defined as
‖F‖D,2 = √ Ex∼D‖F (x)‖22.
For two functions F,G, both are supp(D)→ Rd×m, we can verify the norm ‖ · ‖D,2 satisfies triangle inequality as shown in the following. Applying the triangle inequality of the spectral norm, and with some algebra manipulation, it holds that
‖F +G‖D,2 = √ Ex∼D‖F (x) +G(x)‖22
≤ √ Ex∼D (‖F (x)‖2 + ‖G(x)‖2)2
= √ Ex∼D‖F (x)‖22 + Ex∼D‖G(x)‖22 + 2Ex∼D‖F (x)‖2‖G(x)‖2
= √ ‖F‖2D,2 + ‖G‖2D,2 + 2Ex∼D‖F (x)‖2‖G(x)‖2. (2)
Applying the Cauchy-Schwarz inequality, we can see that Ex∼D‖F (x)‖2‖G(x)‖2 ≤ √ Ex∼D‖F (x)‖22 · Ex∼D‖G(x)‖22
= ‖F‖D,2 · ‖G‖D,2. Plugging this into (2) would complete the proof, i.e.,
(2) ≤ √ ‖F‖2D,2 + ‖G‖2D,2 + 2‖F‖D,2 · ‖G‖D,2
= √ (‖F‖D,2 + ‖G‖D,2)2
= ‖F‖D,2 + ‖G‖D,2.

heading: B PROOF OF PROPOSITION 1
text: Proposition 1 (Restated). Both τ1 and τ2 are in [0, 1].
Proof. We are to prove that τ1 and τ2 are both in the range of [0, 1]. As τ1 is squared cosine, it is trivial that τ1 ∈ [0, 1]. Therefore, we will focus on τ2 in the following. Recall that the τ2 from f1 to f2 is defined as
τf1→f22 = 〈2∆f2,δf1 −A∆f1,δf1 ,A∆f1,δf1 〉D
‖∆f2,δf1‖ 2 D
,
whereA is a constant matrix defined as A = proj(Ex∼D[∆f2,δf1 (x)∆f1,δf1 (x) >] ( Ex∼D[∆f1,δf1 (x)∆f1,δf1 (x) >] )† , ‖∆f2,δf1 ‖D ‖∆f1,δf1 ‖D ).
For notation convenience, we will simply use τ2 to denote τ f1→f2 2 in this proof.
τ2 characterizes how similar are the changes in both the function values of f1 : Rn → Rm and f2 : Rn → Rd in the sense of linear transformable, given attack generated on f1. That is being said, it is associated to the function below, i.e,
h(B) = ∥∥∆f2,δf1 −B∆f1,δf1∥∥2D = Ex∼D ∥∥∆f2,δf1 (x)−B∆f1,δf1 (x)∥∥22 ,
where ∆f1,δf1 ∈ R m, ∆f2,δf1 ∈ R d, andB ∈ Rd×m.
As ∥∥∆f2,δf1 (x)−B∆f1,δf1 (x)∥∥22 is convex with respect to B, its expectation, i.e. h(B), is also convex.
Therefore, h(B) it achieves global minima when ∂h∂B = 0.
∂h ∂B = Ex∼D ∂ ∂B (∥∥∆f2,δf1 (x)−B∆f1,δf1 (x)∥∥22) = 2Ex∼D [( B∆f1,δf1 (x)−∆f2,δf1 (x) ) ∆f1,δf1 (x)
>] = 2Ex∼D [ B∆f1,δf1 (x)∆f1,δf1 (x) > −∆f2,δf1 (x)∆f1,δf1 (x) >]
= 2BEx∼D [ ∆f1,δf1 (x)∆f1,δf1 (x) >]− 2Ex∼D [∆f2,δf1 (x)∆f1,δf1 (x)>] . Letting ∂h∂B = 0, and denoting the solution asB ∗, we have
B∗ = Ex∼D [ ∆f2,δf1 (x)∆f1,δf1 (x) >] (Ex∼D [∆f1,δf1 (x)∆f1,δf1 (x)>])† . Noting thatA = proj(B∗,
‖∆f2,δf1 ‖D ‖∆f1,δf1 ‖D ) is scaledB∗, we denoteA = ψB∗, where ψ a scaling factor
depending onB∗ and ‖∆f2,δf1 ‖D ‖∆f1,δf1 ‖D . According to the definition of the projection operator, we can see that 0 < ψ ≤ 1. ReplacingB byA we have,
h(A) = ∥∥∆f2,δf1 −A∆f1,δf1∥∥2D = 〈∆f2,δf1 −A∆f1,δf1 ,∆f2,δf1 −A∆f1,δf1 〉D
= ∥∥∆f2,δf1∥∥2D − 〈2∆f2,δf1 −A∆f1,δf1 ,A∆f1,δf1 〉D
= (1− τ2) ∥∥∆f2,δf1∥∥2D .
It is obvious that h(A) = ∥∥∆f2,δf1 −A∆f1,δf1∥∥2D ≥ 0, thus we have τ2 ≤ 1.
As for the lower bound for τ2, we will need to use properties ofB. DenotingO as an all-zero matrix, it holds that
h(B∗) = min B {h(B)} ≤ h(O). (3)
For A = ψB∗, according to the convexity of h(·) and the fact that ψ ∈ [0, 1], we can see the following, i.e.,
h(A) = h(ψB∗) = h(ψB∗ + (1− ψ)O) ≤ ψh(B∗) + (1− ψ)h(O).
Applying (3) to the above, we can see that
h(A) ≤ h(O).
Noting that h(A) = (1− τ2) ∥∥∆f2,δf1∥∥2D and h(O) = ∥∥∆f2,δf1∥∥2D, the above inequality suggests that
(1− τ2) ∥∥∆f2,δf1∥∥2D ≤ ∥∥∆f2,δf1∥∥2D ,
0 ≤ τ2.
Therefore, τ2 is upper bounded by 1 and lower bounded by 0.

heading: C PROOF OF THEOREM 1
text: Before actually proving the theorem, let us have a look at what τ1 and τ2 are in the case where fS and fT are both Rn → R. In this case, both τ1 and τ2 come out in an elegant form. Let us show what the two metrics are to have further intuition on what τ1 and τ2 characterize.
First, let us see what the attack is in this case. As function f has one-dimensional output, its gradient is a vector∇f ∈ Rn. Thus,
δf, (x) = arg max ‖δ‖≤ ‖∇f(x)>δ‖2 = ∇f(x) ‖∇f(x)‖2
Then, the τ1 becomes
τ1(x) = 〈∇fS(x),∇fT (x)〉2
‖∇fS(x)‖22 · ‖∇fT (x)‖22 which is the squared cosine (angle) between two gradients.
For τ2, the matrixA degenerates to a scalar constant
A = 〈∆fT ,δfS ,∆fS ,δfS 〉D ‖∆fS ,δfS ‖ 2 D ,
and the second metric becomes
τfS→fT2 = 〈∆fS ,δfS ,∆fT ,δfS 〉 2 D
‖∆fS ,δfS ‖ 2 D · ‖∆fT ,δfS ‖ 2 D
We can see, it is interestingly in the same form of the first metric τ1. We will simply use τ2 to denote τfS→fT2 afterwards. Theorem 1 (Restated). For two functions fS and fT that both are Rn → R, there is an affine function g : R→ R, so that
‖∇fT −∇(g ◦ fS)‖2D = Ex∼D [ (1− τ1(x)τ2)‖∇fT (x)‖22 ] ,
where g is defined as g(x) = Ax+ Const.
Moreover, if assuming that fT is L-Lipschitz continuous, i.e., ‖∇fT (x)‖2 ≤ L for ∀x ∈ supp(D), we can have a more elegant statement:
‖∇fT −∇(g ◦ fS)‖2D ≤ (1− τ1τ2)L2.
Proof. In the case where g is a one-dimensional affine function, we write is as g(z) = Az+ b, where A is defined in the definition of τ2 (Definition 3). In this case, it enjoys a simple form of
A = 〈∆fT ,δfS ,∆fS ,δfS 〉D ‖∆fS ,δfS ‖ 2 D .
Then, we can see that
‖∇fT −∇(g ◦ fS)‖2D = ‖∇fT −A∇fS‖2D = Ex∼D [ ‖∇fT (x)−A∇fS(x)‖22 ] . (4)
To continue, we split ∇fT as two terms, i.e., one on the direction on ∇fS and one orthogonal to ∇fS . Denoting φ(x) as the angle between∇fT (x) and ∇fS(x) in Euclidean space, we have
∇fT (x) = cos(φ(x)) ‖∇fT (x)‖2 ‖∇fS(x)‖2 ∇fS(x) +∇fT (x)− cos(φ(x)) ‖∇fT (x)‖2 ‖∇fS(x)‖2 ∇fS(x)
= cos(φ(x)) ‖∇fT (x)‖2 ‖∇fS(x)‖2 ∇fS(x) + v(x), (5)
where we denote v(x) = ∇fT (x)− cos(φ(x))‖∇fT (x)‖2‖∇fS(x)‖2∇fS(x) for notation convenience.
We can see that v(x) is orthogonal to ∇fS(x), thus ‖v(x)‖2 = √
1− cos2(φ(x))‖∇fT (x)‖2. Recall that actually τ1(x) = cos2(φ(x)), it can be written as ‖v(x)‖2 = √ 1− τ1(x)‖∇fT (x)‖2.
Then, plugging (5) into (4) we have (4) = Ex∼D [ ‖ cos(φ(x))‖∇fT (x)‖2
‖∇fS(x)‖2 ∇fS(x) + v(x)−A∇fS(x)‖22 ] = Ex∼D [∥∥∥∥(cos(φ(x))‖∇fT (x)‖2‖∇fS(x)‖2 −A ) ∇fS(x) + v(x) ∥∥∥∥2 2 ]
= Ex∼D [∥∥∥∥(cos(φ(x))‖∇fT (x)‖2‖∇fS(x)‖2 −A ) ∇fS(x) ∥∥∥∥2 2 + ‖v(x)‖22 ]
= Ex∼D [∥∥∥∥(cos(φ(x))‖∇fT (x)‖2‖∇fS(x)‖2 −A ) ∇fS(x) ∥∥∥∥2 2 + (1− τ1(x))‖∇fT (x)‖22 ]
= Ex∼D [∥∥∥∥(cos(φ(x))‖∇fT (x)‖2‖∇fS(x)‖2 −A ) ∇fS(x) ∥∥∥∥2 2 ] + Ex∼D(1− τ1(x))‖∇fT (x)‖22
= Ex∼D [( cos(φ(x))
‖∇fT (x)‖2 ‖∇fS(x)‖2
−A )2 ‖∇fS(x)‖22 ] + Ex∼D(1− τ1(x))‖∇fT (x)‖22.
(6)
Now let us deal with the first term by plugging in
A = 〈∆fT ,δfS ,∆fS ,δfS 〉D ‖∆fS ,δfS ‖ 2 D ,
where ∆fT ,δfS (x) = cos(φ(x))‖∇fT (x)‖2 and ∆fS ,δfS (x) = ‖∇fS(x)‖2, and we have
Ex∼D (
cos(φ(x)) ‖∇fT (x)‖2 ‖∇fS(x)‖2
−A )2 ‖∇fS(x)‖22
= Ex∼D (cos(φ(x))‖∇fT (x)‖2 −A ‖∇fS(x)‖2) 2
= 1
2 Ex∼D ( ∆fT ,δfS (x)−A∆fS ,δfS (x) )2 = 1
2 Ex∼D ( ∆fT ,δfS (x) 2 +A2∆fS ,δfS (x) 2 − 2A∆fT ,δfS (x)∆fS ,δfS (x) ) = 1
2 (∥∥∥∆fT ,δfS ∥∥∥2D +A2 ∥∥∥∆fS ,δfS ∥∥∥2D − 2A〈∆fT ,δfS ,∆fS ,δfS 〉D )
= 1
2 (∥∥∥∆fT ,δfS ∥∥∥2D + 〈∆fT ,δfS ,∆fS ,δfS 〉 2 D
‖∆fS ,δfS ‖ 2 D
− 2 〈∆fT ,δfS ,∆fS ,δfS 〉 2 D
‖∆fS ,δfS ‖ 2 D
)
= ∥∥∥∆fT ,δfS ∥∥∥2D 2 ( 1− 〈∆fT ,δfS ,∆fS ,δfS 〉 2 D
‖∆fS ,δfS ‖ 2 D · ‖∆fT ,δfS ‖ 2 D ) = (1− τ2)Ex∼D [ cos2(x)‖∇fT (x)‖22
] = (1− τ2)Ex∼D [ τ1(x)‖∇fT (x)‖22 ] . (7)
Plugging (7) into (6), we finally have
‖∇fT −∇(g ◦ fS)‖2D = (1− τ2)Ex∼D [ τ1(x)‖∇fT (x)‖22 ] + Ex∼D(1− τ1(x))‖∇fT (x)‖22
= Ex∼D [ (1− τ2τ1(x))‖∇fT (x)‖22 ] ≤ (1− τ1τ2)L2,
which completes the proof.

heading: D PROOF OF THEOREM 2
text: Theorem 2 (Restated). For two functions fS : Rn → Rm, and fT : Rn → Rd, there is an affine function g : Rm → Rd, so that
‖∇fT −∇(g ◦ fS)‖2D ≤ 5Ex∼D  ( (1− τ1(x)τ2) + (1− τ1(x))(1− τ2)λfT (x)2 ) ‖∇fT (x)‖22
+ (λfT (x) + λfS (x)) 2 ‖∇fS(x)‖ 2 2
‖∇fS‖2D,2 ‖∇fT ‖2D,2  , where g is defined as g(z) = Az +Const.
Moreover, if assuming that fT is L-Lipschitz continuous, i.e., ‖∇fT (x)‖2 ≤ L for ∀x ∼ supp(D), and considering the worst-case singular value ratio λ, we can have a more elegant statement:
‖∇fT −∇(g ◦ fS)‖2D ≤ ( (1− τ1τ2) + (1− τ1)(1− τ2)λ2fT + (λfT + λfS ) 2 ) 5L2.
Proof. Recall that the matrixA is defined in Definition 3, i.e.,
A = proj(Ex∼D[∆fT ,δfS (x)∆fS ,δfS (x) >] ( Ex∼D[∆fS ,δfS (x)∆fS ,δfS (x) >] )† , ‖∆fT ,δfS ‖D ‖∆fS ,δfS ‖D ),
and we can see
‖∇fT −∇(g ◦ fS)‖2D,2 = ‖∇f>T −∇(g ◦ fS)>‖2D,2 = ‖∇f>T −A∇f>S ‖2D,2 = Ex∼D‖∇fT (x)> −A∇fS(x)>‖22 = Ex∼D max
‖t‖2=1 ‖∇fT (x)>t−A∇fS(x)>t‖22, (8)
where the last equality is due to the definition of matrix spectral norm.
Denoting∇f> as either the Jacobian matrix∇f>T or∇f>S , Singular Value Decomposition (SVD) suggests that ∇f(x)> = UΣV >, where Σ is a diagonal matrix containing all singular values ordered by their absolute values. Let σ1, · · · , σn denote ordered singular values. Nothing that the number of singular values that are non-zero may be less than n, so we fill the empty with zeros, such that each of them have corresponding singular vectors, i.e., the column vectors v1, · · · ,vn in V . That is being said, ∀i ∈ [n], we have
‖∇f(x)>vi‖2 = |σi|.
Let θi and vi denote the singular values and vectors for ∇fS(x)>. Noting that {vi}ni=1 define a orthonormal basis for Rn, we can represent
t = n∑ i=1 θivi, (9)
where ∑n i=1 θ 2 i = 1.
As adversarial attack is about the largest eigenvalue of the gradient, plugging (9) into (8), we can split it into two parts, i.e.,
(8) = Ex∼D max ‖t‖2=1 ∥∥∥∥∥∇fT (x)> ( n∑ i=1 θivi ) −A∇fS(x)> ( n∑ i=1 θivi )∥∥∥∥∥ 2
2
= Ex∼D max ‖t‖2=1 ∥∥∥∥∥∥∥ ∇fT (x)> (θ1v1)−A∇fS(x)> (θ1v1) +∇fT (x)> ( n∑ i=2 θivi ) −A∇fS(x)> ( n∑ i=2 θivi )∥∥∥∥∥∥∥ 2
2
. (10)
Denoting u = ∑n i=2 θivi, we can see this vector is orthogonal to v1. Let us denote v ′ 1 as the singular vector with the biggest absolute singular value of ∇fT (x)>, parallel with attack δfT . Now we split
u = u1 + u2 into two terms, where u1 is parallel to v′1, and u2 is orthogonal to u1. As u1 is in the orthogonal space to v1 while parallel with v′1, it is bounded by the sine value of the angle between v1 and v′1, i.e., √ 1− τ1(x). Hence, noting that u is part of the unit vector t,
‖u1‖2 ≤ √ 1− τ1(x)‖u‖2 ≤ √ 1− τ1(x). (11)
Plugging u in (10), we have
(10) = Ex∼D max ‖t‖2=1 ∥∥∥∥∥∇fT (x)> (θ1v1)−A∇fS(x)> (θ1v1)+∇fT (x)> (u1 + u2)−A∇fS(x)>u ∥∥∥∥∥ 2
2
≤ Ex∼D max ‖t‖2=1  ∥∥∇fT (x)> (θ1v1)−A∇fS(x)> (θ1v1)∥∥2︸ ︷︷ ︸ X1 + ∥∥∇fT (x)>u1∥∥2︸ ︷︷ ︸
X2
+ ∥∥∇fT (x)>u2 −A∇fS(x)>u∥∥2︸ ︷︷ ︸
X3
 2 , (12)
where the inequality is due to triangle inequality.
There are three terms we have to deal with, i.e., X1, X2 and X3. Regarding the first term, v1 in X1 aligns with the attack δfS (x), which we have known through adversarial attack. The second term X2 is trivially bounded by (11). Although adversarial attacks tell us nothing about X3, it can be bounded by the second largest singular values.
Let us first deal with two easiest, i.e., X2 and X3. Applying (11) on X2 directly, we have X2 = ‖∇fT (x)>‖2 · ‖u1‖2 ≤ √ 1− τ1(x)‖∇fT (x)>‖2.
For X3, noting that u2 is orthogonal to v′1, and u is orthogonal to v1, we can see that u2 has no components of the largest absolute singular vector of ∇fT (x)>, and u has no components of the largest absolute singular vector of∇fT (x)>. Therefore,
X3 ≤ ∥∥∇fT (x)>u2∥∥2 + ∥∥A∇fS(x)>u∥∥2
≤ σfT ,2(x) ‖u2‖2 + σfS ,2(x) ‖A‖2 ‖u‖2 = λfT (x) ∥∥∇fT (x)>∥∥2 ‖u2‖2 + λfS (x)∥∥∇fS(x)>∥∥2 ‖A‖2 ‖u‖2 ≤ λfT (x)
∥∥∇fT (x)>∥∥2 + λfS (x)∥∥∇fS(x)>∥∥2 ‖A‖2 , where the first inequality is due to triangle inequality, the second inequity is done by the attributes of singular values, and the definition of matrix 2-norm. The equality is done simply by applying the definition of singular values ratio (Definition 4), and the third inequality is due to the fact that ‖u2‖2 ≤ ‖u‖2 ≤ 1. Before dealing with X1, let us simplify (12) by relax the square of summed terms to sum of squared terms, as the following.
(12) = Ex∼D max ‖t‖2=1 (X1 +X2 +X3) 2
= Ex∼D max ‖t‖2=1 X21 +X 2 2 +X 2 3 + 2X1X2 + 2X2X3 + 2X1X3
≤ Ex∼D max ‖t‖2=1 X21 +X 2 2 +X 2 3 + 2 max{X21 , X22}+ 2 max{X22 , X23}+ 2 max{X21 , X23}
≤ Ex∼D max ‖t‖2=1 X21 +X 2 2 +X 2 3 + 2(X 2 1 +X 2 2 ) + 2(X 2 2 +X 2 3 ) + 2(X 2 1 +X 2 3 )
= Ex∼D max ‖t‖2=1 5(X21 +X 2 2 +X 2 3 ). (13)
We note that this relaxation is not necessary, but simply for the simplicity of the final results without breaking what our theory suggests.
Bring what we we have about X2 and X3, and noting that θ1 ≤ 1 depends on t, we can drop the max operation by
(13) = Ex∼D max ‖t‖2=1 5(X21 +X 2 2 +X 2 3 )
= Ex∼D max ‖t‖2=1
5( ∥∥∇fT (x)> (θ1v1)−A∇fS(x)> (θ1v1)∥∥22 +X22 +X23 )
≤ 5Ex∼D ∥∥∇fT (x)>v1 −A∇fS(x)>v1∥∥22 + (1− τ1(x)) ‖∇fT (x)‖22 + ( (λfT (x) + λfS (x)) ∥∥∇fS(x)>∥∥2 ‖A‖2)2 .  (14)
Now, let us deal with the first term. As v1 is a unit vector and is in fact the direction of fS(x)’s adversarial attack, we can write δfS , (x) = v1. Hence,
Ex∼D ∥∥∇fT (x)>v1 −A∇fS(x)>v1∥∥22
= Ex∼D 1 2 ∥∥∇fT (x)>δfS , (x)−A∇fS(x)>δfS , (x)∥∥22
= Ex∼D 1
2 ∥∥∥∆fT ,δfS (x)−A∆fS ,δfS (x)∥∥∥22 , (15) where the last equality is derived by applying the definition of ∆(x), i.e., equation (1). Note that we omit the in δfS , for notation simplicity.
The matrix A is deigned to minimize (15), as shown in the proof of Proposition 1. Expanding the term we have
(15) = 1
2 Ex∼D [∥∥∥∆fT ,δfS (x)∥∥∥22 + ∥∥∥A∆fS ,δfS (x)∥∥∥22 − 2〈∆fT ,δfS (x),A∆fS ,δfS (x)〉 ]
= 1
2 (∥∥∥∆fT ,δfS ∥∥∥2D + ∥∥∥A∆fS ,δfS ∥∥∥2D − 2〈∆fT ,δfS ,A∆fS ,δfS 〉D )
= ∥∥∥∆fT ,δfS ∥∥∥2D 2 (1− τ2)
= (1− τ2)Ex∼D ∥∥∇fT (x)>v1∥∥22 . (16)
Recall that v1 is a unit vector aligns the direction of δfS , and we have used v ′ 1 to denote a unit vector that aligns the direction of δfT . As τ1 tells us about the angle between the two, let us split v1 into to orthogonal vectors, i.e., v1 = √ τ1(x)v ′ 1 + √ 1− τ1(x)v′1,⊥, where v′1,⊥ is a unit vector that is orthogonal to v′1.
Plugging this into (16) we have
(16) = (1− τ2)Ex∼D ∥∥∥∇fT (x)>(√τ1(x)v′1 +√1− τ1(x)v′1,⊥)∥∥∥2
2 = (1− τ2)Ex∼D [∥∥∥∇fT (x)>√τ1(x)v′1∥∥∥2 2 + ∥∥∥∇fT (x)>√1− τ1(x)v′1,⊥∥∥∥2 2 ] = (1− τ2)Ex∼D [ τ1(x)
∥∥∇fT (x)>∥∥22 + (1− τ1(x))λfT (x)2 ∥∥∇fT (x)>∥∥22] , where the second equality is due to the image of v′1 and v ′ 1,⊥ after linear transformation∇fT (x)> are orthogonal, which can be easily observed through SVD.
Plugging this in (14), and with some regular algebra manipulation, finally we have
(14) = 5Ex∼D  (1− τ2) [ τ1(x) ∥∥∇fT (x)>∥∥22 + (1− τ1(x))λfT (x)2 ∥∥∇fT (x)>∥∥22] +(1− τ1(x)) ‖∇fT (x)‖22 + (λfT (x) + λfS (x)) 2 ∥∥∇fS(x)>∥∥22 ‖A‖22 
= 5Ex∼D  (1− τ1(x)τ2) ∥∥∇fT (x)>∥∥22 +(1− τ1(x))(1− τ2)λfT (x)2
∥∥∇fT (x)>∥∥22 + (λfT (x) + λfS (x)) 2 ∥∥∇fS(x)>∥∥22 ‖A‖22  . (17) Recall thatA is from a norm-restricted matrix space, i.e., theA is scaled so that its spectral norm is no greater than ‖∆fT ,δfS ‖D ‖∆fS,δfS ‖D , thus
‖A‖22 ≤ ‖∆fT ,δfS ‖ 2 D ‖∆fS ,δfS ‖ 2 D ≤ ‖∆fT ,δfT ‖ 2 D ‖∆fS ,δfS ‖ 2 D
= Ex∼D‖∆fT ,δfT (x)‖ 2 2
Ex∼D‖∆fS ,δfS (x)‖ 2 2 = Ex∼D‖∇f>T (x)‖22 Ex∼D‖∇f>S (x)‖22
= ‖∇f>T ‖2D,2 ‖∇f>S ‖2D,2 . (18)
Hence, plugging the above inequality to (17), the first statement of the theorem is proven, i.e.,
(17) ≤ 5Ex∼D  (1− τ1(x)τ2) ∥∥∇fT (x)>∥∥22 +(1− τ1(x))(1− τ2)λ2fT ∥∥∇fT (x)>∥∥22 + (λfT (x) + λfS (x))
2 ∥∥∇fS(x)>∥∥22 ‖∇f>T ‖2D,2‖∇f>S ‖2D,2  . (19) To see the second statement of the theorem, we assume fT is L-Lipschitz continuous, i.e., ‖∇fT (x)‖2 ≤ L for ∀x ∈ supp(D), and considering the worst-case singular value ratio λ = maxx∈supp(D) for either fS , fT , we can continue as
(19) ≤ 5  Ex∼D [ (1− τ1(x)τ2) ∥∥∇fT (x)>∥∥22] +Ex∼D [ (1− τ1(x))(1− τ2)λ2fT ∥∥∇fT (x)>∥∥22] +Ex∼D [ (λfT + λfS ) 2 ∥∥∇fS(x)>∥∥22 ‖∇f>T ‖2D,2‖∇f>S ‖2D,2 ] 
= 5  Ex∼D [ (1− τ1(x)τ2) ∥∥∇fT (x)>∥∥22] +Ex∼D [ (1− τ1(x))(1− τ2)λ2fT
∥∥∇fT (x)>∥∥22] + (λfT + λfS ) 2 ‖∇f>T ‖2D,2  = 5Ex∼D ( (1− τ1(x)τ2) + (1− τ1(x))(1− τ2)λ2fT + (λfT + λfS ) 2 )∥∥∇fT (x)>∥∥22
≤ Ex∼D ( (1− τ1(x)τ2) + (1− τ1(x))(1− τ2)λ2fT + (λfT + λfS ) 2 ) 5L2
= ( (1− τ1τ2) + (1− τ1)(1− τ2)λ2fT + (λfT + λfS ) 2 ) 5L2,
where the first inequality is due to the definition of worst-case singular value ratio, the last inequality is by Lipschitz condition, and the last equality is done be simply applying the definition of τ1.

heading: E PROOF OF THEOREM 3
text: The idea for proving Theorem 3 is straight-forward: bounded gradients difference implies bounded function difference, and then bounded function difference implies bounded loss difference.
To begin with, let us prove the following lemma. Lemma 1. Without loss of generality we assume ‖x‖2 ≤ 1 for ∀x ∈ supp(D). Consider functions fS : Rn → Rm, fT : Rn → Rd, and an affine function g : Rm → Rd, suggested by Theorem 1 or Theorem 2, such that g(fS(0)) = fT (0), if both fT , fS are β-smooth in {x | ‖x‖ ≤ 1}, we have
‖fT − g ◦ fS‖D ≤ ‖∇fT −∇(g ◦ fS)‖D,2 + (
1 + ‖∇fT ‖D,2 ‖∇fS‖D,2
) β.
Proof. Let us denote v(x) = fT (x)− g ◦ fS(x), and we can show the smoothness of v(·). As g(·) is an affine function satisfying g(fS(0)) = fT (0), it can be denoted as g(z) = A(z − fS(0)) + fT (0), whereA is a matrix suggested by Theorem 1 or Theorem 2. Therefore, denoting B1 = {x | ‖x‖ ≤ 1} as a unit ball, for ∀x,y ∈ B1 it holds that ‖∇v(x)−∇v(y)‖2 = ∥∥∇v(x)> −∇v(y)>∥∥ 2
= ∥∥∇fT (x)> −∇fT (y)> −A(∇fS(x)> −∇fS(y)>)∥∥2
≤ ∥∥∇fT (x)> −∇fT (y)>∥∥2 + ∥∥A(∇fS(x)> −∇fS(y)>)∥∥2
≤ ∥∥∇fT (x)> −∇fT (y)>∥∥2 + ‖A‖2 ∥∥∇fS(x)> −∇fS(y)>∥∥2 , (20)
where the last second inequality is due to triangle inequality, and the last inequality is by the property of spectral norm.
Applying the β-smoothness of fS and fT , and noting that ‖A‖2 ≤ ‖∇fT ‖D,2‖∇fS‖D,2 as shown in (18), we can continue as
(20) ≤ β ‖x− y‖2 + ‖A‖2 β ‖x− y‖2 ≤ β ‖x− y‖2 + ‖∇fT ‖D,2 ‖∇fS‖D,2 β ‖x− y‖2
= ( 1 + ‖∇fT ‖D,2 ‖∇fS‖D,2 ) β ‖x− y‖2 ,
which suggests that v(·) is (
1 + ‖∇fT ‖D,2 ‖∇fS‖D,2
) β-smooth.
We are ready to prove the lemma now. Applying the mean value theorem, for ∀x ∈ B1, we have v(x)− v(0) = ∇v(ξx)>x,
where ξ ∈ (0, 1) is a scalar number. Subtracting∇v(x)>x on both sides give v(x)− v(0)−∇v(x)>x = (∇v(ξx)−∇v(x))>x∥∥v(x)− v(0)−∇v(x)>x∥∥ 2 = ∥∥(∇v(ξx)−∇v(x))>x∥∥
2∥∥v(x)− v(0)−∇v(x)>x∥∥ 2 ≤ ‖(∇v(ξx)−∇v(x))‖2 ‖x‖2 .
Let us denote β1 = (
1 + ‖∇fT ‖D,2 ‖∇fS‖D,2
) β for notation convenience, and apply the definition of smooth-
ness: ‖v(x)− v(0)−∇v(x)>x‖2 ≤ β1(1− ξ)‖x‖22 ≤ β1. (21) Noting that v(0) = 0 and applying the triangle inequality, we have
‖v(x)− v(0)−∇v(x)>x‖2 ≥ ‖v(x)‖2 − ‖∇v(x)>x‖2 ≥ ‖v(x)‖2 − ‖∇v(x)>‖2 Plugging it into (21), we have
‖v(x)‖2 ≤ β1 + ‖∇v(x)>‖2 ‖v(x)‖22 ≤ β21 + ‖∇v(x)>‖22 + 2β1‖∇v(x)>‖2
Ex∼D‖v(x)‖22 ≤ β21 + Ex∼D‖∇v(x)>‖22 + 2β1Ex∼D‖∇v(x)>‖2 Ex∼D‖v(x)‖22 ≤ β21 + Ex∼D‖∇v(x)‖22 + 2β1Ex∼D‖∇v(x)‖2
‖v‖2D ≤ β21 + ‖∇v‖2D,2 + 2β1Ex∼D‖∇v(x)‖2
Applying Jensen’s inequality to the last term, we get ‖v‖2D ≤ β21 + ‖∇v‖2D,2 + 2β1 √ Ex∼D‖∇v(x)‖22
= β21 + ‖∇v‖2D,2 + 2β1 √ ‖∇v‖2D,2 = β 2 1 + ‖∇v‖2D,2 + 2β1‖∇v‖D,2
= (‖∇v‖D,2 + β1)2 Plugging β1 = (
1 + ‖∇fT ‖D,2 ‖∇fS‖D,2
) β and v = fT − g ◦ fS into the above inequality completes the
proof.
With the above lemma, it is easy to show the mean squared loss on the transferred model is also bounded. Theorem 3 (Restated). Without loss of generality we assume ‖x‖2 ≤ 1 for ∀x ∈ supp(D). Consider functions fS : Rn → Rm, fT : Rn → Rd, and an affine function g : Rm → Rd, suggested by Theorem 1 or Theorem 2, such that g(fS(0)) = fT (0). If both fT , fS are β-smooth, then
‖g ◦ fS − y‖2D ≤ ( ‖fT − y‖D + ‖∇fT −∇g ◦ fS‖D,2 + ( 1 + ‖∇fT ‖D,2 ‖∇fS‖D,2 ) β )2
Proof. Let us denote β1 = (
1 + ‖∇fT ‖D,2 ‖∇fS‖D,2
) β, and according to Lemma 1 we can see
‖fT − g ◦ fS‖D ≤ ‖∇fT −∇(g ◦ fS)‖D,2 + β1 (22)
Applying a standard algebra manipulation to the left hand side, and then applying triangle inequality, we have
‖fT − g ◦ fS‖D = ‖fT − y + y − g ◦ fS‖D ≥ ‖y − g ◦ fS‖D − ‖fT − y‖D.
Plugging this directly into (22), it holds that
‖y − g ◦ fS‖D − ‖fT − y‖D ≤ ‖∇fT −∇(g ◦ fS)‖D,2 + β1 ‖y − g ◦ fS‖D ≤ ‖fT − y‖D + ‖∇fT −∇(g ◦ fS)‖D,2 + β1
Replacing β1 by (
1 + ‖∇fT ‖D,2 ‖∇fS‖D,2
) β and taking the square, we can see Theorem 3 is proven.

heading: F PROOF OF PROPOSITION 2
text: Proposition 2 (Restated). If `T is mean squared loss and fT achieves zero loss on D, then the adversarial loss defined in Definition 6 is approximately upper and lower bounded by
Ladv(fT , δfS ; y,D) ≥ 2Ex∼D [ τ1(x) ‖∇fT (x)‖22 ] +O( 3),
Ladv(fT , δfS ; y,D) ≤ 2Ex∼D [( λ2fT + (1− λ 2 fT )τ1(x) ) ‖∇fT (x)‖22 ] +O( 3),
where O( 3) denotes a cubic error term.
Proof. Recall that the empirical adversarial transferability is defined as a loss
Ladv(fT , δfS , ; y,D) = Ex∼D `T (fT (x+ δfS , (x)), y(x)).
As `T is mean squared loss, and fT achieves zero loss, i.e., fT = y, we have
Ladv(fT , δfS , ; y,D) = Ex∼D ‖fT (x+ δfS , (x))− y(x)‖ 2 2
= Ex∼D ‖fT (x+ δfS , (x))− fT (x)‖ 2 2 .
Denoting δfS , (x) = δfS ,1(x), and define an auxiliary function h as
h(t) = fT (x+ tδfS ,1(x))− fT (x),
we can see that ‖fT (x+ δfS , (x))− fT (x)‖ 2 2 = ‖h( )‖ 2 2.
We can then apply Taylor expansion to approximate h( ) with a second order error term O( 2), i.e.,
h( ) = ∂h
∂t
∣∣ t=0 +O( 2) = ∇fT (x)>δfS ,1 +O( 2).
Therefore, assuming that ‖∇fT (x)‖2 is bounded for x ∈ supp(D), we have
‖fT (x+ δfS , (x))− fT (x)‖ 2 2 = ‖h( )‖ 2 2 = 2 ∥∥∇fT (x)>δfS ,1(x)∥∥22 +O( 3), (23)
where we have omit higher order error term, i.e., O( 4). Next, let us deal with the term ∥∥∇fT (x)>δfS ,1(x)∥∥22. Same us the technique we use in the proof of Theorem 2, we split δfS ,1(x) = v1 + v2, where v1 aligns the direction of δfT ,1(x), and v2 is orthogonal to v1. Noting that τ1(x) is the squared cosine of the angle between δfS ,1(x) and δfT ,1(x), we can see that
‖v1‖22 = τ1(x) ‖δfS ,1(x)‖ 2 2 = τ1(x), ‖v2‖22 = (1− τ1(x)) ‖δfS ,1(x)‖ 2 2 = (1− τ1(x)).
Therefore, we can continue as∥∥∇fT (x)>δfS ,1(x)∥∥22 = ∥∥∇fT (x)>(v1 + v2)∥∥22 = ∥∥∇fT (x)>v1∥∥22 + ∥∥∇fT (x)>v2∥∥22
= τ1(x) ‖∇fT (x)‖22 + ∥∥∇fT (x)>v2∥∥22 , (24)
where the second equality is because that v1 is corresponding to the largest singular value of ∇fT (x)>, and v2 is orthogonal to v1. Next, we derive the lower bound and upper bound for (24). The lower bounded can be derived as
τ1(x) ‖∇fT (x)‖22 + ∥∥∇fT (x)>v2∥∥22 ≥ τ1(x) ‖∇fT (x)‖22 ,
and the upper bounded can be derived as
τ1(x) ‖∇fT (x)‖22 + ∥∥∇fT (x)>v2∥∥22 ≤ τ1(x) ‖∇fT (x)‖22 + λfT (x)2 ‖∇fT (x)‖22 ‖v2‖22
= τ1(x) ‖∇fT (x)‖22 + λfT (x) 2 ‖∇fT (x)‖22 (1− τ1(x)) ≤ τ1(x) ‖∇fT (x)‖22 + λ 2 fT ‖∇fT (x)‖ 2 2 (1− τ1(x))
= ( λ2fT + (1− λ 2 fT )τ1(x) ) ‖∇fT (x)‖22 ,
where λfT (x) is the singular value ratio of fT at x, and λfT is the maximal singular value of fT .
Applying the lower and upper bound to (23), we finally have
‖fT (x+ δfS , (x))− fT (x)‖ 2 2 ≥ 2τ1(x) ‖∇fT (x)‖22 +O( 3), ‖fT (x+ δfS , (x))− fT (x)‖ 2 2 ≤ 2 ( λ2fT + (1− λ 2 fT )τ1(x) ) ‖∇fT (x)‖22 +O( 3). (25)
Noting that
Ladv(fT , δfS , ; y,D) = Ex∼D ‖fT (x+ δfS , (x))− fT (x)‖ 2 2 ,
we can see that taking expectation to (25) completes the proof.

heading: G EXPERIMENT DETAILS
text: All experiments are conducted on 4 RTX 2080 Ti GPUs and in python3 Ubuntu 16.04 environment.
G.1 ATTACK METHODS
PGD Attack is generated iteratively: denote step size as ξ, the source model as fS , and the loss function on the source problem. `S(·, ·). We initialize x0 to be uniformly sampled from the -ball B (x) of radius centered as instance x, and then generate the adversarial instance iteratively: at step t we compute xt+1 = xt + ξ · sign(∇xt`S(fS(xt), fS(x))). Denoting the adversarial example at instance x using PGD on source model fS as PGDfS (x), we measure the adversarial loss from fS to fT based on the loss `T (·, y) of fT on target data D given attacks generated on fS , i.e.,
LT (fT ◦ PGDfS ; y,D) = Ex∼D `T (fT (PGDfS (x)), y(x)).
TextFooler iteratively replaces words in target sentences by looking up similar words in the dictionary. It pauses when the predicted label is changed or runs out of the attack budget. We modify it such that it pauses when the percentage of changed words reaches 10%.
G.2 ADVERSARIAL TRANSFERABILITY INDICATES KNOWLEDGE-TRANSFER AMONG DATA DISTRIBUTIONS
Details of Dataset construction For the image domain, we divide the classes of the original datasets into two categories, animals (bird, cat, deer, dog) and transportation vehicles (airplane, automobile, ship, truck). Each of the source datasets consists of different a percentage of animals and transportation vehicles, while the target dataset contains only transportation vehicles, which is meant to control the closeness of the two data distributions. Details of Model Training Image: we train five source models on the five source datasets from 0% animals to 100% animals, and one reference models on STL-10 with identical architectures and hyperparameters. We use SGD optimizer and standard cross-entropy loss with learning rate 0.1, momentum 0.9, and weight decay 10−4. Each model is trained for 300 epochs. Natural Language: we fine-tune a Bert on each of the datasets with Adam and learning rate 0.0003 for 100 epochs. For transferred models, we run Adam with a smaller learning rate 0.0001 for 3 epochs.
G.3 ADVERSARIAL TRANSFERABILITY INDICATING KNOWLEDGE-TRANSFER AMONG ATTRIBUTES
Details of Model Training We train 40 binary source classifiers on each of the 40 attributes of CelebA with ResNet18 (He et al., 2016). All the classifiers are trained with optimizer Adadelta with a learning rate of 1.0 for 14 epochs. We also train a facial recognition model as a reference model on CelebA with 10,177 identities using ResNet18 as the controlled experiment.The reference facial recognition model is optimized with SGD and initial learning rate 0.1 on the ArcFace (Deng et al., 2019) with focal loss (Lin et al., 2017) for 125 epochs. For each source model, we construct a transferred model by stripping off the last layers and attaching a facial recognition head without parameters. Then we use the 40 transferred models to evaluate the knowledge transferability on 7 facial recognition benchmarks.
G.4 ADVERSARIAL TRANSFERABILITY INDICATING KNOWLEDGE-TRANSFER AMONG TASKS
Details of Model Training We use 15 pretrained models released in the task bank (Zamir et al., 2018) as the source models. Each source model consists of two parts, an encoder, and a decoder. The encoder is a modified ResNet50 without pooling, homogeneous across all tasks, whereas the decoder is customized to suit the output of each task. When measuring the adversarial transferability, we will use each source model as a reference model and compute the transferability matrix as described below.
Adversarial Transferability Matrix (ATM) is used here to measure the adversarial transferability between multiple tasks, modified from the Affinity Matrix in (Zamir et al., 2018). In the experiment of determining similarity among tasks, it is hard to compare directly and fairly, since each task is of different loss functions, which is usually in a very different scale with each other. To solve this problem, we take the same ordinal normalization approach as Zamir et al. (2018). Suppose we have N tasks in the pool, a tournament matrix MT for each task T is constructed, where the element of the matrix mi,j represents what percentages of adversarial examples generated from the ith task transfers better to task T than the ones of the jth task (untargeted attack success rate is used here).
Then we take the principal eigenvectors of the N tournament matrices and stack them together to build the N ×N adversarial transferability matrix. To generate the corresponding “task categories" for comparison, we sample 1000 images from the public dataset and perform a virtual adversarial attack on each of the 15 source models. Adversarial perturbation with (L∞ norm) as 0.03,0.06 are used and we run 10 steps PGD-based attack for efficiency. Then we measure these adversarial examples’ effectiveness on each of the 15 tasks by the corresponding loss functions. After we obtain the 15×15 ATM, we take columns of this matrix as features for each task and perform agglomerative clustering to obtain the Task Similarity Tree.

Review summaries:
This paper study the fundamental relationship between adversarial transferability and knowledge transferability. Theoretical analysis is conducted, revealing that adversarial transferability can indicate knowledge transferability. In this procedure, two quantities are formally defined to measure adversarial transferability from different aspects. Furthermore, empirical evaluation in three different transfer learning scenarios on diverse datasets are carried out, showing a strong positive correlation between the adversarial transferability and knowledge transferability.