Does Adversarial Transferability Indicate Knowledge Transferability?
Authors: Kaizhao Liang
Jacky Y. Zhang
Oluwasanmi O Koyejo
Bo Li
Venue:  (2021)
Keywords: adversarial transferability
knowledge transferability

Abstract: Despite the immense success that deep neural networks (DNNs) have achieved, \emph{adversarial examples}, which are perturbed inputs that aim to mislead DNNs to make mistakes, have recently led to great concerns. On the other hand, adversarial examples exhibit interesting phenomena, such as \emph{adversarial transferability}. DNNs also exhibit knowledge transfer, which is critical to improving learning efficiency and learning in domains that lack high-quality training data. To uncover the fundamental connections between these phenomena, we investigate and give an affirmative answer to the question: \emph{does adversarial transferability indicate knowledge transferability?} We theoretically analyze the relationship between adversarial transferability and knowledge transferability, and outline easily checkable sufficient conditions that identify when adversarial transferability indicates knowledge transferability. In particular, we show that composition with an affine function is sufficient to reduce the difference between two models when they possess high adversarial transferability. Furthermore, we provide empirical evaluation for different transfer learning scenarios on diverse datasets, showing a strong positive correlation between the adversarial transferability and knowledge transferability, thus illustrating that our theoretical insights are predictive of practice.

Under review as a conference paper at ICLR 2021
DOES ADVERSARIAL TRANSFERABILITY INDICATE KNOWLEDGE TRANSFERABILITY?
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the immense success that deep neural networks (DNNs) have achieved, adversarial examples, which are perturbed inputs that aim mislead DNNs make mistakes, have recently led to great concerns. On the other hand, adversar- ial examples exhibit interesting phenomena, such adversarial transferability. DNNs also exhibit knowledge transfer, which is critical to improving learning efficiency and learning in domains that lack high-quality training data. To uncover the fundamental connections between these phenomena, we investigate and give an affirmative answer to the question: does adversarial transferability indicate knowl- edge transferability? We theoretically analyze the relationship between adversarial transferability and knowledge transferability, and outline easily checkable suffi- cient conditions that identify when adversarial transfcrability indicates knowledge transferability. In particular, we show that composition with an affine function is sufficient to reduce the difference between the two models when they possess high adversarial transferability. Furthermore, we provide empirical evaluation for different transfer learning scenarios on diverse datasets, showing strong positive correlation between the adversarial transferability and knowledge transferability, thus illustrating that our theoretical insights are predictive of practice.
INTRODUCTION
Knowledge transferability and adversarial transferability are two fundamental properties when learned model transfers to other domains. Knowledge transferability, also known as learning trans- ferability, has attracted extensive studies in machine learning. Long before it was formally defined, the computer vision community has exploited to perform important visual manipulations (Johnson et al.,,210000 2016), such as style transfer and super-resolution, where pretrained VGG networks (Simonyan & Zisserman 2014) are utilized to encode images into semantically meaningful features. After the release of ImageNet (Russakovsky et al. 2013), pretrained ImageNet models (e.g., TensorFlow Hub or PyTorch-Hub) has quickly become the default option for the transfer source, because of its broad coverage of visual concepts and compatibility with various visual tasks (Huh al. 2016). Adversarial transfcrability, on the other hand, is a phenomenon that adversarial examples can not only attack the model they are generated against, but also affect other models (Goodfellow al. 2014; Papernot et al. 2016). Thus, ad adversarial transfcrability is extensively exploited to inspire black-box attacks (Ilyas al. 2018: Liu et al. 2016). Many theoretical analyses have been conducted to establish sufficient conditions of adversarial transferability (Demontis et al. 2019 Ma et al. 2018).
Knowledge transferability and adversarial transferability both reveal some nature of machine learning models and the corresponding data distributions. Particularly, the relation between these two phenom- ena interests us the most. We begin by showing that adversarial transferability can indicate knowledge transfcrability. This tie potentially provide similarity measure between data distributions, an identifier of important features focused by a complex model, and an affinity map between complicated tasks. Thus, we believe our results have further implications in model interpretability and verification, fairness, robust and efficient transfer learning, and etc.
To the best of our knowledge, this is the first work studying the fundamental relationship between adversarial transferability and knowledge transferability both theoretically and empirically. Our main contributions are follows.

---

Under review as a conference paper at ICLR 2021
We formally define two quantities, Ti and T2. to measure adversarial transferabilityy from different aspects, which enables in-depth understanding of adversarial transferability from a geometric point of view in the feature representation space.
We derive an upper bound for knowledge transferability with respect adversarial transfer- ability. We rigorously depict their underlying relation and show that adversarial transferabil- ity can indicate knowledge transferability.
We conduct thorough controlled experiments for diverse knowledge transfer scenarios (e.g. knowledge transfer among data distributions, attributes, and tasks) on benchmark datasets including STL.100, CIFAR-10, CelebA, Taskonomy-data and four language datasets. Our empirical results show strong positive correlation between adversarial and knowledge transferability, which validates our theoretical prediction.
2 RELATED WORK
Knowledge transferability has been widely applied in scenarios where the available data for certain
domain is limited, and has achieved great success (Van Opbroek et al. 2014 Wurm et al. 2019 Wang
itt al. 2017 7] Kim & Park [017] Maqueda et al. [018 Devlin et al. 20188) Several studies have been
conducted understand the factors that affect knowledge transferability (Yosinski et al. 2014 Long
iet al. 2015b; Wang et al. 2019: Xu et al. 2019 Shinya et al. 2019). Empirical observations show that the correlation between learning tasks (Achille al. 2019 Zamir et al., 2018), the similarity of model architectures, and data distribution are all correlated with different knowledge transfer effects.
Adversarial Transferability has been observed by several works (Papernot et al. 2016 Goodfellow iet al., (2144) Joon Oh ct al. 2017). Since the early work, a lot of studies have been conducted, aiming to further understand the phenomenon and design more transferable adversarial attacks. Regardless of the threat model, a lot of attack methods have been proposed to boost adversarial transferability (Zhou) jet al. 20188] Demontis et al., [2119] Dong et al.] 2019 Xie et al.) 2019). Naseer et al. (2019) propose to produce adversarial examples that transfer cross-domain via generative adversarial network. In addition to the efficacy, efficiency (Ilyas et al. 2018) and practicality (Papernot et al., 2017) are also optimized. Beyond the above empirical studies, there is some work dedicated to analyzing this phenomenon, showing different conditions that may enhance adversarial transferability (Athalye et al., 2018, Tramer et al. 2017 Ma et al. 2018: Demontis et al. 2019). Building upon these observations, is clear that there exist certain connections between adversarial transferability and other know ledge transfer scenarios, and here we aim to provide the first theoretic justification to verify it and design systematic empirical studies to measure such correlation.
3 ADVERSARIAL TRANSFERABILITY VS. KNOWLEDGE TRANSFERABILITY
In this section, we establish connections between adversarial examples and knowledge transferability rigorously. We first formally state the problem studied in this section. Then, we move on to subsection 3.1 to introduce two metrics that encode information about adversarial attacks. Finally, we present our theoretical results about the relationship between adversarial and knowledge transferability in subsection 3.2
Notations. We blackboard bold to denote sets, e.g., R. We use calligraphy denote distributions, e.g., D. The support of a distribution D is denoted as supp( (D). We use bold lower case letters to denote vectors, e.g., x E R", We use bold uppercase letter to denote a matrix, e.g., A. We use Al to denote the Moore-Penrose inverse of matrix A. We use to denote the composition of functions, i..., g fx) (z)))). We use || |l2 to denote Euclidean norm induced by standard inner product (..)). Given function f. we use f(x) to denote its evaluated value at æ, and we use to represent this function in function space. We use (D)) to denote inner product induced by distribution D, Accordingly, we use || . || to denote norm induced by inner product (,, i.e., |I v flD.).. For matrix function F supp(D) + Rdxm define its -.2 in accordance with matrix 2-norm as || = v EE F(x)) I.. We define projection operator proj(-,r) r) to project a matrix to a hyperball of spectral norm radius r, i.e.,
i.e. (f1, f2) = ( (x).
A,
if
||||| <r ||||2
proj( A, =
|rA/|A|/2
if
> r
2

---

Under review as a conference paper at ICLR 2021
Setting. Assume we given a target problem defined by data distribution D, where E R", and y In + R represent the ground truth labeling function. As first try, a reference model fr I + Rd trained the target dataset is obtained through optimizing over function class fo E Fr. Now suppose we have a source model fs IR + R. pretrained on source data, and we curious how would is transfer the target data DD
Knowledge transferability. Given a trainable function g IR" - Rd, where 9 E G is from a small function class for efficiency purpose, we care about whether fs can achieve low loss L(: D), e.g.. mean squared error, after stacking with a trainable function g comparing with fr, i.e.,
min geG
C(g o fsi D) compare with D,)
Clearly, the solution this optimization problem depends on the choice of G. Observing that in practice it is common to stack and fine-tune a linear layer given a pretrained feature extractor, we consider the class of affine functions. Formally, the problem that is studied in theory stated as follows.
Problem 1. Given reference model JT trained on target distribution D, and a source model fs pre-trained on source data. Can we predict the best possible performance of the composite function go fs on D, where g is from a bounded affine function class, given adversarial transferabiliity between fs and fr?
3.1 ADVERSARIAL TRANSFERABILITY
We use the lz- norm to characterize the effectiveness of an attack.
Definition 1 (Virtual Adversarial Attack (Miyato et al. 2018)). Given a model f 1R" - Wed, the
attack on point within c-ball defined as arg max s () f(x 6)ll2 As this intractable in practice we consider the use of the tangent function to approximate the difference:
(x)
arg max [[
[[)(()
where f(z) E Baxd is the Jacobian matrix. The will be dropped in clear context or when it is irrelevant.
To provide a quantitative view of adversarial transferability, we define two metrics and Tg. Both the metrics are the range of [o, 1), where higher values indicate more adversarial transferability.
Definition 2 (Adversarial Transferability (Angle)). Given two function f1, f2. we assume they have the same input dimension, and may have different output dimensions. The Adversarial Transferability (Angle) of f1 and f2 at point is defined as the squared cosine value of the angle between the two attacks, ie.,
(z)
TI
| (2) 1 (
We denote its expected value as T1 (TT
Intuitively, 7i characterizes the similarity of the two attacks. The higher the cosine similarity, the better they can be attacked together. Noting that we are suggesting use the square of their cosine values, which means that cosine value being either 1 or -1 has the same indication of high knowledge transferability. This is because fine-tuning the last layer can rectify such difference by changing the sign of the last linear layer. However, it is not sufficient to fully characterize how good fs will perform only knowing the angle of two attack directions. For example, it not difficult construct two functions with highest oberserved our experiments that only Tj is not sufficient.
1, but not transferable with affine functions. Moreover, it is also
Therefore, in addition to the information of attacks ô, captured by TI, also need information about deviation function given attacks. We denote the deviation of a function f. given attack d(x), as f( 8(x)) f(x), and we define its approximation as
4イ,((z) fz(x) 8(z).
Accordingly, we define another metric answer the following question: applying 's adversarial attacks on both the models, how much can the deviation of their function value aligned by affine transformations?
3

---

Under review as a conference paper at ICLR 2021
Definition (Adversarial Transferability (Deviation)). Given two functions fi, f2 with the same input dimensions and potentially different output dimensions, the Adversarial Transferability (Deviation) of adversarial attacks from fi to f2 given data distribution D is defined as
fi (2Ar2.6n
2
|1 11
where constant matrix defined as
||522,8, 11
A = (nnj(BeaDlAA2.bb (x) (a) n.,6 (z) (x) (x)
|| ,, 11
We note that is the best linear map trying to align the two deviations (2 and A f1,6, in the function space. It serves guess on the best linear map align fi and f2, using only the information from adversarial attacks. To have better sense of T2 and the relationships with other quantities, we present an example for visual illustration in Figure [ Note that high 72 does not necessarily require A fi .613 and Aa,5,, to be similar, but they can be well aligned by the constant linear transformation A. We refer to the proof of Proposition T at section in appendix for detailed explanation of 72-
Proposition! 1. Both Tı and T2 are in [0, 1].
3.2 ADVERSARIAL TRANSFERABILITY INDICATES KNOWLEDGE TRANSFERABILITY
In this subsection, we will provide theoret- ical results. First, to have a better intuition, we will show special case where the theorems are simplified, i.e., where fs and fr are both En R. Then, we present the general case where fs and fr are multi-dimensiona. Note that their output dimensions are not necessarily the same.
output space of Ayjay, (x)
Input space
6/2 xx)
8, (x)
(x(
output space h
vector
linear map through
When fs and fr are both R" R. the Ti and 72 come out in a surprisingly elegant form. Let us show what the two metries are to have further intuition on what Ti and T2 characterize.
Figure 1:
Illustration of the key variables.
First, let us see what the attack is in this case. As function f has one-dimensional output, its gradient
a vector f E Ra. Thus,
(ax)
(z))(z) = arg max [V(x)) T 8112 |V](2)] ||ā|
simply the gradient with its scale normalized. Then, the Tı becomes
(V/s(z), (x)))))
n (x)
f>> fr)]|
which is the squared cosine (angle) between two gradients. For T2, the matrix A degenerates to scalar constant, which makes T2 simpler as well, i.c..
A (A fr 85g
A fs. 1s.81g) )D
(A f3.01g A fr.dig 18
and
| 12
As 1B || fr.6:1 11
We see, in this 7o denote T afterwards.
is interestingly in the same form of the first metric TI. We will simply
Accordingly, when fs and fr are both IR + E. the result also comes out in an elegant form. In this case, adversarial attacks reflect all the information of the gradients of the two models, enabling 7i and T2 to encode all the information we need to prove the following theorem.
Theorem 1. For two functions fs and fr that both are TR. + R. there is an affine function g RR such that
| (go fs) EaLD (2))
4

---

Under review as a conference paper at ICLR 2021
where g(x) Ax Const. Moreover, though not necessarily, if asssmmnng that fr is L-Lipschitz continuous, (r.)]l & Lforr Va E supp(D), we have a more elegant statement:
I V fs) = -1-11-97 (1-777))2
The theorem suggests that, if adversarial transferability is high, there exists an affine transformation with bounded norm, such that g g fs is close to ST. As an intuition of the proof, the difference between two gradients can be represented by the angle between them, which can be characterized by T1; and the norm difference between them, which can be characterized by T2.
As for the general case, consider when the output dimensions of both functions multi- dimensional and not necessarily the same. In this scenario, adversarial attacks correspond to the largest singular value of the Jacobian matrix. Therefore, need to introduce the following definition to capture other information that is not revealed by adversarial attacks.
Definition 4 (Singular Value Ratio). For any function f. the Singular Value Ratio for the function o2(#),
gradient is defined as Xy (z (m)) where on (z),, 02(x)) are the largest and the second largest
singular value in absolute value of V f(x), respectively. In addition, we define the worst-case singular value ratio as X5 = maxxE=pp(P) X5 (a))
Theorem 2. For two functions fs R" - R", and fr : R² - R. assuming that fr is L-Lipschitz continuous, i.e., [r [|z S for Va supp(D), there is an affine function g 18Th 12 such that
V - Vgo fs) S ( (11 + (1- -)
+ (A5r + +>50) 5L2,
where g is defined as g(z) A= Const
We note that this theorem also has a statement offering tighter bound where we do not assume Lipschitz continuous. The full version of this theorem is provided in appendix. Theorem 2 suggests that big Tı and 72 indicate potentially small differences of gradients between the target model and the transferred model. Based on this, intuitively, given the right constant value shift, minimal difference in gradients implies minimal difference in function value, which should result in bounded loss. Indeed, we prove in Theorem 3 that the squared loss of the transferred model g o fs is bounded by the loss of fi and their gradient difference, by assuming the 8-smoothness of both the functions.
Definition 5 5 (3-smoothness)) A function f is 8-smooth for all æ. y.
S 3| yll2-
For the target data distribution 2, and its ground truth labeling function y. the mean squared loss of the transferred model is En-polg fs(x=) )()) ||g fs =11 Therefore, the following theorem presents upper bound on the mean squared loss of the transferred model.
Theorem 3. Without loss of generality we assume ||ø||2 < 1 1. Vo E supp(D). Consider functions
fs : R" R" fr R Rd, and an affine function R. Rd, suggested by Theorem 1 or Theorem with the constant set to let o(fs(0)) = (o). If both fr. fs are B-smooth, then
fsllD llg fs vllo < (1 (1 V voo + (1 (1+ |T
[ 3
3.3 PRACTICAL MEASUREMENT OF ADVERSARIAL TRANSFERABILITY
Existing studies have shown that similar models share high adversarial transferability (Liu et al. 2016 Papernot et al. 2016, Tramer et al. 2017). In previous work, it is common to use cross adversarial loss as an indication of adversarial transferability, e.g., the loss of fr with attacks generated on fs. It is intuitive to consider that the higher cross adversarial loss, the higher adversarial transfcrability. However, may have a drawback comparing to the TI, T2 defined in this work.
Definition (Cross Adversarial Loss). Given a loss function Cr(. y) on the target domain, where y is ground truth, the adversarial loss of fr with attack 8 fs generated against source model fs is
Lado 5 53 y, D)
EEDD
Cr(Sr(x + 81s (x)), v(x)).
5

---

Under review as a conference paper at ICLR 2021
The cross adversarial loss depends on the choice of loss function, the output dimension, etc. Thus, it can be incomparable when we want to test adversarial transferability among different fr, unlike that , T2 always between [0. 1]. To investigate the relationship between the adversarial loss and the adversarial transferability we defined, we show in the following proposition that the cross adversarial loss is similar to 9I- In the next section, verify the theoretical predictions through thorough experiments.
Proposition 2. If cy is mean squared loss and fr achieves zero loss on DD then the adversarial loss defined in Definition iss approximately upper and lower bounded by
Lado fsi D)= e² BEEDD
+ +
Ladolfr, 9. S 2² B
[ + (1- (z) +0()).
where OEe²) denotes a cubic error term.
4 EXPERIMENTAL EVALUATION
The empirical evaluation of the relationship between adversarial transferability and knowledge transferability is done by four different sets of experiment. First we present a set of synthetic experiment that verifies world datasets with models widely used in practice, described three knowledge transfer scenarios: knowledge transfer data distributions, attributes, and tasks. Details regarding the three scenarios are claborated below, and all training details are deferred to the Appendix.
theoretical study, and then present empirical study on real-
bhowledge-transteen among data distributions is the most common setting of transfer learning. It transfers the knowledge of a model trained/gained from one data domain to the other data domains. For instance, [Shie et al.] (2015) manage to use pre-trained ImageNet representations to achieve saute-of-the-art accuracy for medical data analysis. The relation between adversarial and knowledge transferability can not only determine the best pretrained models to use, but also detect distribution shifts, which is crucial in learning agents deployed in continual setting (Diethe et al. 2019).
Knowledge-trand seaeddge-rranfer among attributes is a popular method to handle zero-shot and few-shot learn- ing (Jayaraman & Grauman 2014 Haamera-Paredes & Torr 2015). It transfers the knowledge learned from the attributes of the source problem to new target problem Russakovsky & Fei-Fei 2010))
The relation between adversarial and knowledge transferability can be used probe deployed classification models to verify attributes that their decisions are based on. This will have profound implications on fairness and interpretability.
Knowledge-transter among tasks is widely applied across various vision tasks, such as super resolution (Johnson al. ) style transfer (Gatys et al.) 2016),, semantic and instance segmenta- tion Girshick 2015 He et al. 2017| [Long et 20155). It involves transferring the knowledge the model gains by learning to do one task to another novel task. The relation between adversarial and
knowledge transferability,, as many recent works (Achille et al. [019:; Standley et al., 2019 Zamir ct al. 2018), can be used to charting the affinity map between tasks, aiming to guide potential transfer.
4.1 SYNTHETIC EXPERIMENT ON RADIAL BASIS FUNCTIONS REGRESSION
In the synthetic experiment, we compute quantities that are otherwise inefficient to compute to verify our theoretical results. We also try different settings to see how other factors affect the results. Details follow.
Models. Both the source model Is and the target model fr are one-hidden-layer neural networks with sigmoid activation.
First, sample D = {(.,, y from a distribution (details later), where x is n--imensional, y is d-dimensional, and there are N samples. Then we train a target model fr on D. Denoting the weights of fT W, we randomly sample direction V where each entry of V is sampled from U-0.5,, 0.5), and choose a scale t E [o, 1]. To derive the source model, we perturb the target model as W : W tV. Define the source model fs to be ane-hidden-layer neural network with weights W Then, we compute each of the quantitics we care about, including TI, T2. cross adversarial loss (Definition 6) the upper bound in theorem 2 on the difference of gradients, etc.
Overall Steps.
6

---

Under review as a conference paper at ICLR 2021
Noting that reported the cross adversarial loss normalized by its adversarial loss, defined as a A fr.de .tg /l A fr. l z Lade f y. D)/ 6 y, D) when fr achieves low error. Note that a E [0, 1]. Finally, we fine-tune the last layer of fs, and get the true transferred loss.
Dataset. Denote radial basis function (x)=)) e w...ee...112///0 (2a,) and we set the target ground truth function to be the sum of M 100 basis functions as Ei Φi. where each entry of the parameters are sampled once from ( 0.5). We set the dimension of æ to be 30, and the 1200 samples of from a Gaussian mixture formed by three Gaussian with different centers but the same covariance matrix Σ = I. The centers are sampled randomly from (.5,, 0.5)". We use the ground truth regressor f to derive the corresponding y for each T. That is, we want our neural networks to approximate on the Gaussian mixture.
dimension of y be 10. We generate
Results. We present two sets of experiment in Figure The correlations between adversarial transferabilities (T1, 72, a) and the knowledge transferability (transferred loss) are observed. The upper bound for the difference gradients (Theorem 2 basically tracks its true value. Although absolute value of the upper bound on the transferred Toss (Theorem 3 can be big compared to the true transferred loss, their trends are similar. We note the big difference in absolute value is due to the of B B-smmothsssss, which considers the worst case scenario. It is also observed that tracks the normalized adversarial cross loss (t as Proposition suggests.
(a) hidden hidden-layer layer- width 50
(b) hidden-layer-widtee 100
Figure 2: The x-axis is the E [o, 1] that controls how much the source model deviates from the target model. There in total 7 quantities reported. placed under 4 y-axes. Specifically, T1, 72, and the normalized cross adversarial loss are plotted as green curves with green y-axis; the upper bound theorem on the transferred gradients difference shown blue curves with blue y-axis; the true transferred gradients difference is shown as red curves with red y-axis; the upper bound in theorem on the transferred loss is shown as magenta curves with magenta y-axis; the true transferred loss shown as black curves with black y-axis.
4.2 ADVERSARIAL TRANSFERABILITY INDICATES KNOWLEDGE-TRANSTER AMONG DATA DISTRIBUTIONS
knowledge transferability
Data
Yelp
AG
Fake
loss
T1
0.49
0.47
0.50
I 1.34e-3
8e-4
3e-6
adv loss
0.16
0.13
0.12
Knowledge-trans
0.89
0.66
0.52
Figure 3: (left) correlation between the adversarial transferability and knowledge trransferability in image domain (All values normalized to [0,1]). (right) adversarial transferability and knowledge transferability NLP domain.
7

---

Under review as a conference paper at ICLR 2021
In this experiment, we show that the closer the source data distribution is to the target data distribution, the more adversarially transferable the source model to the reference model, thus we observe that the source model is more knowledge transferable to the target dataset. We demonstrate this on both image and natural language domains.
Dataset. Image: 5 source datasets (5 source models) constructed based on CIFAR- 10 (Hinton et al. 2012) and a single target dataset (1 reference model) based on STL-10 (Coates et al. [2011) Each of the source datasets consists of classes from CIFAR 10 and the target dataset also consists of
4 classes from STL10. Natural Language: We select 4 diverse natural language datasets: AG's s News
(AG), Fake News Detection (Fake), IMDB, Yelp Polarity (Yelp). Then we pick IMDB as the target
and the rest sources.
We take 1000 images (STL-10) from the target dataset and generate 1000 adversarial examples on each of the five source models. We run 10 step PGD I. attack with e = 0.1. Then we measure the effectiveness of the adversarial examples by the cross-cntropy loss on the reference model. Natural Language: We take 100 sample sentences from target (ILD)) and generate adversarial sentences on each of the source models(AG, Fake, Yelp) with TextFoolersin et al., (219).. The ratio of changed words is constrained less equal to 0.1. Then, we measure their adversarial transferability against the reference modeI(IMDB).
Adversarial Transferability. Image:
Knowledge Transferability. To measure the knowledge transferability, we fine-tune a new linear layer on the target dataset to replace the last layer of each source model to generate the corresponding transferred models. Then we measure the performance of the transferred models on the target dataset based on the standard accuracy and cross-entropy loss.
Results From Figure [.2] it's clear that if the source models that has highest adversarial transferability, its corresponding transferred model achieves the highest transferred accuracy. This phenomenon is prominent both image and natural language domains. The results in Figure 4.2 (b) could verify the implication by our theory that only is not sufficient for indicating knowledge transferability.
Data
Young
Male
Attractive
Eyebrows
Lipstick
T1
0.0707
0.0679
0.0612
0,0609
0.0678
T2
0.0759
0.0521
0.0418
0.0529
0.0388
adv loss
17.83
16.21
14.13
13.22
12.54
knowledge-Ho
0.593
0.589
0.562
0.551
0.554
Table Top 5 Attributes with the highest adversarial transferability and their corresponding average accuracy on the validation benchmarks.
4.3 ADVERSARIAL TRANSFERABILITY INDICATING KNOWLEDGE-TRANSEE AMONG ATTRIBUTES
In addition to the data distributions, we validate our theory on another dimension, attributes. This experiment suggests that the more adversarially transferable the source model of certain attributes is to the reference model, the better the model performs on the target task aiming to learn target attributes.
Dataset CelebA (Liu et al. 2018) consists of 202,599 face images from 10, 10,177 identities. A reference facial recognition model is trained on this identities. Each image also comes with 40 binary attributes, on which we train 40 source models. Our goal is to test whether source models of source altributes, can transfer perform facial recognition.
A Adversarial Transferability We sample 1000 images from CelebA and perform a virtual adversarial attack as described in section on each of the 40 attribute classifiers. Then we measure the adversarial transfer effectiveness of these adversarial examples on the reference facial recognition model. Knowledge Transferability To fairly assess the snoweedee transferability, we test the 40 transferred models on 7 well-known facial recognition benchmarks, LFW (Huang et al. (007)) CFP-FF CFP-FP (S. Senguptal 2016), AgeDB (Moschoglou et al., 2017). CALFW, CPLFW (Zheng et al.) 2017 and VGG2-IP (Cao et al. 2018). We report the average classification accuracy target datasets.
Result In Table J we list the top-5 attribute source models that share the highest adversarial transferability and the performance of their transferred models the 7 target facial recognition benchmarks. We observe that the attribute "Young" has the highest adversarial transferability; as a result, it also achieves highest classification average performance across the 7 benchmarks.
8

---

Under review as a conference paper at ICLR 2021
4.4
ADVERSARIAL TRANSFERABILIT INDICATING KNOWLEDGE AMONG TASKS
Curvature
Edje-DD
Segmennttiion-0 20
Fijje-DD 3D
Roo huda
10. Surface No mo!
14. Segmentation semantic
2
3
12
33
14
10
7
9
11
14
13
11
10
8
13 Deject Candriation
Figure 4: Left: Emprically confirmed taskonomy prediction of task categories (Zamir et al. 2018). Right: Task category prediction based on adversarial transferability. Different colors represent different task categories including 2D, 3D, Semantic. is obvious that the adversarial transferability able to predict similar task categories aligned with the pure knowledge-trance transfer empirical observation.
In this experiment, show that adversarial transferability can also indicate the knolwdge trans- ferability among different machine learning tasks. Zamir et al.] (2018) shows that models trained on different tasks can transfer other tasks well, especially when the tasks belong to the same "category". Here we leverage the same dataset, and pick 15 single image tasks from the task pool, including Autoencoding, 2D Segmentation, 3D Keypoint and etc. Intuitively, these tasks can be categorized into 3 categories, semantic task, 2D tasks as well as 3D tasks. Leveraging the tasks within the same category, which would hypothetically have higher adversarial transferability, we evaluate the corresponding knowledge transferability.
Dataset The Taskonomy-data consists of million images of indoor scenes from about 600 indoor images, every one of which has annotations for every task listed in the pool. We use a public subset of these images to validate our theory.
Adversarial Transferability Adversarial Transferability Matrix (ATM) is used here to measure the adversarial transferability between multiple tasks, modified from the Affinity Matrix (Zamir et al., 2018). To generate the corresponding "task categories" for comparison, from the public dataset and perform a virtual adversarial attack on each of the 15 source models. Adversarial perturbation with (Lee norm) as 0.03,0.06 are used and we run 10 steps PGD-based attack for efficiency. Detailed settings about adversarial transferability are deferred to the Appendix. Knowledge Transferability We use the affinity scores provided a 15 x 15 affinity matrix to compute the categories of tasks. Then we take columns of this matrix as features for each task and perform agglomerative clustering obtain the Task Similarity Tree.
sample TO00 images
Results Figure 4 compares the predictions of task categories generated based on adversarial trans- ferability and knowledge transferability in Taskonomy. formed, i.e. 2D, 3D, and Semantic tasks for both adversarial and knowledge transferability. To provide a quantitative measurement of the similarity, we also compute the average inner category entropy based adversarial transferability with the categories in Taskonomy the ground truth (the lower entropy indicates higher correlation between adversarial and knowledge transferability). In figure 5| (Appendix), the adversarial transferability based category prediction shows low entropy when the number of categories is greater or equal to 3, which indicates that the adversarial tranferability faithful with the category prediction in Taskonomy. This result shows strong positive correlation between the adversarial transfcrability and knowledge transferability among learning tasks in terms of predicting the similar task categories.
easy to see three intuitive categories
5 CONCLUSION
We theoretically analyze the relationship between adversarial transferability and knowledge transfer- ability, along with thorough experimental justifications in diverse scenarios, Both our theoretical and empirical results show that adversarial transferability can indicate knowledge transferability, which reveal important properties of machine learning models. We hope discovery inspire and facilitate further investigations, including model interpretability, faimess, robust and efficient transfer learning, and etc.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In 2019.
Proceedings of the IEEE International Conference on Computer Vision, pp.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pp. 274-283, 2018.
Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman. Vggface2: A dataset for recognising faces across pose and age. In International Conference Automatic Face and Gesture Recognition, 2018.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223, 2011.
Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita- Mota-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. 28th {USENIX) Security Symposium ( (!USENIX) Security 19), pp. 121-338,20 2019.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 4690-4699, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0488 04805, 2018.
Tom Diethe, Tom Borchert, Eno Thereska, Borja Balle, and Neil Lawrence. Continual learning in practice. arXiv preprint vrXiv:1903.05200 2019.
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by munuslationinvariann attacks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4312-4321, 2019.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2414-2423, 2016.
Ross Girshick. Fast r-cnn. Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint /2X7v:/207.05800 2012.
Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst, October 2007.
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? arXiv preprint arXiv: 1608.000161111 2016.
10

---

Under review as a conference paper at ICLR 2021
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International Conference on Machine Learning, pp. 2137-2146, 2018.
Dinesh Jayaraman and Kristen Grauman. Zero-shot recognition with unreliable attributes. In Advances in neural information processing systems, pp. 3464-3472 2014.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. bert really robust? natural language attack text classification and entailment. arXiv preprini arXiv: 1932, 2, 2019.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Seong Joon Oh, Mario Fritz, and Bernt Schiele. Adversarial image perturbation for privacy protection- game theory perspective. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1482-1491, 2017.
Jiman Kim and Chanjong Park. End-to-end ego lane estimation based sequential transfer learning for self-driving cars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 30-38, 2017.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks, arXiv preprint arXiv:1611.02770 2016.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15:2018, 2018.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015a.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. International Conference on Machine Learning, pp. 97-105, 2015b. 5b.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song. Michael Houle, and James Bailey. Characterizing adversarial subspaces using local 0.06112 2018.
intrinsic dimensionality. arXiv preprint
Ana Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso García, and Davide Scaramuzza. Event-based vision meets deep learning on steering prediction for self-driving cars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5419-5427, 2018.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 1979 2018
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop, volume 2, pp. 2017.
Muhammad Muzammal Naseer, Salman H Khan, Muhammad Haris Khan, Fahad Shahbaz Khan, and Fatih Porikli. Cross-domain transferability of adversarial perturbations. In Advances in Neural Information Processing Systems, pp. 12885-12895, 2019.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black- box attacks using adversarial samples. arXiv preprint arXiv: arXiv:1605.0727 1605.07277, 2016.
11

---

Under review as a conference paper at ICLR 2021
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference computer and communications security, pp- 506-519, 2017.
Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learning. In International Conference Machine Learning, pp. 2152-21666 2015.
Olga Russakovsky and Li Fei-Fei. Attribute learning in large-scale datasets. In European Conference on Computer Vision, pp. 1-14. Springer, 2010.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-222 2015.
C.D. Castillo V.M. Patel R. Chellappa D.W. Jacobs S. Sengupta, J.C. Cheng. Frontal to profile face verification in the wild. In IEEE Conference on Applications of Computer Vision, February 2016.
Chuen-Kai Shie, Chung-Hisame Chuang, Chun-Nan Chou, Meng-Hsi Wu, and Edward Y Chang. Transfer representation learning for medical image analysis. In 2015 37th anmual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 711-714. IEEE, 2015.
Yosuke Shinya, Edgar Simo-Serra, and Taiji Suzuki. Understanding the effects of pre-training for object detectors via eigenspectrum. In Proceedings of the IEEE International Conference on Computer Vision Workshops. pp. 0-0, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1eXXiv:400915555 2014.
Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? arXiv preprint arXiv: 1eXXiv/:05.000 /00.. 00.505533 2019.
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv: 1704.03477 2017.
Annegreet Van Opbroek, M Arfan Ikram, Meike W Vernooij, and Marleen De Bruijne. Transfer learning improves supervised image segmentation across imaging protocols. IEEE transactions on medical imaging, 34(5):1018 1030- 2014.
Weiyue Wang, Naiyan Wang, Xiaomin Wu, Suya You, and Uirich Neumann. Self-paced cross- modality transfer learning for efficient road segmentation. 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1394-1401. IEEE, 2017.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the ICEE Conference Computer Vision and Pattern Recognition, pp. 11293-11302, 2019.
Michael Wurm, Thomas Stark, Xiao Xiang Zhu, Matthias Weigand, and Hannes Taubenböcc. Semantic segmentation of slums satellite images using transfer learning on fully convolutional neural networks. ISPRS journal of photogrammetry and remote sensing. 150:59 69, 2019.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2730-2739 2019.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive fea- ture norm approach for unsupervised domain adaptation. In Proceedings of the IEEE International Conference Computer Vision, pp. 1426-1453 1435, 2019.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320-3328, 2014.
12

---

Under review as a conference paper at ICLR 2021
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3712-3722, 2018.
Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age LFW: A database for studying cross-age face recognition in unconstrained environments. CoRR, abs/1 hts//708.08199 2017. URL http:
// //txiv.org/abb//T00 08197
Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transferable adversarial perturbations. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 452-467, 2018.
13

---

Under review as a conference paper at ICLR 2021
A DISCUSSION ABOUT VALIDNESS OF THE NOTATIONS
Before starting proving our theory. it is necessary to show that our mathematical tools are indeed valid. It is easy to verify that (:, 1) is valid inner product inherited form standard Euclidean inner product. Therefore, the norm || I/. induced by the inner product, is also a valid norm.
What does not come directly is the validness of the norm || || Particularly, whether it satisfies the triangle inequality. Recall that, for function of matrix output F supp(D) Rdxm its I,3 (nn)))oo in accordance with matrix 2-norm defined as
F D.2
V E F(x))13
For two functions F, G. both supp(D) > Rd we verify the norm || 11,, satisfies triangle inequality as shown in the following. Applying the triangle inequality the spectral norm, and with some algebra manipulation, it holds that
|F + G.IDD V FFLDDIF())+GGG
BEND H
VIFF1 | || + 2D F (
Applying the Cauchy-Schwarz ineuuysayhwaazz/nquuulttyyy
inequality, we can see that
Ez> Ee--D|/((()la](((())) V / /Bo-DlIF(=)1 Eae
| G
Plugging this into 2 would complete the proof, i.e.,
S /F + |G||2,2 + 2 2
< 2,]
= ||Flp222 + ||]|
B PROOF OF PROPOSITION 1
Proposition (Restated). Both Tı and T2 are in [0, 1).
Proof. We are prove that Ti and 72 are both the range of [0, 1]. As 7i is squared cosine, is trivial that E [0, 1]. Therefore, we will focus on in the following.
Recall that the 72 from fi fa defined as
f- (2A fa.6, AA AA,,AA,,.o AA,,8,
| 11
where is a constant matrix defined as = proj(Emm/AApaa 82 (z) (x: (Bm-olAn.,, (z) (x) (m)
| 12,20 1I Ar 81 I
For notation convenience, will simply use T2 denote 7/ f11 in this proof.
characterizes how similar are the changes in both the function values of fi R + R and f2 _dth d in the sense of linear transformable, given attack generated on f1. That is being said, it is associated to the function below, i.e,
h B) = | - BA 8, 1 = EEnDD AlA (x) - BAJ, (2))
14

---

Under review as a conference paper at ICLR 2021
where 11,5, E R. A2.5% E R and B E R
As A5.55 (x) BA n (a0)l1² is convex with respect to B, its expectation, i.e. h(B), is also
convex.
Therefore, h(B) it achieves global minima when ah JB = 0.
0h ② (|As2,5, (x) 1. (x)1l()
0B
0B
2R [[ [BA f (x) Aeion (a:)) A 11:51 (ar)
2ENDD [B 51 (z) 51.6% (a:) A 52,5%1 (x) (z) 51 (x)) [
2BEEnD [A (x) 11 (0t) [] 2E A2 12.8,1 (z) (2)) J.
Letting 0h 8B = 0, and denoting the solution as B", we have
B EEDD fz,ōn (a: 61 (x: (EaND [ An.o, (x) (a)) 7T'
Noting that A proj( B* 1, 1e
scaled B", we denote A = BB where & a scaling factor
|
| 12 | x
depending on B and that 0 + S 1.
According to the definition of the projection operator, can see
Replacing B by A we have,
h(A)= = A2,88 AA AA 51 1. = (Ajzr -AA 5 Aeio - AA AA f1,8/
= |218/ 1,3 - (2D 1215;, ADI,, AAs1,5,,0 10
= (1 -2) A 11
is obvious that h ((A) = A 12,6 AA 11 2 0, thus we have T2 S 1.
As for the lower bound for T2, we will need to use properties of B. Denoting O as an all-zero zero matrix, it holds that
h(B²) min(B())) < OO)).
For B', according to the convexity of h(.) and the fact that vb E [0, 1], we can see the following, i.e., ((A) h(bB") hhbb'((B--- + 0) s vL(B") (1-1))(()).
Applying the above, we can see that
h(A) s <hO)).
Noting that I(A) (1 T2) | 12 1 and l(O) A fo I| the above inequality suggests that
(1 - T2) 12
11
A f2.5,, f
D
0 T2-
Therefore, 72 is upper bounded by 1 and lower bounded by 0.
C PROOF OF THEOREM
Before actually proving the theorem, let have look what T3 and 72 in the where fs and fr are both R - IR. In this case, both Ti and Ta come out in an elegant form. Let us show what the metrics are to have further intuition what and 72 characterize.
15

---

Under review as a conference paper at ICLR 2021
First, let what the attack is in this case. As function f has one-dimensional output, its gradient a vector Vf E R". Thus,
<V)(()
(5,(())
arg max [V)
8||2 [Vf(z)]
1
Then, the becomes
(v)s V/r(2))2
Ti (œ)
[ Vfr(2)
which is the squared cosine (angle) between two gradients. For T2, the matrix A degenerates to scalar constant
( 5
1s )D
A
55 D
and the second metric becomes
(A 81 A
fr. 11198
| S5.65s
1 A 1b
We can see, it is interestingly in the same form of the first metrie Tı- We will simply use to denote
fs-trr afterwards.
T2
Theorem (Restated). For two functions fs and fr that both are R R, there is affine R + R, so that
function
IV (g o fs) EEED (x=-m())
where g is defined as g(z) = Ax + Const.
Moreover, assuming that fr is L-Lipschitz continuous, i.e., |fr(x)) I L Va e supp(D), we can have a more elegant statement:
IV V fs) S (1-
Proof. In the case where is one-dimensional affine function, write is as g(2) Az + b, where A is defined in the definition of 72 (Definition 3). In this case, it enjoys a simple form of
A Sn A50001)
A
AA 1b
Then, we can see that
V V o fs) V AV
E ( -VV
(4)
To continue, split frr two terms, i.e., one the direction Vss and orthogonal to Vss-
Denoting (fxr) as the angle between V (x)) and Vss( in Euclidean space, have
V ff(w))ll (x) Vfr (()()))coo(((()))
V ff
(x) cos()
fs(x))
V fs(x)]l2
Vfs(z)]l
V fr(())la
cos()(x))
[fs(=)] fz() + v
(5)
where we denote v(x) = fr(()) cos (
(r(()) fs()) for notation convenience.
[
We can see that (œ) is orthogonal to fs))), thus |ex /1 - -os (r)
Recall that actually Ti (æ:) cos?(((())) it can be written ||0( -
16

---

Under review as a conference paper at ICLR 2021
Then, plugging into
we have
(fr 2 fz( + v(xx AV fz(()))
4
EoND
cos ( x
[fs(x))l2
1 fr (a))l
E
cos(6(2)
A
(z5( + =((x))
< fr (2)
= -
cos( (x))
V (x)
+ +110(x))12
[f)(2))l
I [V/(())
=EgDDD
cos( (o)))
V Is (az)
+ - +
2
[
E
cos( (x))
A
fs (a:
+ 10-0011-1160 [
(coo(o(1z)
f (2)
2
-EED
A
[Vfs(>)]12
+ ELwDD(1 - -Bo-D(1-((=)11V//(((
[Vs(())
(6)
Now let us deal with the first term by plugging in
{ An.Bi 15.813
A
1 1B
where 1r. (x) = c =ccos(((=)))l]//((()) and A As.dss (a:) <s/Vss( ||,, and we have
ffr())l2 a) 2
EyDD
cos(
[[V
Vfs(2) 12
EEDD (cos()(z)))
Vfs(x)))
2 BEWD
A Arr66s (x) - 13 (a))
A (x)² A A
(x) (2A)-2AA, 5ig (x) (x) (2))
(|)
2
|Ammi
12
-2A fr. sr
A A5.11ggD)
||
2
(A (mmrdyss
15,818
(A (Arr A 1s
15.8101 12
2
D
|
1
fz. 1b
1
2
(A fr Sig
13.5 12
A f3.d,5
1B
Arr,8;0 1D)
(1 T2) )2) ERND (x))(()) //
(1 72 B [
Plugging 7 into 6 we finally have
|v
(gg f
(1 T2 EODD (n (z)) (x)
EE TI
(1))
Eg [11 - T2T (x)) (z))
$ (1 - LP
which completes the proof.
17

---

Under review as a conference paper at ICLR 2021
D PROOF OF THEOREM 2
Theorem (Restated), For two functions fs :R" - IRm, and fr RR - RR, there is an affine function g R Rdd, that
+ (11--11))))--01((a))))11)))-)111
fV Vigo fs)ll S 5LDDD
[ VV
tys
where g defined as g(z) Az + Const.
Moreover, if assuming that JT is I--Lipschitz continuous, i.e., [T S T. for Va supp(D), and considering the worst-case singular value ratio X, we can have a more elegant statement:
VV - Vg fs)ll ( (17-) + (1 (11-1172)
7) -2) + (Asr +>10) )) 5L2.
Proof. Recall that the matrix A is defined in Definition 3 i.e.
| 5 112 A f5.8,8 11
FE A 13 (x)) (o) (2))))
A (E
( 5r 1s: (a))
and we can see
||V V fs) = IVV (g fs) 11 = IVV AV 13 li..2
-E V(r(2) AV
/v)(2> T t - AV (s(r) 1 = Em max ||||a=1
(8)
where the last equality is due to the definition of matrix spectral norm.
Denoting f as either the Jacobian matrix f or Vfg. Singular Value Decomposition (SVD) suggests that Vf(=) = VΣVT, where Σ is a diagonal matrix containing all singular values ordered by their absolute values. Let 01, , denote ordered singular values. Nothing that the number of singular values that are non-zero may be less than so we fill the empty with zeros, such that each of them have corresponding singular vectors, i.e., the column vectors 01, ... Uy in V. That is being said, Vi [n]. we have
Let 02 and v, denote the singular values and vectors for fs())) Noting that {w)= define orthonormal basis for I", we can represent
0,01
where Σ " 8 1
As adversarial attack is about the largest eigenvalue of the gradient, plugging 9 into (8) we can split into two parts, i.e.,
E E
max ||d| 2=1
V
(
Σ 0,00
AV
Σ
0,00
1=1
1=1
V fx)) T (8101)
AVis(x)) (01m))
EE EE max
(10)
||t| 2=1
+ V fx)) T
Σ
0:0;
As
0,0;
Denoting u Σm= 0,0, vector with the biggest absolute singular value of (x((x) parallel with attack d Now split
this vector is orthogonal V1. Let us denote of the singular
18

---

Under review as a conference paper at ICLR 2021
W1 U2 into two terms, where u1 is parallel v1, and uz is orthogonal to u1. As u1 is in the orthogonal space to 21 while parallel with vf. it is bounded by the sine value of the angle between v1 and v/,, i.c., /1 - Ti (x). Hence, noting that u is part of the unit vector t,
|
< v1 (x))
(11)
Plugging u in 10
we have
V AVfs(())) (0/v))
10
E B
2=
+ V (u) + u2) u2) AV/s(a)) u
( (0101) AV (0101))
X1
< EEND
max |+]|==1
(12)
ffr(2) u1l1 + Vfr(v) T U2 - (s())))) u
X2
X8
where the inequality due to triangle inequality.
There are three terms we have to deal with. i.c., X1, X2, and X3. Regarding the first term, in X aligns with the attack 8 fz (a:). which we have known through adversarial attack. The second term X₂ is trivially bounded by [11.. Although adversarial attacks tell us nothing about X3, it can be bounded by the second largest singular values.
Let us first deal with two easiest, i.e., X2 and X3. Applying 11 on X2 directly, we have
X₂ [T) T 1|2 ||211/22 / Ti (a) / /r 112
For X3, noting that u2 orthogonal to v/, and u is orthogonal to vi, we can see that u2 has no components of the largest absolute singular vector of V Vr())) T, and has components of the largest absolute singular vector of V frr(a) Therefore,
X3 < V
22 2 + AV fs(()) ull2
S ()))((
||
+ (
|| l1 |2122
(x)
V frr(z) 1|2 U2lla
As (x)
V f=(
1l || |2ll2
S Ay (x)
fr((x) T12 Ays [=)
|| ||Al
where the first inequality is due triangle inequality, the second inequity is done by the attributes of singular values, and the definition of matrix 2-norm. The equality done simply by applying the definition of singular values ratio (Definition 4 and the third inequality is due to the fact that ||uzl|2 I ||ul|22 I
Before dealing with X,, let us simplify 12 by relax the square of summed terms to of squared terms, as the following.
12
ENLD max (Xi 2=
X2 X2)
E max 2=
X + X² + X3 + 2X + 2X2x,, + 23
EgDD max <
x² + X² + x² + max x2) 2mxx (X2, X3) + 2 max mmxx(X?. (X)xx3)
||t|==1
<
EEDD max X² + x3 + x3 + 2(X? + +X3) +2 - x3) + 2(x + X3)
2=
EgAD
max 5(X? x + x
(13)
||t|
We note that this relaxation is not necessary, but simply for the simplicity of the final results without breaking what theory suggests.
19

---

Under review as a conference paper at ICLR 2021
Bring what we have about X2 and X3, and noting that 01 S 1 depends t, drop the max operation by
13
E
max 2/2=1
5(x] + x2 + x3)
E D
max 5( V /T (z) (0101) AVSs(m)) (0,01)ll²+ + X3 + x3) + (2)
|
V|/r(ax) 1 - (V T 4t
S 5BBDD
(14)
+ ((Arr (a) + ) (a) 11 [141))
Now, let us deal with the first term. As 01 is a unit vector and fact the direction of fs(s)'s
adversarial attack, can write fs (x) e0i. Hence,
EnnD
V
01 AV vi
Em-D
1
V [T(x))
( (
AVSs(a)) (
2
1
A (x) - AA (2z) (<))
BanD 2
(15)
where the last equality is derived by applying the definition of A(x), i.e., equation Note that
omit the in 6 fs.r for notation simplicity.
The matrix A is deigned to minimize 15).. shown in the proof of Proposition term we have
Expanding the
1 EAND
||
(c0) A 1s (1))l sr.88. (x)), AA f0.8,a (a:)
15
2
1 (|Arrodrs 1| ||AADssssss 11--200 -2(A AD AA
2
|
A
(1 (1-22 T2)
=(1 T2 BONDD
V (
To
(16)
Recall that V1 is unit vector aligns the direction of 8 and we have used of to denote unit vector that aligns the direction of fz. As 7i tells about the angle between the two, let split in into to orthogonal vectors, i.e., 11 = (a)o + v1 (z)u,, where vi, a unit vector that is orthogonal to vi
Plugging this into 16 have
(fr(z)((VT((j))) + VI- ( )
16
(1 - T2) E
[f) (z) | [f( V1 ( 27
(1 BaND
(1 Ba)DD
[1(=)||
(1- (z)² (.)((>) 112]
where the second equality is due to the image of of and o' after linear transformation V (x) orthogonal, which can be easily observed through SVD.
20

---

Under review as a conference paper at ICLR 2021
Plugging this in (14 and with some regular algebra manipulation, finally we have
V/T(z) T|2 + (1 12) (1(2))
(1- [/)(<>))))))))))))))))>)>))))))) [1[]]
14
DD
- Vrr
(1) + A/2 2 )² V
|
(1 - ( (x) 1
5B
+(1 - (
(17)
+ (1), (g [z)
|
Recall that A is from norm-restricted matrix space, i.e., the A is scaled so that its spectral norm is no greater than //A ll
thus
A 5r.55= f3
| 1T
A <
1
|A5s, 113
5g
BeDD
(x)
E E
(x)
E
As.0,1
(x)l1
E E
VJ (+)
I 1l.22
(18)
V S 1l.22
Hence, plugging the above inequality to
17 the first statement of the theorem is proven, i.e.,
(1 - n (x) V/r(2> 2
+(1 - (1) > | fr
17
5EDDD
(19)
V fr 1
+ (As, (x) + AsS (z)) (=)
1
V
f 1B.2
To the second statement of the theorem, assume fr is L-Lipschitz continuous, i.e., fr & for Vo E supp(D), and considering the worst-case singular value ratio maxp))))(D) for either fs. fr, we can continue
(2) (x) [1]]
E
E E
fV [1 - Ti (x)) -2)
|
19
S 5
V fT 11:.2
E L
(A + )1g) (ss(
| 11..2
E
- ( V )
= 5
+ E E
(1-7(z)) 72) | |/ff(=
(Asr +\a3)² ||V 1b.2
5E
(1- (1))) (1 7I (x)) T2) + (Azz )15)²) (x)
< E (1 (1-71(x:)1) + (1-7(x)) 12) (2pr + > 5I2
(11 T(72) + +1 (1- m) + (A + Aya )²) 5L2,
where the first inequality is due the definition of worst-case singular value ratio, the last inequality is by Lipschitz condition, and the last equality is done be simply applying the definition of 71.
21

---

Under review as a conference paper at ICLR 2021
E PROOF OF THEOREM 3
The idea for proving Theorem 3 is straight-forward: bounded gradients difference implies bounded function difference, and then bounded function difference implies bounded loss difference.
To begin with, let us prove the following lemma.
Lemma 1. Without loss of generality we assume ||æ||2 S 1 for Va E supp(D). Consider functions
fs 1nn Rm fr IN 12d, and an affine function g : IR.m + -R, suggested by Theorem or
Theorem such that (O)) = fr(0), if both fr, fs are 3-smooth in {0r ||>||| I 1}. we have
V fr Fr
1r go
V Vig fs)
(1+
B.
V D.2
Proof. Let denote v(x) fr(() - g fs(x), and we can show the smoothness of v(-).
As g(-) )) is an affine function satisfying (())
(0), it can be denoted as g(z) A( fs(0)) (O), where A a matrix suggested by Theorem Dor Theorem 2 Therefore, denoting B1 { |||||| S } as umit ball, for Vz, y E B it holds that
V (o) Vu(y)l2 Vv(x)] Vo(y) 112
V (xx) V/r(y) A(VSs(m)T -V[ss))
S V (xx) V/r(v) ||A[s((()) - Vfs(y)
V (xx V.r + ||All2 Vfs(2)) vfs(y) 112 2
(20)
where the last second inequality is due to triangle inequality, and the last inequality is by the property of spectral norm.
Applying the 3-smoothness of fs and fr. and noting that || S can continue as
V as shown (18). [illlo
| p.2 3 || y112
20
S ||:
pall-yl2-00 + || 1| 2ll2 S 3
|| 1/2
[ D.2
Vfrll
3 $1100 - 2112
/r|1m which suggests that v(-) is (1+ [Vss|l
3-smooth.
We are ready to prove the lemma now. Applying the mean value theorem, for Va B1, have u(x) v0) Vu(fx) x,
where & (0,1) 1) is a scalar number. Subtracting Vu(x) T x on both sides give
v(x) v(0) Ve(x)) x = (Vu(Ew) - Vu(x)) Tx [o(z)) v(0) Vu(x) x 2 = [[Vo(E Vu(x:) T=(1)))<==11 T0112 |((())) v(0) Vu( Tx x S [((V((g)) Vo(x))]l |
Let us denote 31 = |rrl [ for notation convenience, and apply the definition smooth-
[
ness:
||u(z) - v(0 - Vo(x) 2112 < f1(1 | < 31.
(21)
Noting that (0) = 0 and applying the triangle inequality, we have
[[(x)) v0) V=(=))< x 2 ||o())]12 [ æ112 2 ||(())]22 [[Vx l2
Plugging it into 21 we have
[|u(=)) < V 1|2
(((=))]/33 0 [Vv((>) T 23 || (r(x)) ll2
B
[[= /87 Pa>D//Vu(a) T 1 + 23 Em= 1
E Ex [1] < 68 + EE>D Vo(=)] Ea
||0||
< 887 |
D.2 + 2, (
22

---

Under review as a conference paper at ICLR 2021
Applying Jensen's 's inequality to the last term, get
|el13 $ 8 + | 1r. + 2/31 V
L + | p. + /V +V + 23:
(l/Voll,, + +1)12
Plugging 31 proof.
//rlpa L + [V
8 and fr g fs into the above inequality completes the
With the above lemma, it is easy to show the mean squared loss on the transferred model is also bounded.
Theorem 3 (Restated). Without loss of generality we assume ||0||2 S 1 for Va E supp(D). Consider functions [s R" Thm Ji II - Rd, and an affine function g IR Rdd, suggested by such that g(Ss(0)) = (r) If both Jr, Is are 3-smooth, then
Theorem or Theorem
11g fs vllo < - + V
(1+
Vfrlp22
3
2
V |
Proof. Let us denote 31 = [1
Vir 2. B, and according to Lemma we can see
[V
|r - go fs & | - Vig o fs) + 31
(22)
Applying a standard algebra manipulation to the left hand side, and then applying triangle inequality, we have
11 - go ssllp = || y ーーシー g ssllo 2 ly go fsllo || ull
Plugging this directly into 22 holds that
|yy go fs fsll llfr vllo < Vr Vgg fs)llp.2 81
|| go fsllD S [1 ull V fr (g fs)llD.2 + B1
Replacing B1 by
[1 [1+
(rilna)) (
8 and taking the square, we can see Theorem is proven.
F PROOF OF PROPOSITION
Proposition (Restated). lT is mean squared loss and fT achieves zero loss on D, then the adversarial loss defined in Definition 6 approximately upper and lower bounded by
Lade 8 133 D) 2 E
T1(2) [V 2 + +
Ladu (TT f333 D) < 2 E
(A7, + - (x)) [V + +((),,
where ) denotes a cubic error term.
Proof. Recall that the empirical adversarial transferability is defined as loss
Loav (fr, òss.e: D) EEND lx(fr( + ( v(x)).
As lr is mean squared loss, and fT achieves zero loss, i.e., fr y, we have
Ladolfr, 8 U, D) EE |l j (x)) y(x))1 2
= EZaD [| je (x)) - fr
Denoting (x)) f0 (x), and define an auxiliary function
h(t) fr( 10 (x)) fr(
23

---

Under review as a conference paper at ICLR 2021
that [] We can then apply Taylor expansion to approximate h(()) with second order error term O( i...
(s))(=))) fr(2))11
llh(c)]l
0h 0t
h(t)
t= =Vfr(<)))))))) ò5s, + O(22)
Therefore, assuming that |
|| (z) l| is bounded for æ E supp(D), we have
fx + و5 (x)) ( 1| |M())12 ( (s. 0((),
(23)
where we have omit higher order error term, i.e., O(c*).
Next, let us deal with the term Vr (,,,1) Same us the technique we use in the proof of Theorem 2 we split f8,1 (x) V1 + 122, where aligns the direction of fr.1 (ær), and is orthogonal to ø1 Noting that 71 (x) is the squared cosine of the angle between d fs. (x) and fr. (æ). we can see that
|| _ (z) |6 1 (x)))
|/ 2 = (1 - T (2)) 11 12)1 (1- (xr))
Therefore, continue
[r(()) fs. (ax) (3) (.) T (m) + (2)+
(V T [ 02 = <o (z) 22 12112
(24)
where the second equality is because that vn is corresponding to the largest singular value of frr()),, and U2 orthogonal V1.
Next, we derive the lower bound and upper bound for
24 The lower bounded can be derived as
T (z) (x))2 || [x)
02
n (x)
fr
and the upper bounded can derived
n [+\ + [v)((=))) T 22 s (2: V fr (z) 11 (x) 1 /v) 12 | 12
(x) fr 2 ( Vz
(1-7
S (x)) /V
+ X / / (1 (1-7(x))
(>) + (1- (z)) | xz) ||
where Xyn (x) is the singular value ratio of fr at æ. and A, is the maximal singular value of fr.
Applying the lower and upper bound to 22 we finally have
(x + (x)) fr (m) || ( +
[1,r(( + (z))((x)) - fr)2)ll < 2 (x, + (1 - (z)) +
(25)
Noting that
Laaulf 8 D) = EEED [| (x)) fr(x)
we can see that taking expectation
25 completes the proof.
G EXPERIMENT DETAILS
All experiments conducted on 4 RTX 2080 Ti GPUs and in python3 Ubuntu 16.04 environment.
24

---

Under review as a conference paper at ICLR 2021
G.1
ATTACK METHODS
PGD Attack is generated iteratively: denote step size as &, the source model as fs, and the loss function on the source problem. ()(--). We initialize zo be uniformly sampled from the e-ball (x) of radius e centered as instance a, and then generate the adversarial instance iteratively: at step I we compute 2 æt + sigu V V Ls(S (w:), f=))))) Denoting the adversarial example at instance æ using PGD on source model fs as PGDis (2), we measure the adversarial loss from fs to fr based on the loss Er(-2) of fr on target data DD given attacks generated on fs, i.e.
Cr(JT poi D)
E (T(Ir(PG); (x)), v()))
ToxtFooler iteratively replaces words in target sentences by looking up similar words in the dictionary. pauses when the predicted label is changed or runs out of the attack budget. We modify such that it pauses when the percentage of changed words reaches 10%.
G.2 ADVERSARIAL TRANSFERABILITY INDICATES KNOWLEDGE- AMONG DATA DISTRIBUTIONS
Details of Dataset construction For the image domain, we divide the classes of the original datasets into two categories, animals (bird, cat, deer, dog) and transportation vehicles (airplane, automobile, ship, truck). Each of the source datasets consists different a percentage of animals and transportation vehicles, while the target dataset contains only transportation vehicles, which is meant to control the closeness of the two data distributions. Detaíls of Model Training Image: animals to 100% animals, and one reference models on STL-10 with identical architectures and hyperparameters. We use SGD optimizer and standard cross-entropy loss with learning rate 0. momentum 0.9, and weight decay 10-4, Each model is trained for 300 epochs.
train five source models the five source datasets from 0%
Natural Language: we fine-tune a Bert on each of the datasets with Adam and learning rate 0.0003 for 100 epochs. For transferred models, we run Adam with a smaller learning rate 0.0001 for 3 epochs.
G.3 ADVERSARIAL TRANSFERABILIT INDICATING KNOWLEDGE- AMONG ATTRIBUTES
Details of Model Training We train 40 binary source classifiers each of the 40 attributes of CelebA with ResNer18 (He et al. 2016). All the classifiers are trained with optimizer Adadelta with a learning rate of 1.0 for 14 epochs. We also train facial recognition model model on CelebA with 10,177 identities using ResNet18 8 as the controlled experiment The reference facial recognition model is optimized with SGD and initial learning rate 0.1 on the ArcFace (Deng et al. 2019) with focal loss (Lin et al. 2017) for 125 epochs. For each source model, we construct a transferred model by stripping off the last layers and attaching a facial recognition head without parameters. Then we use the 40 transferred models to evaluate the knowledge transferability on 7 facial recognition benchmarks.
reference
G.4 ADVERSARIAL TRANSFERABILITY INDICATING KNOWLEDGE- TRANDSFERRRAAEEERRRR AMONG TASKS
Details of Model Training We use 15 pretrained models released in the task bank (Zamir al.) 2018) as the source models. Each source model consists of two parts, an encoder, and a decoder. The encoder is a modified ResNet50 without pooling, homogeneous across all tasks, whereas the decoder is customized to suit the output of each task. When measuring the adversarial transferability, we will use each source model as a reference model and compute the transferability matrix as described below.
Adversarial Transferability Matrix (ATM) is used here to measure the adversarial transferability between multiple tasks, modified from the Affinity Matrix in (Zaiir et al. 20188). In the experiment of determining similarity among tasks, it is hard to compare directly and fairly, since each task is of different loss functions, which is usually in a very different scale with each other. To solve this problem, we take the same ordinal normalization approach as Zamir et al. (2018). Suppose we have N tasks in the pool, tournament matrix My for each task T is constructed, where the element of the matrix m,j represents what percentages of adversarial examples generated from the ith task transfers better to task T than the ones of the jth task (untargeted attack success rate is used here).
25

---

Under review as a conference paper at ICLR 2021
Then we take the principal eigenvectors of the N tournament matrices and stack them together to build the N N adversarial transferability matrix. To generate the corresponding "task categories" for comparison, we sample 1000 images from the public dataset and perform a virtual adversarial attack on each of the 15 source models. Adversarial perturbation with e (1 norm) as 0.03,0.06 used and examples' effectiveness on each of the 15 tasks by the corresponding loss functions. After we obtain the 15 15 ATM, we take columns of this matrix as features for each task and perform agglomerative clustering to obtain the Task Similarity Tree.
10 steps PGID-based attack for efficiency. Then measure these adversarial
Average inner category entropy with different granularty
0.5
0.4
entropy
0.3
0.1
0.0
number of categories
Figure 5: We also quantitatively compare our prediction with the Taskonomy (Zamir al. 2018 prediction when different number of categories is enforced. We find our prediction is similar with theirs with n 2 3.
26


Reviews:
[{"review_id": "gCwkKYAc9hl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\n \nThis paper demonstrates the relationship between adversarial transferability and knowledge transferability. The authors first formulate these two phenomena. Then under mild assumptions, they prove adversarial transferability indicates knowledge transferability. The authors further justify their theory with experiments.\n\n##########################################################################\nReasons for score: \n\nOverall, I believe this paper studies an interesting problem. The relationship between adversarial transferability and knowledge transferability is of interest to both transfer learning and adversarial attack community. However, the paper does not provide how to further interpret their findings our apply them in practice. \n\n##########################################################################\nPros\n\nThe theoretical analysis seems sound and clearly organized. The authors provide a comprehensive analysis of their hypothesis both theoretically and empirically.\n\n##########################################################################\nCons\n\n1.\tAlthough the problem studied is interesting, the results are not surprising. Adversarial transferability indicates the gradients of the two models are similar, which further indicates similar parameters and predictions. Models with similar parameters and predictions will naturally share knowledge. Even without reading the paper, one can expect the argument above will hold. \n2.\tThe authors did not elaborate on how to use their results in applications. For example, since adversarial transferability and knowledge transferability are related, will adversarial training help transfer learning? On the other hand, will transfer learning improve adversarial robustness? \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gm0TSMtcevn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the relationship between adversarial and knowledge transferability. It develops two metrics to measure adversarial transferability and empirically shows that adversarial transferability correlates with knowledge transferability. \n\nPros: \nThe paper studies an interesting problem. Intuitively, both adversarial and knowledge transferability show how much the decision boundaries of the source and target classifiers are aligned. This paper is interesting in that it tries to quantify their correlation.  \n\nCons:\nThe paper provides an interesting observation but fails to further investigate how this observation can be used to gain better understanding of either fields of adversarial examples or knowledge transfer or how the insights can lead to better techniques. In the following, I provide a few comments and questions.  \n\nOn metrics tau1 and tau2:\n-\tWhy do we need \\tau1 and \\tau2 as proxies for adversarial transferability? It seems from experimental results that the simple metric of adversarial loss can indeed be a simpler and more robust indicator of the knowledge transferability.  \n-\t\\tau1 measures the correlation of the perturbations for the two models. The attack method used finds the perturbation that causes the largest change in output, which is the formulation for the misclassification attack. Different models may classify the perturbed example into different classes and so the output changes would be different. The paper says \\tau1 is not enough and we need to look into the changes made to the output as well and whether an affine layer can map one into another. I feel if paper considered targeted attack, the attack could generate highly correlated output changes and so \\tau2 wouldn’t be necessary. I wondered if authors tried targeted attack formulation and how the new tau1 would correlate with adversarial transferability. \n-\tI’m a bit confused about Theorem 1 and how it relates to definition of \\tau2. As far as I understand, \\tau2 captures how much outputs of two models can be aligned using an affine transformation. Theorem 1 then says large \\tau2 means an affine function g can be found that aligns the two models. Isn’t it just the way that \\tau2 is defined?\n\nAnalysis and results:\n-\tSome parts of the paper seem a bit disconnected or not elaborated well enough. For example, what do theorems 2 and 3 tell us? Similarly for proposition 2. What are the takeaways?\n-\tDo experiments confirm the theoretical bounds and derivations? It might be good to define a simple synthetic task (maybe on a logistic regression or 1-layer neural net) to evaluate the claims and show the relationship between adversarial and knowledge transferability. \n-\tThe results need more investigation. For example, in figure 2 (right), \\tau1 values are almost identical, which as paper mentions, shows \\tau1 is not sufficient for determining transferability. However, although \\tau2 values are correlated with transferability, they are almost zero. What does this imply? Do these results statistically support the claims? Similarly, in table 1, the relationship between \\tau1 and \\tau2 and adversarial-knowledge transfer does not seem to be statistically significant. \n-\tSome statements in observations seem obvious. For example, section 5.1 reads “we show that the closer the source data distribution is to the target data distribution, the more adversarially transferable the source model to the reference model, thus we observe that the source model is more knowledge transferable to the target dataset.” This is a known concept that has been used in both designing better attacks and also better transfer learning methods. \n\n\nPaper edit suggestions:\n- Writing and paper structure can be polished. I suggest moving background info and definitions into a section background and then laying out the structure for the rest of the paper. \n- The discussion of related work and the cited papers need a revision. Some papers are repeatedly cited while others seem to be discussed out of place. For example, page 8 refers to (Zamir et al., 2018) a total of nine times. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GaUkYV8YaL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary and contributions\nThis paper studies the relationship between adversarial transferability and knowledge transferability. By defining two quantities to measure the adversarial transferability, it shows that adversarial transferability measured in this way indicates knowledge transferability both theoretically and empirically.\n\nStrengths\nThis paper is the first work to theoretically focus on the correlation of two prevalent phenomena of DNN--adversarial transferability and knowledge transferability. Its backgrounds and theoretical results and proofs are clearly presented. Moreover, the experiments are complete: they include experiments for three types of knowledge transferability whose datasets, transfering methods and results are very clear. \n\nWeaknesses\nFirst, this paper does not provide the intuition to use squared cosine value rather than the cosine value in the defination of $\\tau_1$(although Proposition 2 theoretically shows the relationship between $\\tau_1$ and Cross Adversarial Loss). Fig.1 in the paper also regards $\\tau_1$ as the cosine value. For an example of the ill-defined parts of $\\tau_1$, given $\\bm \\delta_{f_1}=\\bm \\delta_{f_2}=-\\bm \\delta_{f_3}$, one would claim that the adversarial transferabilities of $f_1$ to $f_2$ and $f_1$ to $f_3$ are same considering their same squared cosine values, which is counterintuitive to make $\\tau_1$ an appropriate quantity to measure the similarity between two attacks. \n\nSecond, in the definition of $\\tau_2$, the authors provide the linear map $A$ and $\\tau_2$ without literally stating their physical meanings. Besides, the demonstration of $\\tau_2$ in Fig.1 is misleading since it seems that $\\tau_2$ is a vector in Fig.1 which is not true. The authors also do not clarify the reasonability of comparing vectors of different dimensions which is necessary for most settings of this paper.\n\nThird, this paper does not provide an indicator with both $\\tau_1$ and $\\tau_2$ for high adversarial transferability. Futhermore, it lacks the theoretical relation between $\\tau_2$ and Cross Adversarial Loss to demonstrate its reasonability. \n\nIf these concerns could be solved appropriately, I would consider to raise the score. \n\nCorrectness\nI have carefully checked the proofs of all theorems in this paper, and ensure that they are correct.\n\nReproducibility\nYes. The setting of their experiments is clear and complete. \n\nClarity\nThe idea, structure and expression of this paper are easy to understand and follow. However, the explanation of Definition 3 is hard to understand. \n\nRelation to the prior work\nYes. Authors show they have a good understanding of prior work's contributions, especially the three types of knowledge transferability. \n\nAddition\nPro. Authors use PGD-attack adversarial transferring in Section 5.1 but virtual adversarial transferring in Section 5.2. Is there any difference between them, and if so, what's the difference?  \nPro. Authors claim that $g$ is a trainable function in Section 3 while use a specified $g$ instead in Theorem 3. Does a better $g$, such as $g = \\arg \\min \\left\\|g \\circ f_{S}-y\\right\\|_{\\mathcal{D}}^{2}$, make the bound in Theorem 3 tighter? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "4LWHPqj-iYT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper study the fundamental relationship between adversarial transferability and knowledge transferability. Theoretical analysis is conducted, revealing that adversarial transferability can indicate knowledge transferability. In this procedure, two quantities are formally defined to measure adversarial transferability from different aspects. Furthermore, empirical evaluation in three different transfer learning scenarios on diverse datasets are carried out, showing a strong positive correlation between the adversarial transferability and knowledge transferability.\n\n\n##########################################################################\n\nReasons for score:\n\nStrengths:\n\n- This work study the relationship between adversarial transferability and knowledge transferability, which is still under explored.\n\n- Both adversarial transferability and knowledge transferability are defined quantitatively, which enables in-depth understanding of them.\n\nWeaknesses:\n\n- The novelty and contribution of this work are marginal. Although the problem is fundamental, this paper does not seem to be an adequate exploration of the relationship between adversarial transferability and knowledge transferability. The proof of a conclusion, which seems intuitive, may only bring us limited inspiration.\n\n- The three experiments repeatedly prove the positive correlation between the adversarial transferability and knowledge transferability in three knowledge transfer scenarios, which seems to be repetitive. More ingenious experiments can be designed and conducted.\n\n- The organization of this paper is somewhat confusing，e.g. Section 4.\n\n##########################################################################\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the relationship between adversarial transferability and knowledge transferability. It develops two metrics to measure adversarial transferability and a theoretical framework to justify the positive correlation between adversarial transferability and knowledge transferability. Synthetic experiments show that adversarial transferability measured by the proposed metrics indicates knowledge transferability.

While the paper studies an interesting and fundamental problem, with a sound theoretical analysis and a clear presentation, reviewers still have several reservations to directly accept it.
- Lack of interpretation. How this observation can be used to gain better understanding of either fields of adversarial examples or knowledge transfer?
- Lack of inspiration. How the insights can lead to better transfer techniques, apply to practical applications, and foster future research?
- Lack of justification. Why such definitions of metrics are the intrinsic ways of measuring adversarial transferability? How well do they correlate with the practical experience with advanced attack, defense, and transfer methods?

AC believes the endeavor made by this paper towards a fundamental problem is highly necessary to our field. But given the above reservations, AC would encourage the authors to further strengthen their work to make it more inspiring and useful.
Decision: Reject