Does Adversarial Transferability Indicate Knowledge Transferability?

Despite the immense success that deep neural networks ( DNNs ) have achieved , adversarial examples , which are perturbed inputs that aim to mislead DNNs to make mistakes , have recently led to great concerns . On the other hand , adversarial examples exhibit interesting phenomena , such as adversarial transferability . DNNs also exhibit knowledge transfer , which is critical to improving learning efficiency and learning in domains that lack high-quality training data . To uncover the fundamental connections between these phenomena , we investigate and give an affirmative answer to the question : does adversarial transferability indicate knowledge transferability ? We theoretically analyze the relationship between adversarial transferability and knowledge transferability , and outline easily checkable sufficient conditions that identify when adversarial transferability indicates knowledge transferability . In particular , we show that composition with an affine function is sufficient to reduce the difference between the two models when they possess high adversarial transferability . Furthermore , we provide empirical evaluation for different transfer learning scenarios on diverse datasets , showing a strong positive correlation between the adversarial transferability and knowledge transferability , thus illustrating that our theoretical insights are predictive of practice . 1 INTRODUCTION . Knowledge transferability and adversarial transferability are two fundamental properties when a learned model transfers to other domains . Knowledge transferability , also known as learning transferability , has attracted extensive studies in machine learning . Long before it was formally defined , the computer vision community has exploited it to perform important visual manipulations ( Johnson et al. , 2016 ) , such as style transfer and super-resolution , where pretrained VGG networks ( Simonyan & Zisserman , 2014 ) are utilized to encode images into semantically meaningful features . After the release of ImageNet ( Russakovsky et al. , 2015 ) , pretrained ImageNet models ( e.g. , on TensorFlow Hub or PyTorch-Hub ) has quickly become the default option for the transfer source , because of its broad coverage of visual concepts and compatibility with various visual tasks ( Huh et al. , 2016 ) . Adversarial transferability , on the other hand , is a phenomenon that adversarial examples can not only attack the model they are generated against , but also affect other models ( Goodfellow et al. , 2014 ; Papernot et al. , 2016 ) . Thus , adversarial transferability is extensively exploited to inspire black-box attacks ( Ilyas et al. , 2018 ; Liu et al. , 2016 ) . Many theoretical analyses have been conducted to establish sufficient conditions of adversarial transferability ( Demontis et al. , 2019 ; Ma et al. , 2018 ) . Knowledge transferability and adversarial transferability both reveal some nature of machine learning models and the corresponding data distributions . Particularly , the relation between these two phenomena interests us the most . We begin by showing that adversarial transferability can indicate knowledge transferability . This tie can potentially provide a similarity measure between data distributions , an identifier of important features focused by a complex model , and an affinity map between complicated tasks . Thus , we believe our results have further implications in model interpretability and verification , fairness , robust and efficient transfer learning , and etc . To the best of our knowledge , this is the first work studying the fundamental relationship between adversarial transferability and knowledge transferability both theoretically and empirically . Our main contributions are as follows . • We formally define two quantities , τ1 and τ2 , to measure adversarial transferability from different aspects , which enables in-depth understanding of adversarial transferability from a geometric point of view in the feature representation space . • We derive an upper bound for knowledge transferability with respect to adversarial transfer- ability . We rigorously depict their underlying relation and show that adversarial transferability can indicate knowledge transferability . • We conduct thorough controlled experiments for diverse knowledge transfer scenarios ( e.g . knowledge transfer among data distributions , attributes , and tasks ) on benchmark datasets including STL-10 , CIFAR-10 , CelebA , Taskonomy-data , and four language datasets . Our empirical results show strong positive correlation between adversarial and knowledge transferability , which validates our theoretical prediction . 2 RELATED WORK . Knowledge transferability has been widely applied in scenarios where the available data for certain domain is limited , and has achieved great success ( Van Opbroek et al. , 2014 ; Wurm et al. , 2019 ; Wang et al. , 2017 ; Kim & Park , 2017 ; Maqueda et al. , 2018 ; Devlin et al. , 2018 ) . Several studies have been conducted to understand the factors that affect knowledge transferability ( Yosinski et al. , 2014 ; Long et al. , 2015b ; Wang et al. , 2019 ; Xu et al. , 2019 ; Shinya et al. , 2019 ) . Empirical observations show that the correlation between learning tasks ( Achille et al. , 2019 ; Zamir et al. , 2018 ) , the similarity of model architectures , and data distribution are all correlated with different knowledge transfer effects . Adversarial Transferability has been observed by several works ( Papernot et al. , 2016 ; Goodfellow et al. , 2014 ; Joon Oh et al. , 2017 ) . Since the early work , a lot of studies have been conducted , aiming to further understand the phenomenon and design more transferable adversarial attacks . Regardless of the threat model , a lot of attack methods have been proposed to boost adversarial transferability ( Zhou et al. , 2018 ; Demontis et al. , 2019 ; Dong et al. , 2019 ; Xie et al. , 2019 ) . Naseer et al . ( 2019 ) propose to produce adversarial examples that transfer cross-domain via a generative adversarial network . In addition to the efficacy , efficiency ( Ilyas et al. , 2018 ) and practicality ( Papernot et al. , 2017 ) are also optimized . Beyond the above empirical studies , there is some work dedicated to analyzing this phenomenon , showing different conditions that may enhance adversarial transferability ( Athalye et al. , 2018 ; Tramèr et al. , 2017 ; Ma et al. , 2018 ; Demontis et al. , 2019 ) . Building upon these observations , it is clear that there exist certain connections between adversarial transferability and other knowledge transfer scenarios , and here we aim to provide the first theoretic justification to verify it and design systematic empirical studies to measure such correlation . 3 ADVERSARIAL TRANSFERABILITY VS . KNOWLEDGE TRANSFERABILITY . In this section , we establish connections between adversarial examples and knowledge transferability rigorously . We first formally state the problem studied in this section . Then , we move on to subsection 3.1 to introduce two metrics that encode information about adversarial attacks . Finally , we present our theoretical results about the relationship between adversarial and knowledge transferability in subsection 3.2 . Notations . We use blackboard bold to denote sets , e.g. , R. We use calligraphy to denote distributions , e.g. , D. The support of a distribution D is denoted as supp ( D ) . We use bold lower case letters to denote vectors , e.g. , x ∈ Rn . We use bold uppercase letter to denote a matrix , e.g. , A . We useA† to denote the Moore–Penrose inverse of matrix A . We use ◦ to denote the composition of functions , i.e. , g ◦ f ( x ) = g ( f ( x ) ) . We use ‖ · ‖2 to denote Euclidean norm induced by standard inner product 〈· , ·〉 . Given a function f , we use f ( x ) to denote its evaluated value at x , and we use f to represent this function in function space . We use 〈· , ·〉D to denote inner product induced by distribution D , i.e. , 〈f1 , f2〉D = Ex∼D〈f1 ( x ) , f2 ( x ) 〉 . Accordingly , we use ‖ · ‖D to denote a norm induced by inner product 〈· , ·〉D , i.e. , ‖f‖D = √ 〈f , f〉D . For a matrix function F : supp ( D ) → Rd×m , we define its L2 ( D ) -norm in accordance with matrix 2-norm as ‖F‖D,2 = √ Ex∼D‖F ( x ) ‖22 . We define projection operator proj ( · , r ) to project a matrix to a hyperball of spectral norm radius r , i.e. , proj ( A , r ) = { A , if ‖A‖2 ≤ r rA/‖A‖2 if ‖A‖2 > r . Setting . Assume we are given a target problem defined by data distribution x ∼ D , where x ∈ Rn , and y : Rn → Rd represent the ground truth labeling function . As a first try , a reference model fT : Rn → Rd trained on the target dataset is obtained through optimizing over a function class fT ∈ FT . Now suppose we have a source model fS : Rn → Rm pretrained on source data , and we are curious how would fS transfer to the target data D ? Knowledge transferability . Given a trainable function g : Rm → Rd , where g ∈ G is from a small function class for efficiency purpose , we care about whether fS can achieve low loss L ( · ; y , D ) , e.g. , mean squared error , after stacking with a trainable function g comparing with fT , i.e. , min g∈G L ( g ◦ fS ; y , D ) compare with L ( fT ; y , D ) . Clearly , the solution to this optimization problem depends on the choice of G. Observing that in practice it is common to stack and fine-tune a linear layer given a pretrained feature extractor , we consider the class of affine functions . Formally , the problem that is studied in our theory is stated as follows . Problem 1 . Given a reference model fT trained on target distribution D , and a source model fS pre-trained on source data . Can we predict the best possible performance of the composite function g◦fS onD , where g is from a bounded affine function class , given adversarial transferability between fS and fT ? 3.1 ADVERSARIAL TRANSFERABILITY . We use the ` 2-norm to characterize the effectiveness of an attack . Definition 1 ( Virtual Adversarial Attack ( Miyato et al. , 2018 ) ) . Given a model f : Rn → Rd , the attack on point x within -ball is defined as argmax‖δ‖≤ ‖f ( x ) − f ( x+ δ ) ‖2 . As this is intractable in practice , we consider the use of the tangent function to approximate the difference : δf , ( x ) = arg max ‖δ‖≤ ‖∇f ( x ) > δ‖2 , where ∇f ( x ) ∈ Rn×d is the Jacobian matrix . The will be dropped in clear context or when it is irrelevant . To provide a quantitative view of adversarial transferability , we define two metrics τ1 and τ2 . Both the metrics are in the range of [ 0 , 1 ] , where higher values indicate more adversarial transferability . Definition 2 ( Adversarial Transferability ( Angle ) ) . Given two function f1 , f2 , we assume they have the same input dimension , and may have different output dimensions . The Adversarial Transferability ( Angle ) of f1 and f2 at point x is defined as the squared cosine value of the angle between the two attacks , i.e. , τ1 ( x ) = 〈δf1 ( x ) , δf2 ( x ) 〉2 ‖δf1 ( x ) ‖22 · ‖δf2 ( x ) ‖22 . We denote its expected value as τ1 = Ex∼D [ τ1 ( x ) ] . Intuitively , τ1 characterizes the similarity of the two attacks . The higher the cosine similarity , the better they can be attacked together . Noting that we are suggesting to use the square of their cosine values , which means that cosine value being either 1 or−1 has the same indication of high knowledge transferability . This is because fine-tuning the last layer can rectify such difference by changing the sign of the last linear layer . However , it is not sufficient to fully characterize how good fS will perform only knowing the angle of two attack directions . For example , it is not difficult to construct two functions with highest τ1 = 1 , but not transferable with affine functions . Moreover , it is also oberserved in our experiments that only τ1 is not sufficient . Therefore , in addition to the information of attacks δf captured by τ1 , we also need information about deviation of a function given attacks . We denote the deviation of a function f , given attack δ ( x ) , as f ( x+ δ ( x ) ) − f ( x ) , and we define its approximation as ∆f , δ ( x ) = ∇f ( x ) > δ ( x ) . ( 1 ) Accordingly , we define another metric to answer the following question : applying f1 ’ s adversarial attacks on both the models , how much can the deviation of their function value be aligned by affine transformations ? Definition 3 ( Adversarial Transferability ( Deviation ) ) . Given two functions f1 , f2 with the same input dimensions and potentially different output dimensions , the Adversarial Transferability ( Deviation ) of adversarial attacks from f1 to f2 given data distribution D is defined as τf1→f22 = 〈2∆f2 , δf1 −A∆f1 , δf1 , A∆f1 , δf1 〉D ‖∆f2 , δf1‖ 2 D , whereA is a constant matrix defined as A = proj ( Ex∼D [ ∆f2 , δf1 ( x ) ∆f1 , δf1 ( x ) > ] ( Ex∼D [ ∆f1 , δf1 ( x ) ∆f1 , δf1 ( x ) > ] ) † , ‖∆f2 , δf1‖D ‖∆f1 , δf1‖D ) . We note that A is the best linear map trying to align the two deviations ( ∆f2 , δf1 and ∆f1 , δf1 ) in the function space . It serves as a guess on the best linear map to align f1 and f2 , using only the information from adversarial attacks . To have better sense of τ2 and the relationships with other quantities , we present an example for visual illustration in Figure 1 . Note that high τ2 does not necessarily require ∆f1 , δf1 and ∆f2 , δf1 to be similar , but they can be well aligned by the constant linear transformationA . We refer to the proof of Proposition 1 at section B in appendix for detailed explanation of τ2 . Proposition 1 . Both τ1 and τ2 are in [ 0 , 1 ] .

Summary: This paper study the fundamental relationship between adversarial transferability and knowledge transferability. Theoretical analysis is conducted, revealing that adversarial transferability can indicate knowledge transferability. In this procedure, two quantities are formally defined to measure adversarial transferability from different aspects. Furthermore, empirical evaluation in three different transfer learning scenarios on diverse datasets are carried out, showing a strong positive correlation between the adversarial transferability and knowledge transferability.