TRS: Transferability Reduced Ensemble via Promoting Gradient Diversity and Model Smoothness

1 Introduction . Machine learning systems , especially those based on deep neural networks ( DNNs ) , have been widely applied in numerous applications [ 27 , 18 , 46 , 10 ] . However , recent studies show that DNNs are vulnerable to adversarial examples , which are able to mislead DNNs by adding small magnitude of perturbations to the original instances [ 47 , 17 , 54 , 52 ] . Several attack strategies have been proposed so far to generate such adversarial examples in both digital and physical environments [ 36 , 32 , 51 , 53 , 15 , 28 ] . Intriguingly , though most attacks require access to the target models ( whitebox attacks ) , several studies show that adversarial examples generated against one model are able to transferably ∗The authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . attack another target model with high probability , giving rise to blackbox attacks [ 39 , 41 , 31 , 30 , 57 ] . This property of adversarial transferability poses great threat to DNNs . Some work have been conducted to understand adversarial transferability [ 48 , 33 , 12 ] . However , a rigorous theoretical analysis or explanation for transferability is still lacking in the literature . In addition , although developing robust ensemble models to limit transferability shows great potential towards practical robust learning systems , only empirical observations have been made in this line of research [ 38 , 23 , 56 ] . Can we deepen our theoretical understanding on transferability ? Can we take advantage of rigorous theoretical understanding to reduce the adversarial transferability and therefore generate robust ensemble ML models ? In this paper , we focus on these two questions . From the theoretical side , we are interested in the sufficient conditions under which the adversarial transferability can be lower bounded and upper bounded . Our theoretical arguments provides the first theoretical interpretation for the sufficient conditions of transferability . Intuitively , as illustrated in Figure 1 , we show that the commonly used gradient orthogonality ( low cosine similarity ) between learning models [ 12 ] can not directly imply low adversarial transferability ; on the other hand , orthogonal and smoothed models would limit the transferability . In particular , we prove that the gradient similarity and model smoothness are the key factors that both contribute to the adversarial transferability , and smooth models with orthogonal gradients can guarantee low transferability . Under an empirical lens , inspired by our theoretical analysis , we propose a simple yet effective approach , Transferability Reduced Smooth ( TRS ) ensemble to limit adversarial transferability between base models within an ensemble and therefore improve its robustness . In particular , we reduce the loss gradient similarity between models as well as enforce the smoothness of models to introduce global model orthogonality . We conduct extensive experiments to evaluate TRS in terms of the model robustness against different strong white-box and blackbox attacks following the robustness evaluation procedures [ 5 , 6 , 49 ] , as well as its ability to limit transferability across the base models . We compare the proposed TRS with existing state-of-the-art baseline ensemble approaches such as ADP [ 38 ] , GAL [ 23 ] , and DVERGE [ 56 ] on MNIST , CIFAR-10 , and CIFAR-100 datasets , and we show that ( 1 ) TRS achieves the state-of-the-art ensemble robustness , outperforming others by a large margin ; ( 2 ) TRS achieves efficient training ; ( 3 ) TRS effectively reduces the transferability among base models within an ensemble which indicates its robustness against whitebox and blackbox attacks ; ( 4 ) Both loss terms in TRS contribute to the ensemble robustness by constraining different sufficient conditions of adversarial transferability . Contributions . In this paper , we make the first attempt towards theoretical understanding of adversarial transferability , and provide practical approach for developing robust ML ensembles . ( 1 ) We provide a general theoretical analysis framework for adversarial transferability . We prove the lower and upper bounds of adversarial transferability . Both bounds show that the gradient similarity and model smoothness are the key factors contributing to the adversarial transferability , and smooth models with orthogonal gradients can guarantee low transferability . ( 2 ) We propose a simple yet effective approach TRS to train a robust ensemble by jointly reducing the loss gradient similarity between base models and enforcing the model smoothness . The code is publicly available2 . ( 3 ) We conduct extensive experiments to evaluate TRS in terms of model robustness under different attack settings , showing that TRS achieves the state-of-the-art ensemble robustness and outperforms other baselines by a large margin . We also conduct ablation studies to further understand the contribution of different loss terms and verify our theoretical findings . 2https : //github.com/AI-secure/Transferability-Reduced-Smooth-Ensemble Related Work The adversarial transferability between different ML models is an intriguing research direction . Papernot et al . [ 40 ] explored the limitation of adversarial examples and showed that , while some instances are more difficult to manipulate than the others , these adversarial examples usually transfer from one model to another . Demontis et al . [ 12 ] later analyzed transferability for both evasion and poisoning attacks . Tramèr et al . [ 48 ] empirically investigated the subspace of adversarial examples that enables transferability between different models : though their results provide a non-zero probability guarantee on the transferability , they did not quantify the probability of adversarial transferability . Leveraging the transferability , different blackbox attacks have been proposed [ 41 , 28 , 15 , 9 ] . To defend against these transferability based attacks , Pang et al . [ 38 ] proposed a class entropy based adaptive diversity promoting approach to enhance the ML ensemble robustness . Recently , Yang et al . [ 56 ] proposed DVERGE , a robust ensemble training approach that diversifies the non-robust features of base models via an adversarial training objective function . However , these approaches do not provide theoretical justification for adversarial transferability , and there is still room to improve the ML ensemble robustness based on in-depth understanding on the sufficient conditions of transferability . In this paper , we aim to provide a theoretical understanding of transferability , and empirically compare the proposed robust ML ensemble inspired by our theoretical analysis with existing approaches to push for a tighter empirical upper bound for the ensemble robustness . 2 Transferability of Adversarial Perturbation . In this section , we first introduce preliminaries , and then provide the upper and lower bounds of adversarial transferability by connecting adversarial transferability with different characteristics of models theoretically , which , in the next section , will allow us to explicitly minimize transferability by enforcing ( or rewarding ) certain properties of models . Notations . We consider neural networks for classification tasks . Assume there are C classes , and let X be the input space of the model with Y = { 1 , 2 , . . . , C } the set of prediction classes ( i.e. , labels ) . We model the neural network by a mapping function F : X → Y . We will study the transferability between two models F and G. For brevity , hereinafter we mainly show the derived notations for F and notations for G are similar . Let the benign data ( x , y ) follow an unknown distribution D supported on ( X , Y ) , and PX denote the marginal distribution on X . For a given input x ∈ X , the classification model F first predicts the confidence score for each label y ∈ Y , denoted as fy ( x ) . These confidence scores sum up to 1 , i.e. , ∑ y∈Y fy ( x ) = 1 , ∀x ∈ X . The model F will predicts the label with highest confidence score : F ( x ) = argmaxy∈Y fy ( x ) . For modelF , there is usually a model-dependent loss function ` F : X×Y → R+ , which is the composition of a differentiable training loss ( e.g. , cross-entropy loss ) ` and the model ’ s confidence score f ( · ) : ` F ( x , y ) : = ` ( f ( x ) , y ) , ( x , y ) ∈ ( X , Y ) . We further assume that F ( x ) = argminy∈Y ` F ( x , y ) , i.e. , the model predicts the label with minimum loss . This holds for common training losses . In this paper , by default we will focus on models that are well-trained on the benign dataset , and such models are the most commonly encountered in practice , so their robustness is paramount . This means we will focus on the low risk classifiers , which we will formally define in Section 2.1 . How should we define an adversarial attack ? For the threat model , we consider the attacker that adds an ` p norm bounded perturbation to data instance x ∈ X . In practice , there are two types of attacks , untargeted attacks and targeted attacks . The definition of adversarial transferability is slightly different under these attacks [ 33 ] , and we consider both in our analysis . Definition 1 ( Adversarial Attack ) . Given an input x ∈ X with true label y ∈ Y , F ( x ) = y . ( 1 ) An untargeted attack crafts AU ( x ) = x+ δ to maximize ` F ( x+ δ , y ) where ‖δ‖p ≤ . ( 2 ) A targeted attack with target label yt ∈ Y crafts AT ( x ) = x+ δ to minimize ` F ( x+ δ , yt ) where ‖δ‖p ≤ . In this definition , is a pre-defined attack radius that limits the power of the attacker . We may refer to { δ : ‖δ‖p ≤ } as the perturbation ball . The goal of the untargeted attack is to maximize the loss of the target model against its true label y . The goal of the targeted attack is to minimize the loss towards its adversarial target label yt . How do we formally define that an attack is effective ? Definition 2 ( ( α , F ) -Effective Attack ) . Consider a input x ∈ X with true label y ∈ Y . An attack is ( α , F ) -effective in untargeted scenario if Pr ( F ( AU ( x ) ) 6= y ) ≥ 1− α . An attack is ( α , F ) -effective in targeted scenario ( with class target yt ) if Pr ( F ( AT ( x ) ) = yt ) ≥ 1− α . This definition captures the requirement that an adversarial instance generated by an effective attack strategy is able to mislead the target classification model ( e.g . F ) with certain probability ( 1− α ) . The smaller the α is , the more effective the attack is . In practice , this implies that on a finite sample of targets , the attack success is frequent but not absolute . Note that the definition is general for both whitebox [ 1 , 12 , 5 ] and blackbox attacks [ 42 , 4 ] . 2.1 Model Characteristics . Given two modelsF and G , what are the characteristics ofF and G that have impact on transferability under a given attack strategy ? Intuitively , the more similar these two classifers are , the larger the transferability would be . However , how can we define “ similar ” and how can we rigorously connect it to transferability ? To answer these questions , we will first define the risk and empirical risk for a given model to measure its performance on benign test data . Then , as the DNNs are differentiable , we will define model similarity based on their gradients . We will then derive the lower and upper bounds of adversarial transferability based on the defined model risk and similarity measures . Definition 3 ( Risk and Empirical Risk ) . For a given model F , we let ` F be its model-dependent loss function . Its risk is defined as ηF = Pr ( F ( x ) 6= y ) ; and its empirical risk is defined as ξF = E [ ` F ( x , y ) ] . The risk represents the model ’ s error rate on benign test data , while the empirical risk is a non-negative value that also indicates the inaccuracy . For both of them , higher value means worse performance on the benign test data . The difference is that , the risk has more intuitive meaning , while the empirical risk is differentiable and is actually used during model training . Definition 4 ( Loss Gradient Similarity ) . The lower loss gradient similarity S and upper loss gradient similarity S between two differentiable loss functions ` F and ` G is defined as : S ( ` F , ` G ) = inf x∈X , y∈Y ∇x ` F ( x , y ) · ∇x ` G ( x , y ) ‖∇x ` F ( x , y ) ‖2 · ‖∇x ` G ( x , y ) ‖2 , S ( ` F , ` G ) = sup x∈X , y∈Y ∇x ` F ( x , y ) · ∇x ` G ( x , y ) ‖∇x ` F ( x , y ) ‖2 · ‖∇x ` G ( x , y ) ‖2 . The S ( ` F , ` G ) ( S ( ` F , ` G ) ) is the minimum ( maximum ) cosine similarity between the gradients of the two loss functions for an input x drawn from X with any label y ∈ Y . Besides the loss gradient similarity , in our analysis we will also show that the model smoothness is another key characteristic of ML models that affects the model transferability . Definition 5 . We call a model F β-smooth if sup x1 , x2∈X , y∈Y ‖∇x ` F ( x1 , y ) −∇x ` F ( x2 , y ) ‖2 ‖x1 − x2‖2 ≤ β . This smoothness definition is commonly used in deep learning theory and optimization literature [ 3 , 2 ] , and is also named curvature bounds in certified robustness literature [ 44 ] . It could be interpreted as the Lipschitz bound for the model ’ s loss function gradient . We remark that larger β indicates that the model is less smoother , while smaller β means the model is smoother . Particularly , when β = 0 , the model is linear in the input space X .

Summary: 1. This paper provides theoretical analysus on the sufficient condition on the adversarial transferability between ML models, and identify the importance of both gradient orthogonality and model smoothness  2. This paper proposes a simple smoothness regularization term and merge it with gradient diversity objective to form TRS robust ensemble training algorithm  3. Thorough experiments are provided to show TRS achieves state-of-the-art ensemble robustness