TRS: Transferability Reduced Ensemble via Promoting Gradient Diversity and Model Smoothness

1 Introduction . Machine learning systems , especially those based on deep neural networks ( DNNs ) , have been widely applied in numerous applications [ 27 , 18 , 46 , 10 ] . However , recent studies show that DNNs are vulnerable to adversarial examples , which are able to mislead DNNs by adding small magnitude of perturbations to the original instances [ 47 , 17 , 54 , 52 ] . Several attack strategies have been proposed so far to generate such adversarial examples in both digital and physical environments [ 36 , 32 , 51 , 53 , 15 , 28 ] . Intriguingly , though most attacks require access to the target models ( whitebox attacks ) , several studies show that adversarial examples generated against one model are able to transferably ∗The authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . attack another target model with high probability , giving rise to blackbox attacks [ 39 , 41 , 31 , 30 , 57 ] . This property of adversarial transferability poses great threat to DNNs . Some work have been conducted to understand adversarial transferability [ 48 , 33 , 12 ] . However , a rigorous theoretical analysis or explanation for transferability is still lacking in the literature . In addition , although developing robust ensemble models to limit transferability shows great potential towards practical robust learning systems , only empirical observations have been made in this line of research [ 38 , 23 , 56 ] . Can we deepen our theoretical understanding on transferability ? Can we take advantage of rigorous theoretical understanding to reduce the adversarial transferability and therefore generate robust ensemble ML models ? In this paper , we focus on these two questions . From the theoretical side , we are interested in the sufficient conditions under which the adversarial transferability can be lower bounded and upper bounded . Our theoretical arguments provides the first theoretical interpretation for the sufficient conditions of transferability . Intuitively , as illustrated in Figure 1 , we show that the commonly used gradient orthogonality ( low cosine similarity ) between learning models [ 12 ] can not directly imply low adversarial transferability ; on the other hand , orthogonal and smoothed models would limit the transferability . In particular , we prove that the gradient similarity and model smoothness are the key factors that both contribute to the adversarial transferability , and smooth models with orthogonal gradients can guarantee low transferability . Under an empirical lens , inspired by our theoretical analysis , we propose a simple yet effective approach , Transferability Reduced Smooth ( TRS ) ensemble to limit adversarial transferability between base models within an ensemble and therefore improve its robustness . In particular , we reduce the loss gradient similarity between models as well as enforce the smoothness of models to introduce global model orthogonality . We conduct extensive experiments to evaluate TRS in terms of the model robustness against different strong white-box and blackbox attacks following the robustness evaluation procedures [ 5 , 6 , 49 ] , as well as its ability to limit transferability across the base models . We compare the proposed TRS with existing state-of-the-art baseline ensemble approaches such as ADP [ 38 ] , GAL [ 23 ] , and DVERGE [ 56 ] on MNIST , CIFAR-10 , and CIFAR-100 datasets , and we show that ( 1 ) TRS achieves the state-of-the-art ensemble robustness , outperforming others by a large margin ; ( 2 ) TRS achieves efficient training ; ( 3 ) TRS effectively reduces the transferability among base models within an ensemble which indicates its robustness against whitebox and blackbox attacks ; ( 4 ) Both loss terms in TRS contribute to the ensemble robustness by constraining different sufficient conditions of adversarial transferability . Contributions . In this paper , we make the first attempt towards theoretical understanding of adversarial transferability , and provide practical approach for developing robust ML ensembles . ( 1 ) We provide a general theoretical analysis framework for adversarial transferability . We prove the lower and upper bounds of adversarial transferability . Both bounds show that the gradient similarity and model smoothness are the key factors contributing to the adversarial transferability , and smooth models with orthogonal gradients can guarantee low transferability . ( 2 ) We propose a simple yet effective approach TRS to train a robust ensemble by jointly reducing the loss gradient similarity between base models and enforcing the model smoothness . The code is publicly available2 . ( 3 ) We conduct extensive experiments to evaluate TRS in terms of model robustness under different attack settings , showing that TRS achieves the state-of-the-art ensemble robustness and outperforms other baselines by a large margin . We also conduct ablation studies to further understand the contribution of different loss terms and verify our theoretical findings . 2https : //github.com/AI-secure/Transferability-Reduced-Smooth-Ensemble Related Work The adversarial transferability between different ML models is an intriguing research direction . Papernot et al . [ 40 ] explored the limitation of adversarial examples and showed that , while some instances are more difficult to manipulate than the others , these adversarial examples usually transfer from one model to another . Demontis et al . [ 12 ] later analyzed transferability for both evasion and poisoning attacks . Tramèr et al . [ 48 ] empirically investigated the subspace of adversarial examples that enables transferability between different models : though their results provide a non-zero probability guarantee on the transferability , they did not quantify the probability of adversarial transferability . Leveraging the transferability , different blackbox attacks have been proposed [ 41 , 28 , 15 , 9 ] . To defend against these transferability based attacks , Pang et al . [ 38 ] proposed a class entropy based adaptive diversity promoting approach to enhance the ML ensemble robustness . Recently , Yang et al . [ 56 ] proposed DVERGE , a robust ensemble training approach that diversifies the non-robust features of base models via an adversarial training objective function . However , these approaches do not provide theoretical justification for adversarial transferability , and there is still room to improve the ML ensemble robustness based on in-depth understanding on the sufficient conditions of transferability . In this paper , we aim to provide a theoretical understanding of transferability , and empirically compare the proposed robust ML ensemble inspired by our theoretical analysis with existing approaches to push for a tighter empirical upper bound for the ensemble robustness . 2 Transferability of Adversarial Perturbation . In this section , we first introduce preliminaries , and then provide the upper and lower bounds of adversarial transferability by connecting adversarial transferability with different characteristics of models theoretically , which , in the next section , will allow us to explicitly minimize transferability by enforcing ( or rewarding ) certain properties of models . Notations . We consider neural networks for classification tasks . Assume there are C classes , and let X be the input space of the model with Y = { 1 , 2 , . . . , C } the set of prediction classes ( i.e. , labels ) . We model the neural network by a mapping function F : X → Y . We will study the transferability between two models F and G. For brevity , hereinafter we mainly show the derived notations for F and notations for G are similar . Let the benign data ( x , y ) follow an unknown distribution D supported on ( X , Y ) , and PX denote the marginal distribution on X . For a given input x ∈ X , the classification model F first predicts the confidence score for each label y ∈ Y , denoted as fy ( x ) . These confidence scores sum up to 1 , i.e. , ∑ y∈Y fy ( x ) = 1 , ∀x ∈ X . The model F will predicts the label with highest confidence score : F ( x ) = argmaxy∈Y fy ( x ) . For modelF , there is usually a model-dependent loss function ` F : X×Y → R+ , which is the composition of a differentiable training loss ( e.g. , cross-entropy loss ) ` and the model ’ s confidence score f ( · ) : ` F ( x , y ) : = ` ( f ( x ) , y ) , ( x , y ) ∈ ( X , Y ) . We further assume that F ( x ) = argminy∈Y ` F ( x , y ) , i.e. , the model predicts the label with minimum loss . This holds for common training losses . In this paper , by default we will focus on models that are well-trained on the benign dataset , and such models are the most commonly encountered in practice , so their robustness is paramount . This means we will focus on the low risk classifiers , which we will formally define in Section 2.1 . How should we define an adversarial attack ? For the threat model , we consider the attacker that adds an ` p norm bounded perturbation to data instance x ∈ X . In practice , there are two types of attacks , untargeted attacks and targeted attacks . The definition of adversarial transferability is slightly different under these attacks [ 33 ] , and we consider both in our analysis . Definition 1 ( Adversarial Attack ) . Given an input x ∈ X with true label y ∈ Y , F ( x ) = y . ( 1 ) An untargeted attack crafts AU ( x ) = x+ δ to maximize ` F ( x+ δ , y ) where ‖δ‖p ≤ . ( 2 ) A targeted attack with target label yt ∈ Y crafts AT ( x ) = x+ δ to minimize ` F ( x+ δ , yt ) where ‖δ‖p ≤ . In this definition , is a pre-defined attack radius that limits the power of the attacker . We may refer to { δ : ‖δ‖p ≤ } as the perturbation ball . The goal of the untargeted attack is to maximize the loss of the target model against its true label y . The goal of the targeted attack is to minimize the loss towards its adversarial target label yt . How do we formally define that an attack is effective ? Definition 2 ( ( α , F ) -Effective Attack ) . Consider a input x ∈ X with true label y ∈ Y . An attack is ( α , F ) -effective in untargeted scenario if Pr ( F ( AU ( x ) ) 6= y ) ≥ 1− α . An attack is ( α , F ) -effective in targeted scenario ( with class target yt ) if Pr ( F ( AT ( x ) ) = yt ) ≥ 1− α . This definition captures the requirement that an adversarial instance generated by an effective attack strategy is able to mislead the target classification model ( e.g . F ) with certain probability ( 1− α ) . The smaller the α is , the more effective the attack is . In practice , this implies that on a finite sample of targets , the attack success is frequent but not absolute . Note that the definition is general for both whitebox [ 1 , 12 , 5 ] and blackbox attacks [ 42 , 4 ] . 2.1 Model Characteristics . Given two modelsF and G , what are the characteristics ofF and G that have impact on transferability under a given attack strategy ? Intuitively , the more similar these two classifers are , the larger the transferability would be . However , how can we define “ similar ” and how can we rigorously connect it to transferability ? To answer these questions , we will first define the risk and empirical risk for a given model to measure its performance on benign test data . Then , as the DNNs are differentiable , we will define model similarity based on their gradients . We will then derive the lower and upper bounds of adversarial transferability based on the defined model risk and similarity measures . Definition 3 ( Risk and Empirical Risk ) . For a given model F , we let ` F be its model-dependent loss function . Its risk is defined as ηF = Pr ( F ( x ) 6= y ) ; and its empirical risk is defined as ξF = E [ ` F ( x , y ) ] . The risk represents the model ’ s error rate on benign test data , while the empirical risk is a non-negative value that also indicates the inaccuracy . For both of them , higher value means worse performance on the benign test data . The difference is that , the risk has more intuitive meaning , while the empirical risk is differentiable and is actually used during model training . Definition 4 ( Loss Gradient Similarity ) . The lower loss gradient similarity S and upper loss gradient similarity S between two differentiable loss functions ` F and ` G is defined as : S ( ` F , ` G ) = inf x∈X , y∈Y ∇x ` F ( x , y ) · ∇x ` G ( x , y ) ‖∇x ` F ( x , y ) ‖2 · ‖∇x ` G ( x , y ) ‖2 , S ( ` F , ` G ) = sup x∈X , y∈Y ∇x ` F ( x , y ) · ∇x ` G ( x , y ) ‖∇x ` F ( x , y ) ‖2 · ‖∇x ` G ( x , y ) ‖2 . The S ( ` F , ` G ) ( S ( ` F , ` G ) ) is the minimum ( maximum ) cosine similarity between the gradients of the two loss functions for an input x drawn from X with any label y ∈ Y . Besides the loss gradient similarity , in our analysis we will also show that the model smoothness is another key characteristic of ML models that affects the model transferability . Definition 5 . We call a model F β-smooth if sup x1 , x2∈X , y∈Y ‖∇x ` F ( x1 , y ) −∇x ` F ( x2 , y ) ‖2 ‖x1 − x2‖2 ≤ β . This smoothness definition is commonly used in deep learning theory and optimization literature [ 3 , 2 ] , and is also named curvature bounds in certified robustness literature [ 44 ] . It could be interpreted as the Lipschitz bound for the model ’ s loss function gradient . We remark that larger β indicates that the model is less smoother , while smaller β means the model is smoother . Particularly , when β = 0 , the model is linear in the input space X .

Summary: The manuscript describes the problem of adversarial transferability, the process of adversarial attacks that can be transferred across models, which can help scale adversarial attack strategies. As a guardrail against such attacks, it is desired that models are robust and protected against such attacks. Towards solving this, the authors first derive a lower and upper bound on a transferability metric that they define. The similarity between gradient losses was found to be the salient cross-model term affecting adversarial transferability. Subsequently, this pointed to a strategy of reducing model smoothness and loss gradient similarity across models as a way to reduce adversarial transferability.