TRS: Transferability Reduced Ensemble via Promoting Gradient Diversity and Model Smoothness
Year: 2021

Abstract:
Adversarial Transferability is an intriguing property – adversarial perturbation crafted against one model is also effective against another model, while these models are from different model families or training processes. To better protect ML systems against adversarial attacks, several questions are raised: what are the sufficient conditions for adversarial transferability and how to bound it? Is there a way to reduce the adversarial transferability in order to improve the robustness of an ensemble ML model? To answer these questions, in this work we first theoretically analyze and outline sufficient conditions for adversarial transferability between models; then propose a practical algorithm to reduce the transferability between base models within an ensemble to improve its robustness. Our theoretical analysis shows that only promoting the orthogonality between gradients of base models is not enough to ensure low transferability; in the meantime, the model smoothness is an important factor to control the transferability. We also provide the lower and upper bounds of adversarial transferability under certain conditions. Inspired by our theoretical analysis, we propose an effective Transferability Reduced Smooth (TRS) ensemble training strategy to train a robust ensemble with low transferability by enforcing both gradient orthogonality and model smoothness between base models. We conduct extensive experiments on TRS and compare with 6 state-of-the-art ensemble baselines against 8 whitebox attacks on different datasets, demonstrating that the proposed TRS outperforms all baselines significantly.

heading: 1 Introduction
text: Machine learning systems, especially those based on deep neural networks (DNNs), have been widely applied in numerous applications [27, 18, 46, 10]. However, recent studies show that DNNs are vulnerable to adversarial examples, which are able to mislead DNNs by adding small magnitude of perturbations to the original instances [47, 17, 54, 52]. Several attack strategies have been proposed so far to generate such adversarial examples in both digital and physical environments [36, 32, 51, 53, 15, 28]. Intriguingly, though most attacks require access to the target models (whitebox attacks), several studies show that adversarial examples generated against one model are able to transferably
∗The authors contributed equally.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
attack another target model with high probability, giving rise to blackbox attacks [39, 41, 31, 30, 57]. This property of adversarial transferability poses great threat to DNNs.
Some work have been conducted to understand adversarial transferability [48, 33, 12]. However, a rigorous theoretical analysis or explanation for transferability is still lacking in the literature. In addition, although developing robust ensemble models to limit transferability shows great potential towards practical robust learning systems, only empirical observations have been made in this line of research [38, 23, 56]. Can we deepen our theoretical understanding on transferability? Can we take advantage of rigorous theoretical understanding to reduce the adversarial transferability and therefore generate robust ensemble ML models?
In this paper, we focus on these two questions. From the theoretical side, we are interested in the sufficient conditions under which the adversarial transferability can be lower bounded and upper bounded. Our theoretical arguments provides the first theoretical interpretation for the sufficient conditions of transferability. Intuitively, as illustrated in Figure 1, we show that the commonly used gradient orthogonality (low cosine similarity) between learning models [12] cannot directly imply low adversarial transferability; on the other hand, orthogonal and smoothed models would limit the transferability. In particular, we prove that the gradient similarity and model smoothness are the key factors that both contribute to the adversarial transferability, and smooth models with orthogonal gradients can guarantee low transferability.
Under an empirical lens, inspired by our theoretical analysis, we propose a simple yet effective approach, Transferability Reduced Smooth (TRS) ensemble to limit adversarial transferability between base models
within an ensemble and therefore improve its robustness. In particular, we reduce the loss gradient similarity between models as well as enforce the smoothness of models to introduce global model orthogonality.
We conduct extensive experiments to evaluate TRS in terms of the model robustness against different strong white-box and blackbox attacks following the robustness evaluation procedures [5, 6, 49], as well as its ability to limit transferability across the base models. We compare the proposed TRS with existing state-of-the-art baseline ensemble approaches such as ADP [38], GAL [23], and DVERGE [56] on MNIST, CIFAR-10, and CIFAR-100 datasets, and we show that (1) TRS achieves the state-of-the-art ensemble robustness, outperforming others by a large margin; (2) TRS achieves efficient training; (3) TRS effectively reduces the transferability among base models within an ensemble which indicates its robustness against whitebox and blackbox attacks; (4) Both loss terms in TRS contribute to the ensemble robustness by constraining different sufficient conditions of adversarial transferability.
Contributions. In this paper, we make the first attempt towards theoretical understanding of adversarial transferability, and provide practical approach for developing robust ML ensembles. (1) We provide a general theoretical analysis framework for adversarial transferability. We prove
the lower and upper bounds of adversarial transferability. Both bounds show that the gradient similarity and model smoothness are the key factors contributing to the adversarial transferability, and smooth models with orthogonal gradients can guarantee low transferability.
(2) We propose a simple yet effective approach TRS to train a robust ensemble by jointly reducing the loss gradient similarity between base models and enforcing the model smoothness. The code is publicly available2.
(3) We conduct extensive experiments to evaluate TRS in terms of model robustness under different attack settings, showing that TRS achieves the state-of-the-art ensemble robustness and outperforms other baselines by a large margin. We also conduct ablation studies to further understand the contribution of different loss terms and verify our theoretical findings.
2https://github.com/AI-secure/Transferability-Reduced-Smooth-Ensemble
Related Work
The adversarial transferability between different ML models is an intriguing research direction. Papernot et al. [40] explored the limitation of adversarial examples and showed that, while some instances are more difficult to manipulate than the others, these adversarial examples usually transfer from one model to another. Demontis et al. [12] later analyzed transferability for both evasion and poisoning attacks. Tramèr et al. [48] empirically investigated the subspace of adversarial examples that enables transferability between different models: though their results provide a non-zero probability guarantee on the transferability, they did not quantify the probability of adversarial transferability.
Leveraging the transferability, different blackbox attacks have been proposed [41, 28, 15, 9]. To defend against these transferability based attacks, Pang et al. [38] proposed a class entropy based adaptive diversity promoting approach to enhance the ML ensemble robustness. Recently, Yang et al. [56] proposed DVERGE, a robust ensemble training approach that diversifies the non-robust features of base models via an adversarial training objective function. However, these approaches do not provide theoretical justification for adversarial transferability, and there is still room to improve the ML ensemble robustness based on in-depth understanding on the sufficient conditions of transferability. In this paper, we aim to provide a theoretical understanding of transferability, and empirically compare the proposed robust ML ensemble inspired by our theoretical analysis with existing approaches to push for a tighter empirical upper bound for the ensemble robustness.

heading: 2 Transferability of Adversarial Perturbation
text: In this section, we first introduce preliminaries, and then provide the upper and lower bounds of adversarial transferability by connecting adversarial transferability with different characteristics of models theoretically, which, in the next section, will allow us to explicitly minimize transferability by enforcing (or rewarding) certain properties of models.
Notations. We consider neural networks for classification tasks. Assume there are C classes, and let X be the input space of the model with Y = {1, 2, . . . , C} the set of prediction classes (i.e., labels). We model the neural network by a mapping function F : X → Y . We will study the transferability between two models F and G. For brevity, hereinafter we mainly show the derived notations for F and notations for G are similar. Let the benign data (x, y) follow an unknown distribution D supported on (X ,Y), and PX denote the marginal distribution on X . For a given input x ∈ X , the classification model F first predicts the confidence score for each label y ∈ Y , denoted as fy(x). These confidence scores sum up to 1, i.e., ∑ y∈Y fy(x) = 1,∀x ∈ X . The model F will predicts the label with highest confidence score: F(x) = argmaxy∈Y fy(x). For modelF , there is usually a model-dependent loss function `F : X×Y → R+, which is the composition of a differentiable training loss (e.g., cross-entropy loss) ` and the model’s confidence score f(·): `F (x, y) := `(f(x), y), (x, y) ∈ (X ,Y). We further assume that F(x) = argminy∈Y `F (x, y), i.e., the model predicts the label with minimum loss. This holds for common training losses.
In this paper, by default we will focus on models that are well-trained on the benign dataset, and such models are the most commonly encountered in practice, so their robustness is paramount. This means we will focus on the low risk classifiers, which we will formally define in Section 2.1.
How should we define an adversarial attack? For the threat model, we consider the attacker that adds an `p norm bounded perturbation to data instance x ∈ X . In practice, there are two types of attacks, untargeted attacks and targeted attacks. The definition of adversarial transferability is slightly different under these attacks [33], and we consider both in our analysis. Definition 1 (Adversarial Attack). Given an input x ∈ X with true label y ∈ Y , F(x) = y. (1) An untargeted attack crafts AU (x) = x+ δ to maximize `F (x+ δ, y) where ‖δ‖p ≤ . (2) A targeted attack with target label yt ∈ Y crafts AT (x) = x+ δ to minimize `F (x+ δ, yt) where ‖δ‖p ≤ .
In this definition, is a pre-defined attack radius that limits the power of the attacker. We may refer to {δ : ‖δ‖p ≤ } as the perturbation ball. The goal of the untargeted attack is to maximize the loss of the target model against its true label y. The goal of the targeted attack is to minimize the loss towards its adversarial target label yt.
How do we formally define that an attack is effective?
Definition 2 ((α,F)-Effective Attack). Consider a input x ∈ X with true label y ∈ Y . An attack is (α,F )-effective in untargeted scenario if Pr (F(AU (x)) 6= y) ≥ 1− α. An attack is (α,F )-effective in targeted scenario (with class target yt) if Pr (F(AT (x)) = yt) ≥ 1− α.
This definition captures the requirement that an adversarial instance generated by an effective attack strategy is able to mislead the target classification model (e.g. F) with certain probability (1− α). The smaller the α is, the more effective the attack is. In practice, this implies that on a finite sample of targets, the attack success is frequent but not absolute. Note that the definition is general for both whitebox [1, 12, 5] and blackbox attacks [42, 4].

heading: 2.1 Model Characteristics
text: Given two modelsF and G, what are the characteristics ofF and G that have impact on transferability under a given attack strategy? Intuitively, the more similar these two classifers are, the larger the transferability would be. However, how can we define “similar” and how can we rigorously connect it to transferability? To answer these questions, we will first define the risk and empirical risk for a given model to measure its performance on benign test data. Then, as the DNNs are differentiable, we will define model similarity based on their gradients. We will then derive the lower and upper bounds of adversarial transferability based on the defined model risk and similarity measures. Definition 3 (Risk and Empirical Risk). For a given model F , we let `F be its model-dependent loss function. Its risk is defined as ηF = Pr (F(x) 6= y); and its empirical risk is defined as ξF = E [`F (x, y)].
The risk represents the model’s error rate on benign test data, while the empirical risk is a non-negative value that also indicates the inaccuracy. For both of them, higher value means worse performance on the benign test data. The difference is that, the risk has more intuitive meaning, while the empirical risk is differentiable and is actually used during model training. Definition 4 (Loss Gradient Similarity). The lower loss gradient similarity S and upper loss gradient similarity S between two differentiable loss functions `F and `G is defined as:
S(`F , `G) = inf x∈X ,y∈Y ∇x`F (x, y) · ∇x`G(x, y) ‖∇x`F (x, y)‖2 · ‖∇x`G(x, y)‖2 ,S(`F , `G) = sup x∈X ,y∈Y ∇x`F (x, y) · ∇x`G(x, y) ‖∇x`F (x, y)‖2 · ‖∇x`G(x, y)‖2 .
The S(`F , `G) (S(`F , `G)) is the minimum (maximum) cosine similarity between the gradients of the two loss functions for an input x drawn from X with any label y ∈ Y . Besides the loss gradient similarity, in our analysis we will also show that the model smoothness is another key characteristic of ML models that affects the model transferability.
Definition 5. We call a model F β-smooth if sup x1,x2∈X ,y∈Y ‖∇x`F (x1, y)−∇x`F (x2, y)‖2 ‖x1 − x2‖2 ≤ β.
This smoothness definition is commonly used in deep learning theory and optimization literature [3, 2], and is also named curvature bounds in certified robustness literature [44]. It could be interpreted as the Lipschitz bound for the model’s loss function gradient. We remark that larger β indicates that the model is less smoother, while smaller β means the model is smoother. Particularly, when β = 0, the model is linear in the input space X .

heading: 2.2 Definition of Adversarial Transferability
text: Based on the model characteristics we explored above, next we will ask: Given two models, what is the natural and precise definition of adversarial transferability?
Definition 6 (Transferability). Consider an adversarial instance AU (x) or AT (x) constructed against a surrogate model F . With a given benign input x ∈ X , The transferability Tr between F and a target model G is defined as follows (adversarial target yt ∈ Y):
• Untargeted: Tr(F ,G, x) = I[F(x) = G(x) = y ∧ F(AU (x)) 6= y ∧ G(AU (x)) 6= y].
• Targeted: Tr(F ,G, x, yt) = I[F(x) = G(x) = y ∧ F(AT (x)) = G(AT (x)) = yt].
Here we define the transferability at instance level, showing several conditions are required to satisfy for a transferable instance. For the untargeted attack, it requires that: (1) both the surrogate model
and target model make correct prediction on the benign input; and (2) both of them make incorrect predictions on the adversarial inputAU (x). The AU (x) is generated via the untargeted attack against the surrogate model F . For the targeted attack, it requires that: (1) both the surrogate and target model make correct prediction on benign input; and (2) both output the adversarial target yt ∈ Y on the adversarial input AT (x). The AT (x) is crafted against the surrogate model F . The predicates themselves do not require AU and AT to be explicitly constructed against the surrogate model F . It will be implied by attack effectiveness (Definition 2) on F in theorem statements. Note that the definition here is a predicate for a specific input x, and in the following analysis we will mainly use its distributional version: Pr (Tr(F ,G, x) = 1) and Pr (Tr(F ,G, x, yt) = 1).

heading: 2.3 Lower Bound of Adversarial Transferability
text: Based on the general definition of transferability, in this section we will analyze how to lower bound the transferability for targeted attack. The analysis for untargeted attack has a similar form and is deferred to Theorem 3 in Appendix A. Theorem 1 (Lower Bound on Targeted Attack Transferability). Assume both models F and G are β-smooth. Let AT be an (α,F)-effective targeted attack with perturbation ball ‖δ‖2 ≤ and target label yt ∈ Y . The transferabiity can be lower bounded by
Pr (Tr(F ,G, x, yt) = 1) ≥ (1−α)−(ηF+ηG)− (1 + α) + cF (1− α) cG + − (1− α) cG +
√ 2− 2S(`F , `G),
where
cF = max x∈X
min y∈Y
`F (AT (x), y)− `F (x, yt) + β 2/2
‖∇x`F (x, yt)‖2 , cG = min x∈X
min y∈Y
`G(AT (x), y)− `G(x, yt)− β 2/2
‖∇x`G(x, yt)‖2 .
Here ηF , ηG are the risks of models F and G respectively.
We defer the complete proof in Appendix C. In the proof, we first use a Taylor expansion to introduce the gradient terms, then relate the dot product with cosine similarity of the loss gradients, and finally use Markov’s inequality to derive the misclassification probability of G to complete the proof.
Implications. In Theorem 1, the only term which correlates both F and G is S(`F , `G), while all other terms depend on individual models F or G. Thus, we study the relation between S(`F , `G) and Pr (Tr(F ,G, x, yt) = 1). Note that since β is small compared with the perturbation radius and the gradient magnitude ‖∇x`G‖2 in the denominator is relatively large, the quantity cG is small. Moreover, 1 − α is large since the attack is typically effective against F . Thus, Pr (Tr(F ,G, x, yt) = 1) has the form C − k √ 1− S(`F , `G), where C and k are both positive constants. We can easily observe the positive correlation between the loss gradients similarity S(`F , `G), and lower bound of adversarial transferability Pr (Tr(F ,G, x, yt) = 1). In the meantime, note that when β increases (i.e., model becomes less smooth), in the transferability lower bound C − k √ 1− S(`F , `G), the C decreases and k increase. As a result, the lower bounds in Theorem 1 decreases, which implies that when model becomes less smoother (i.e., β becomes larger), the transferability lower bounds become looser for both targeted and untargeted attacks. In other words, when the model becomes smoother, the correlation between loss gradients similarity and lower bound of transferability becomes stronger, which motivates us to constrain the model smoothness to increase the effect of limiting loss gradients similarity.
In addition to the `p-bounded attacks, we also derive a transferability lower bound for general attacks whose magnitude is bounded by total variance distance of data distributions. We defer the detail analysis and discussion to Appendix B.

heading: 2.4 Upper Bound of Adversarial Transferability
text: We next aim to upper bound the adversarial transferability. The upper bound for target attack is shown below; and the one for untargeted attack has a similar form in Theorem 4 in Appendix A. Theorem 2 (Upper Bound on Targeted Attack Transferability). Assume both models F and G are β-smooth with gradient magnitude bounded by B, i.e., ‖∇x`F (x, y)‖ ≤ B and ‖∇x`G(x, y)‖ ≤ B for any x ∈ X , y ∈ Y . LetAT be an (α,F)-effective targeted attack with perturbation ball ‖δ‖2 ≤
and target label yt ∈ Y . When the attack radius is small such that `min− B ( 1 + √ 1+S(`F ,`G)
2
) −
β 2 > 0, the transferability can be upper bounded by
Pr (Tr(F ,G, x, yt) = 1) ≤ ξF + ξG `min − B ( 1 + √ 1+S(`F ,`G)
2
) − β 2 ,
where `min = min x∈X (`F (x, yt), `G(x, yt)). Here ξF and ξG are the empirical risks of models F and G respectively, defined relative to a differentiable loss.
We defer the complete proof to Appendix D. In the proof, we first take a Taylor expansion on the loss function at (x, y), then use the fact that the attack direction will be dissimilar with at least one of the model gradients to upper bound the transferability probability.
Implications. In Theorem 2, we observe that along with the increase of S(`F , `G), the denominator decreases and henceforth the upper bound increases. Therefore, S(`F , `G)—upper loss gradient similarity and the upper bound of transferability probability is positively correlated. This tendency is the same as that in the lower bound. Note that α does not appear in upper bounds since only completely successful attacks (α = 0%) needs to be considered here to upper bound the transferability.
Meanwhile, when the model becomes smoother (i.e., β decreases), the transferability upper bound decreases and becomes tighter. This implication again motivates us to constrain the model smoothness. We further observe that smaller magnitude of gradient, i.e., B, also helps to tighten the upper bound. We will regularize both B and β to increase the effect of constraining loss gradients similarity.
Note that the lower bound and upper bound jointly show smaller β leads to a reduced gap between lower and upper bounds and thus a stronger correlation between loss gradients similarity and transferabiltiy. Therefore, it is important to both constrain gradient similarity and increase model smoothness (decrease β) to reduce model transferability and improve ensemble robustness.

heading: 3 Improving Ensemble Robustness via Transferability Minimization
text: Motivated by our theoretical analysis, we propose a lightweight yet effective robust ensemble training approach, Transferability Reduced Smooth (TRS), to reduce the transferability among base models by enforcing low loss gradient similarity and model smoothness at the same time.

heading: 3.1 TRS Regularizer
text: In practice, it is challenging to directly regularize the model smoothness. Luckily, inspired from deep learning theory and optimization [14, 37, 45], succinct `2 regularization on the gradient terms ‖∇x`F‖2 and ‖∇x`G‖2 can reduce the magnitude of gradients and thus improve model smoothness. For example, for common neural networks, the smoothness can be upper bounded via bounding the `2 magnitude of gradients [45, Corollary 4]. An intuitive explanation is that, the `2 regularization on the gradient terms reduces the magnitude of model’s weights, thus limits its changing rate when non-linear activation functions are applied to the neural network model. However, we find that directly regularizing the loss gradient magnitude with `2 norm is not enough, since a vanilla `2 regularizer such as ‖∇x`F‖2 will only focus on the local region at data point x, while it is required to ensure the model smoothness over a large decision region to control the adversarial transferability based on our theoretical analysis.
To address this challenge, we propose a min-max framework to regularize the “support” instance x̂ with “worst” smoothness in the neighborhood region of data point x, which results in the following model smoothness loss:
Lsmooth(F ,G, x, δ) = max ‖x̂−x‖∞≤δ ‖∇x̂`F‖2 + ‖∇x̂`G‖2 (1)
where δ refers to the radius of the `∞ ball around instance x within which we aim to ensure the model to be smooth. In practice, we leverage projection gradient descent optimization to search for support instances x̂ for optimization. This model smoothness loss can be viewed as promoting margin-wise smoothness, i.e., improving the margin between nonsmooth decision boundaries and data point x. Another option is to promote point-wise smoothness that only requires the loss landscape
at data point x itself to be smooth. We compare the ensemble robustness of the proposed min-max framework which promotes the margin-wise smoothness with the naïve baseline which directly applies `2 regularization on each model loss gradient terms to promote the point-wise smoothness (i.e. Cos-`2) in Section 4.
Given trained “smoothed" base models, we also decrease the model loss gradient similarity to reduce the overall adversarial transferability between base models. Among various metrics which measure the similarity between the loss gradients of base model F and G, we find that the vanilla cosine similarity metric, which is also used in [23], may lead to certain concerns. By minimizing the cosine similarity between∇x`F and∇x`G , the optimal case implies∇x`F = −∇x`G , which means two models have contradictory (rather than diverse) performance on instance x and thus results in turbulent model functionality. Considering this challenge, we leverage the absolute value of cosine similarity between ∇x`F and ∇x`G as similarity loss Lsim and its optimal case implies orthogonal loss gradient vectors. For simplification, we will always use the absolute value of the gradient cosine similarity as the indicator of gradient similarity in our later description and evaluation.
Based on our theoretical analysis and particularly the model loss gradient similarity and model smoothness optimization above, we propose TRS regularizer for model pair (F ,G) on input x as:
LTRS(F ,G, x, δ) = λa · Lsim + λb · Lsmooth = λa · ∣∣∣∣ (∇x`F )>(∇x`G)‖∇x`F‖2 · ‖∇x`G‖2 ∣∣∣∣+ λb · [ max‖x̂−x‖∞≤δ ‖∇x̂`F‖2 + ‖∇x̂`G‖2 ] .
Here ∇x`F and ∇x`G refer to the loss gradient vectors of base models F and G on input x, and λa, λb the weight balancing parameters.
In Section 4, backed up by extensive empirical evaluation, we will systematically show that the local min-max training and the absolute value of the cosine similarity between the model loss gradients significantly improve the ensemble model robustness with negligible performance drop on benign accuracy, as well as reduce the adversarial transferability among base models.

heading: 3.2 TRS Training
text: We integrate the proposed TRS regularizer with the standard ensemble training loss, such as Ensemble Cross-Entropy (ECE) loss, to maintain both ensemble model’s classification utility and robustness by varying the balancing parameter λa and λb. Specifically, for an ensemble model consisting of N base models {Fi}Ni=1, given an input (x, y), our final training loss train is defined as:
Ltrain = 1
N N∑ i=1 LCE(Fi(x), y) + 2 N(N − 1) N∑ i=1 N∑ j=i+1 LTRS(Fi,Fj , x, δ)
where LCE(Fi(x), y) refers to the cross-entropy loss between Fi(x), the output vector of model Fi given x, and the ground-truth label y. The weight of LTRS regularizer could be adjusted by the tuning λa and λb internally. We present one-epoch training pseudo code in Algorithm 1 of Appendix F. The detailed hyper-parameter setting and training criterion are discussed in Appendix F.

heading: 4 Experimental Evaluation
text: In this section, we evaluate the robustness of the proposed TRS-ensemble model under both strong whitebox attacks, as well as blackbox attacks considering the gradient obfuscation concern [1]. We compare TRS with six state-of-the-art ensemble approaches. In addition, we evaluate the adversarial transferability among base models within an ensemble and empirically show that the TRS regularizer can indeed reduce transferability effectively. We also conduct extensive ablation studies to explore the effectiveness of different loss terms in TRS, as well as visualize the trained decision boundaries of different ensemble models to provide intuition on the model properties. We open source the code3 and provide a large-scale benchmark.

heading: 4.1 Experimental Setup
text: Datasets. We conduct our experiments on widely-used image datasets including hand-written dataset MNIST [29]; and colourful image datasets CIFAR-10 and CIFAR-100 [26].
3https://github.com/AI-secure/Transferability-Reduced-Smooth-Ensemble
Baseline ensemble approaches. We mainly consider the standard ensemble, as well as the state-ofthe-art robust ensemble methods that claim to be resilient against adversarial attacks. Specifically, we consider the following baseline ensemble methods which aim to promote the diversity between base models: AdaBoost [19]; GradientBoost [16]; CKAE [25]; ADP [38]; GAL [23]; DVERGE [56]. The detailed description about these approaches are in Appendix E. DVERGE, which has achieved the state-of-the-art ensemble robustness to our best knowledge, serves as the strongest baseline.
Whitebox robustness evaluation. We consider the following adversarial attacks to measure ensembles’ whitebox robustness: Fast Gradient Sign Method (FGSM) [17]; Basic Iterative Method (BIM) [34]; Momentum Iterative Method (MIM); Projected Gradient Descent (PGD); Auto-PGD (APGD); Carlini & Wanger Attack (CW); Elastic-net Attack (EAD) [8], and we leave the detailed description and parameter configuration of these attacks in Appendix E. We use Robust Accuracy as our evaluation metric for the whitebox setting, defined as the ratio of correctly predicted adversarial examples generated by different attacks among the whole test dataset.
Blackbox robustness evaluation. We also conduct blackbox robustness analysis in our evaluation since recent studies have shown that robust models which obfuscate gradients could still be fragile under blackbox attacks [1]. In the blackbox attack setting, we assume the attacker has no knowledge about the target ensemble, including the model architecture and parameters. In this case, the attacker is only able to craft adversarial examples based on several surrogate models and transfer them to the target victim ensemble. We follow the same blackbox attack evaluation setting in [56]: We choose three ensembles consisting of 3, 5, 8 base models which are trained with standard Ensemble Cross-Entropy (ECE) loss as our surrogate models. We apply 50-steps PGD attack with three random starts and two different loss functions (CrossEntropy and CW loss) on each surrogate model to generate adversarial instances (i.e. for each instance we will have 18 attack attempts). For each instance, among these attack attempts, as long as there is one that can successfully attack the victim model, we will count it as a successful attack. In this case, we use Robust Accuracy as our evaluation metric, defined as the number of unsuccessful attack attempts divided by the number of all attacks. We also consider additional three strong blackbox attacks targeting on reducing transferability (i.e., ILA [21], DI2-SGSM [55], IRA [50]) in Appendix J, which leads to similar observations.

heading: 4.2 Experimental Results
text: In this section, we present both whitebox and blackbox robustness evaluation results, examine the adversarial transferability, and explore the impacts of different loss terms in TRS. Furthermore, in Appendix I.1, we visualize the decision boundary; in Appendix I.2, we show results of further improving the robustness of the TRS ensemble by integrating adversarial training; in Appendix I.3, we study the impacts of each of the regularization term Lsim and Lsmooth; in Appendix I.4, we show the convergence of robust accuracy under large attack iterations to demonstrate the robustness stability of TRS ensemble; in Appendix I.5, we analyze the trade-off between the training cost and robustness of TRS by varying PGD step size and the total number of steps within Lsmooth approximation. Whitebox robustness. Table 1 presents the Robust Accuracy of different ensembles against a range of whitebox attacks on MNIST and CIFAR-10 dataset. We defer results on CIFAR-100 in Appendix K, and measure the statistical stability of our reported robust accuracy in Appendix H. Results shows that the proposed TRS ensemble outperforms other baselines including the state-of-the-art DVERGE significantly, against a range of attacks and perturbation budgets, and such performance gap could be even larger under stronger adversary attacks (e.g. PGD attack). We note that TRS ensemble is slightly less robust than DVERGE under small perturbation with weak attack FGSM. We investigate this based on the decision boundary analysis in Appendix I.1, and find that DVERGE tends to be more robust along the gradient direction and thus more robust against weak attacks which only focus on the gradient direction (e.g., FGSM); while TRS yields a smoother model along different directions leading to more consistent predictions within a larger neighborhood of an input, and thus more robust against strong iterative attacks (e.g., PGD). This may be due to that DVERGE is essentially performing adversarial training for different base models and therefore it protects the adversarial (gradient) direction, while TRS optimizes to train a smooth ensemble with diverse base models. We also analyze the convergence of attack algorithms in Appendix I.4, showing that when the number of attack iterations is large, both ADP and GAL ensemble achieve much lower robust accuracy against such iterative attacks; while both DVERGE and TRS remain robust.
Blackbox robustness. Figure 2 shows the Robust Accuracy performance of TRS compared with different baseline ensembles under different perturbation budget . As we can see, the TRS ensemble achieves competitive robust accuracy with DVERGE when is very small, and TRS beats all the baselines significantly when is large. Precisely speaking, TRS ensemble achieves over 85% robust accuracy against transfer attack with = 0.4 on MNIST while the second-best ensemble (DVERGE) only achieves 20.2%. Also on CIFAR-10, TRS ensemble achieves over 25% robust accuracy against transfer attack when = 0.06, while all the other baseline ensembles achieve robust accuracy lower than 6%. This implies that our proposed TRS ensemble has stronger generalization ability in terms of robustness against large adversarial attacks compared with other ensembles. We also put more details of the robust accuracy under blackbox attacks in Appendix G.
Adversarial transferability. Figure 3 shows the adversarial transferability matrix of different ensembles against 50-steps PGD attack with = 0.3 for MNIST and = 0.04 for CIFAR-10. Cell (i, j) where i 6= j represents the transfer attack success rate evaluated on j-th base model by using the i-th base model as the surrogate model. Lower number in each cell indicates lower transferability and thus potentially higher ensemble robustness. The diagonal cell (i, i) refers to i-th base model’s attack success rate, which reflects the vulnerability of a single model. From these figures, we can see that while base models show their vulnerabilities against adversarial attack, only DVERGE and TRS ensemble could achieve low adversarial transferability among base models. We should also notice that though GAL applied a similar gradient cosine similarity loss as our loss term Lsim, GAL still can
not achieve low adversarial transferability due to the lack of model smoothness enforcement, which is one of our key contributions in this paper.
Gradient similarity only vs. TRS. To further verify our theoretical analysis on the sufficient condition of transferability as model smoothness, we consider only applying similarity loss Lsim without model smoothness loss Lsmooth in TRS (i.e. λb = 0). The result is shown as “Cos-only” method of Table 1. We observe that the resulting whitebox robustness is much worse than standard TRS. This matches our theoretical analysis that only minimizing the gradient similarity cannot guarantee low adversarial transferability among base models and thus lead to low ensemble robustness. In Appendix I.3, we investigate the impacts of Lsim and Lsmooth thoroughly, and we show that though Lsmooth contribute slightly more, both terms are critical to the final ensemble robustness. `2 regularizer only vs. Min-max model smoothing. To emphasis the importance of our proposed min-max training loss on promoting the margin-wise model smoothness, we train a variant of TRS ensemble Cos-`2, where we directly apply the `2 regularization on ‖∇x`F‖2 and ‖∇x`G‖2. The results are shown as “Cos-`2” in Table 1. We observe that Cos-`2 achieves lower robustness accuracy compared with TRS, which implies the necessity of regularizing the gradient magnitude on not only the local training points but also their neighborhood regions to ensure overall model smoothness.

heading: 5 Conclusion
text: In this paper, we deliver an in-depth understanding of adversarial transferability. Theoretically, we provide both lower and upper bounds on transferability which shows that smooth models together with low loss gradient similarity guarantee low transferability. Inspired by our analysis, we propose TRS ensemble training to empirically reduce transferability by reducing loss gradient similarity and promoting model smoothness, yielding a significant improvement on ensemble robustness.
Acknowledgments and Disclosure of Funding
This work is partially supported by the NSF grant No.1910100, NSF CNS 20-46726 CAR, the Amazon Research Award, and the joint CATCH MURI-AUSMURI.

Review summaries:
The manuscript describes the problem of adversarial transferability, the process of adversarial attacks that can be transferred across models, which can help scale adversarial attack strategies. As a guardrail against such attacks, it is desired that models are robust and protected against such attacks. Towards solving this, the authors first derive a lower and upper bound on a transferability metric that they define. The similarity between gradient losses was found to be the salient cross-model term affecting adversarial transferability. Subsequently, this pointed to a strategy of reducing model smoothness and loss gradient similarity across models as a way to reduce adversarial transferability.

This work first theoretically analyzes and outlines sufficient conditions for adversarial transferability between models, then proposes a practical algorithm to reduce the transferability between base models within an ensemble to improve its robustness. They also provide the lower and upper bounds of adversarial transferability under certain conditions and propose an effective Transferability Reduced Smooth (TRS) ensemble training strategy to train a robust ensemble with low transferability. Extensive experiments demonstrate that the proposed TRS outperforms all baselines significantly.

1. This paper provides theoretical analysus on the sufficient condition on the adversarial transferability between ML models, and identify the importance of both gradient orthogonality and model smoothness  2. This paper proposes a simple smoothness regularization term and merge it with gradient diversity objective to form TRS robust ensemble training algorithm  3. Thorough experiments are provided to show TRS achieves state-of-the-art ensemble robustness