TRS: Transferability Reduced Ensemble via Promoting Gradient Diversity and Model Smoothness
Authors: Zhuolin Yang
Linyi Li
Xiaojun Xu
Shiliang Zuo
Qian Chen
Pan Zhou
Benjamin I. P. Rubinstein
Ce Zhang
Bo Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: robustness
ensemble
adversarial transferability
diversity

Abstract: Adversarial Transferability is an intriguing property - adversarial perturbation crafted against one model is also effective against another model, while these models are from different model families or training processes. To better protect ML systems against adversarial attacks, several questions are raised: what are the sufficient conditions for adversarial transferability, and how to bound it? Is there a way to reduce the adversarial transferability in order to improve the robustness of an ensemble ML model? To answer these questions, in this work we first theoretically analyze and outline sufficient conditions for adversarial transferability between models; then propose a practical algorithm to reduce the transferability between base models within an ensemble to improve its robustness. Our theoretical analysis shows that only promoting the orthogonality between gradients of base models is not enough to ensure low transferability; in the meantime, the model smoothness is an important factor to control the transferability. We also provide the lower and upper bounds of adversarial transferability under certain conditions. Inspired by our theoretical analysis, we propose an effective Transferability Reduced Smooth (TRS) ensemble training strategy to train a robust ensemble with low transferability by enforcing both gradient orthogonality and model smoothness between base models. We conduct extensive experiments on TRS and compare with 6 state-of-the-art ensemble baselines against 8 whitebox attacks on different datasets, demonstrating that the proposed TRS outperforms all baselines significantly. 

TRS: Transferability Reduced Ensemble via Promoting Gradient Diversity and Model Smoothness
Zhuolin Yang
Linyi Li Benjamin Rubinstein
Xiaojun Xu!
Shiliang Zuo
Qian Chen
Pan Zhou
Ce Zhang
Bo Li
University of Illinois Urbana-Champaign Tencent Inc.
University of Melbourne
Huazhong University of Science and Technology ETH Zurich
zhuolin5,
linyi2,
xiaojun3, szuo3, [bo ]bo @illinoiss.ed
q gianchen&tenccent com
jannamii.co Sennjjammi...ummmmmemmmmmmmmmml edu. au
Sanzhou@hust.edu.
ce. zazzaaannnnntt.. .et.t.the...
Abstract
Adversarial Transferability is an intriguing property adversarial perturbation crafted against one model also effective against another model, while these mod- els are from different model families or training processes. To better protect ML systems against adversarial attacks, several questions are raised: what are the suffi- cient conditions for adversarial transferability and how to bound it? Is there way to reduce the adversarial transferability in order improve the robustness of an ensemble ML model? To answer these questions, in this work first theoretically analyze and outline sufficient conditions for adversarial transferability between models; then propose practical algorithm to reduce the transferability between base models within an ensemble to improve its robustness. Our theoretical analysis shows that only promoting the orthogonality between gradients of base models is not enough to ensure low transferability; in the meantime, the model smoothness is important factor control the transferability. We also provide the lower and upper bounds of adversarial transferability under certain conditions. Inspired by theoretical analysis, propose effective Transferability Reduced Smooth (TRS) ensemble training strategy to train a robust ensemble with low transferability by enforcing both gradient orthogonality and model smoothness between base mod- els. We conduct extensive experiments on TRS and compare with state-of-the tfrth-rrr ensemble baselines against 8 whitebox attacks on different datasets, demonstrating that the proposed TRS outperforms all baselines significantly.
Introduction
Machine learning systems, especially those based on deep neural networks (DNNs), have been widely
applied in numerous applications [27 [8] 46, [0]. However, recent studies show that DNNs are
vulnerable to adversarial examples, which are able to mislead DNNs by adding small magnitude of
perturbations the original instances 47 17 54 [2]. Several attack strategies have been proposed so far to generate such adversarial examples in both digital and physical environments [36 32 51
53 15 28 Intriguingly, though most attacks require access to the target models (whitebox attacks), several studies show that adversarial examples generated against one model are able to transferably
'The authors contributed equally.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

attack another target model with high probability, giving rise to blackbox attacks This property of adversarial transferability poses great threat to DNNs.
39 41
31
30
57
Some work have been conducted to understand adversarial transferability 48 33 12 However, rigorous theoretical analysis or explanation for transferability is still lacking in the literature. In addition, although developing robust ensemble models to limit transferability shows great potential towards practical robust learning systems, only empirical observations have been made in this line
of research [8 23 [6] Can we deepen our theoretical understanding on transferability? Can we
take advantage of rigorous theoretical understanding to reduce the adversarial transferabilityy and therefore generate robust ensemble ML models?
In this paper, we focus these two questions. From the theoretical side, we are interested in the sufficient conditions under which the adversarial transferabil- ity can be lower bounded and upper bounded. Our theoretical arguments provides the first theoretical interpretation for the sufficient conditions of trans- ferability. Intuitively, as illustrated in Figure T we
Docision Boundary
Foalure
Feature
show that the commonly used gradient orthogonality (low cosine similarity) between learning models [12 cannot directly imply low adversarial transferabilityy; on the other hand, orthogonal and smoothed models would limit the transferability. In particular, we prove that the gradient similarity and model smoothness are the key factors that both contribute to the adversarial transferability, and smooth models with orthogonal gradients can guarantee low transferability.
Decisian Bourdary
Rtscision Beundars
Feature
Feature
(a) Orthogonal but Not Smooth
(h) Orthogonal and Smooth
Figure I: An illustration of the relationship be- tween adversarial trunsferability. gradient orthog- onality. and model smoothness. (a) Gradient or- thogonality alone cannot minimize transferabil- ity as the decision boundaries between two clas- sifiers can be arbitrarily close yet orthogonal al- most everywhere; (b) Gradient orthogonality with model smoothness provides a stronger guarantee on model diversity, as our theorems will show.
Under an empirical lens, inspired by our theoretical analysis, we propose simple yet effective approach, Transferability Reduced Smooth (TRS) ensemble to limit adversarial transfcrability between base models within an ensemble and therefore improve its robustness. In particular, we reduce the loss gradient similarity between models as well as enforce the smoothness of models to introduce global model orthogonality.
We conduct extensive experiments to evaluate TRS in terms of the model robustness against different strong boxte-box and blackbox attacks following the robustness evaluation procedures (5] 6 49]
as well as its ability to limit transferability across the base models. We compare the proposed
TRS with existing state-of-the-an baseline ensemble approaches such as ADP [38] GAL [2]] and DVERGE [56] on MNIST, CIFAR- 10, and CIFAR-100 datasets, and we show that (1) TRS achieves the state-of-the-ar art ensemble robustness, outperforming others by large margin; (2) TRS achieves efficient training; (3) TRS effectively reduces the transferabilityy among base models within an ensemble which indicates its robustness against whitebox and blackbox attacks; (4) Both loss terms in TRS contribute to the ensemble robustness by constraining different sufficient conditions of adversarial transferability.
Contributions, In this paper, we make the first attempt towards theoretical understanding of adver- sarial transferability, and provide practical approach for developing robust ML ensembles.
(/) We provide general theoretical analysis framework for adversarial transferability. We prove the lower and upper bounds of adversarial transferability. Both bounds show that the gradient similarity and model smoothness are the key factors contributing to the adversarial transferability, and smooth models with orthogonal gradients can guarantee low transferability.
(2) We propose a simple yet effective approach TRS to train a robust ensemble by jointly reducing the loss gradient similarity between base models and enforcing the model smoothness. The code is publicly available"]
(3) We conduct extensive experiments to evaluate TRS in terms of model robustness under different attack settings, showing that TRS achieves the state-of-the-ar ensemble robustness and outper- forms other baselines by a large margin. We also conduct ablation studies to further understand the contribution of different loss terms and verify our theoretical findings.
Smoothh Bertotth--nsemmlee

---

Related Work
The adversarial transferability between different ML models is an intriguing research direction. Papernot et al. [4] explored the limitation of adversarial examples and showed that, while some instances are more difficult to manipulate than the others. these adversarial examples usually transfer from one model to another. Demontis et al. [12] later analyzed transferability for both evasion and poisoning attacks. [Tramèr al. [48] empirically investigated the subspace of adversarial examples that enables transferabilityy between different models: though their results provide a non-zero probability guarantee on the transferability, they did not quantify the probability of adversarial transferability.
Leveraging the transferability, different blackbox attacks have been proposed 41 28
15
9
To
defend against these transferability based attacks, Pang et al. (8 proposed a class entropy based adaptive diversity promoting approach to enhance the ML ensemble robustness. Recently, Yang al.) [56] proposed DVERGE, a robust ensemble training approach that diversifies the non-robust features of base models via an adversarial training objective function. However, these approaches do not provide theoretical justification for adversarial transferability, and there is still room to improve the MI. ensemble robustness based on in-depth understanding on the sufficient conditions of transferability. In this paper, we aim to provide theoretical understanding of transferability, and empirically compare the proposed robust ML. ensemble inspired by our theoretical analysis with existing approaches to push for a tighter empirical upper bound for the ensemble robustness.
Transferability of Adversarial Perturbation
In this section, we first introduce preliminaries, and then provide the upper and lower bounds of adversarial transferability by connecting adversarial transferability with different characteristics of models theoretically, which, in the next section, will allow us explicitly minimize transferability by enforcing (or rewarding) certain properties of models.
Notations. We consider neural networks for classification tasks. Assume there are C' classes, and let X be the input space of the model with y = 21. C} the set of prediction classes (i.e., labels). We model the neural network by mapping function F X' - V. We will study the transfcrability between two models F and G. For brevity, hereinafter we mainly show the derived notations for F and notations for 9 are similar. Let the benign data (zt, y) follow an unknown distribution DD supported on (X, )), and Pxx denote the marginal distribution on X.
For given input x E X, the classification model FF first predicts the confidence score for each label y E V, denoted as (z)). These confidence scores sum up to 1, i.e. Eyey J (x) = 1, Ve E X. The model FF will predicts the label with highest confidence score: F(x) = arg max yEy (z))
For model F, there is usually model-dependent loss function f x' - R4. which is the compo- sition of a differentiable training loss (e.g.. cross-entropy loss) and the model's confidence score f()): (=(x, v) := {( (=r y), (z., v) E (X,)). We further assume that F(x) = arg min (r v). i.e., the model predicts the label with minimum loss. This holds for common training losses.
In this paper, by default we will focus on models that are well-trained on the benign dataset, and such models are the most commonly encountered in practice, so their robustness is paramount. This means we will focus on the low risk classifiers, which we will formally define in Section 2.
How should we define an adversarial attack? For the threat model, we consider the attacker that adds an Ep norm bounded perturbation to data instance x E X. In practice, there are two types of attacks, untargeted attacks and targeted attacks. The definition of adversarial transferability is slightly different under these attacks [3), and we consider both in our analysis.
Definition (Adversarial Attack), Given an input x E X with true label y E y, F(x)= = y. An An untargeted attack crafts Az( = + â to maximize Ex(x + 8,9) y) where [81p S (2) A targeted attack with target label Yt E y crafts (x) = x + to minimize E (x + 5, yt) where ||01|p Se e.
In this definition, c is pre-defined attack radius that limits the power of the attacker. We may refer to {8 ||1 I <} as the perturbation ball. The goal of the untargeted attack is to maximize the loss of the target model against its true label y. The goal of the targeted attack is to minimize the loss towards its adversarial target label yt-
How do we formally define that an attack is effective?
3

---

Definition 2 ((a. F...FFFetttiyy Attack), Consider input E X with tuee label E y. An attack is (a, F)-efecctive in untargeted scenario Pf (F(Ar(z) + a. An attack is (a, -effeectivv in targeted scenario (with class larget y1) if Pr (F vs) 2 a.
This definition captures the requirement that an adversarial instance generated by an effective attack strategy is able to mislead the target classification model (c.g. FF) with certain probability (1 a). The smaller the a is, the more effective the attack is. In practice, this implics that on a finite sample of targets, the attack success is frequent but not absolute. Note that the definition is general for both whitebox
12 and blackbox attacks 42
2.1 Model Characteristics
Given two models F and 9, what are the characteristies of F and that have impact on transferability under given attack strategy? Intuitively, the more similar these two classifers are, the larger the transfcrability would be. However, how can we define "similar" and how can we rigorously connect it to transferability? To answer these questions, given model to measure its performance on benign test data. Then, as the DNNs are differentiable, will define model similarity based their gradients. We will then derive the lower and upper bounds of adversarial transferability based on the defined model risk and similarity measures.
will first define the risk and empirical risk for
Definition 3 (Risk and Empirical Risk). For a given model F, we let E be its model-depene mrdee-ddpendent loss function. IIs risk is defined as oF Pr (FF() + y); and its empirical risk is defined as E = (r,
The risk represents the model's error rate benign test data, while the empirical risk is nog-neettive value that also indicates the inaccuracy. For both of them, higher value means worse performance on the henign test data. The difference is that, the risk has more intuitive meaning, while the empirical risk is differeeniable and is actually used during model training.
Definition (Loss Gradient Similarity). The lower loss gradient similarity S and upper loss gradient similarity S between two differentiable loss functions & and Eg is defined as:
v,lv(E,u) Valo(E.w) f() = inf
valr(s,n) Vela(a.v) ZEX.YEY | Vadg(z, vl
=EX.NEy |V,tr(e, (V 6()
The S(Er, Eg) (S(E), (g)) is the minimum (maximum) cosine similarity between the gradients of the two loss functions for an input z drawn from X' with any label E V. Besides the loss gradient similarity, our analysis we will also show that the model smoothness another key characteristic of ML models that affects the model transferability.
|2 v) V.l=(x2.8lll S B.
Definition 5. We call a model F 8-smooth if sup E
| 22 l1
This smoothness definition is commonly used in deep learning theory and optimization literature 3 21. and is also named curvature bounds in certified robustness literature 44]. It could be interpreted as the Lipschitz bound for the model's loss function gradient. We remark that larger 3 indicates that the model is less smoother, while smaller /3 means the model is smoother. Particularly, when 3 = 0, the model is linear in the input space X.
2.2 Definition of Adversarial Transferability
Based on the model characteristics we explored above, next we will ask: Given two models, what is the natural and precise definition of adversarial transferability?
Definition 6 (Transfcrabilityy. Consider an adversarial instance Ar(x) or Ar(x) constructed against surrogate model F. With a given benign input E X, The transferability T, between F and a target model defined as follows (adversarial target yt E YY);
(0])===))))))))) = A FFAu # A (x)) + vl. * Untargeted: T,(F, [[F g(2)
Targeted: T, 9. xx 1[F(x) g()))))>(((>)>) = F((r(()) = ((Ar(z)) vd)
Here we define the transferability at instance level, showing several conditions are required to satisfy for transferable instance. For the untargeted attack, it requires that: (1) both the surrogate model

---

and target model make correct prediction on the benign input; and (2) both of them make incorrect predictions on the adversarial input (z). The A;; is generated via the untargeted attack against the surrogate model F. For the targeted attack, it requires that: (1) both the surrogate and target model make correct prediction on benign input; and (2) both output the adversarial target Yt E ) on the adversarial input (z:). The Ar(æ) is crafted against the surrogate model F. The predicates themselves do not require Ar; and Ar to be explicitly constructed against the surrogate model F. It will be implied by attack effectiveness (Definition 2) on J in theorem statements. Note that the definition here is a predicate for a specific input æ, and in the following analysis we will mainly use its distributional version: Pr (J, g,z) = 1) and Pr (T, G, V1) 1).
2.3 Lower Bound of Adversarial Transferability
Based on the general definition of transferability, in this section we will analyze how to lower bound the transferability for targeted attack. The analysis for untargeted attack has a similar form and is deferred to Theorem in Appendix [A]
Theorem 1 (Lower Bound on Targeted Attack Transfcrabilityy. Assume both models F and G are 3-smooth. Let Ar be an (a. F-effecctive targeted attack with perturbation ball ||ō|| I c and target label y Y. The transferabiity can be lower bounded by
Pr (T, 9, ve) = 1) 2 (1-a) tns) <(1 + a) + cF(1 - a) <(1 a)
- (25((((),)))
Cg +E
Cg +e
where
min Eo (AA(())), (x) =2212
min (g(Ar(>), v) lo(x, Uo) - 362/2
CF max
yEY
min
yEV
[V ve)1
Valg(I,
Here qa, mg are the risks of models F and 9 respectively.
We defer the complete proof in Appendix C In the proof, we first use a Taylor expansion to introduce the gradient terms, then relate the dot product with cosine similarity of the loss gradients, and finally use Markov's 's inequality to derive the misclassification probability of G to complete the proof.
Implications. In Theorem 1 the only term which correlates both F and G is S(CF,, fq), while all other terms depend on individual models F or G. Thus, study the relation between S(Ex, Eg) and Pr (T, 9, yt) = 1).
Note that since 3 is small compared with the perturbation radius e and the gradient magnitude [V in the denominator is relatively large, the quantity eg is small. Moreover, large since the attack is typically effective against F. Thus, Pr (T, (FF,G,, G, = 1) has the form
a is
C' C S(Cr,lg) where C' and & are both positive constants. We can easily observe the positive correlation between the loss gradients similarity S(Lr,fg), and lower bound of adversarial transferability Pr (T, G, yt) = 1).
In the meantime, note that when / increases (i.e., model becomes less smooth), in the transferability lower bound C - k-1 S(L,, the C' decreases and increase. As a result, the lower bounds in Theorem | decreases, which implies that when model becomes less smoother (i.e., 3 becomes larger), the transferability lower bounds become looser for both targeted and untargeted attacks. In other words, when the model becomes smoother, the correlation between loss gradients similarity and lower bound of transferability becomes stronger, which motivates us to constrain the model smoothness to increase the effect of limiting loss gradients similarity.
In addition to the bo-bddde attacks, we also derive transferability lower bound for general attacks whose magnitude is bounded by total variance distance of data distributions. We defer the detail analysis and discussion Appendix B
2.4 Upper Bound of Adversarial Transferability
We next aim to upper bound the adversarial transferability. The upper bound for target attack is shown below; and the one for untargeted attack has a similar form in Theorem 4 in Appendix A
Theorem 2 (Upper Bound on Targeted Attack Transferability). Assume both models F. and G are
3-smooth with gradient magnitude bounded by B, i.e,, [|Valz(x, y)ll S B and V [ y) S B
for any x E X.y y E y. Let AT be an (a, -effeectte targeted attack with perturbation ball ||0|| <
5

---

1 (o)
and target label y V. When the attack radius is small such that min cB 1+
Be² the transferabiliity can be upper bounded by
Ex + 6
Pr (I, =
1 (1+1143(0r00))
mmin cB
Bc²
=EX where min = min (f>(a, Eo( 8c)). Here Ex and Eg are the empirical risks of models F and G
respectively, defined relative to a differentiable loss.
We defer the complete proof Appendix D In the proof, we first take Taylor expansion the loss function at (I, y). then use the fact that the attack direction will be dissimilar with at least one of the model gradients to upper bound the transferability probability.
observe that along with the increase of ((Cr, the denominator decreases and henceforth the upper bound increases. Therefore, S(E=. f-pt.pperpp upper loss gradient similarity and the upper bound of transferability probability is positively correlated. This tendency is the same that in the lower bound. Note that a does not appear in upper bounds since only completely successful attacks (a- 0%) needs to be considered here to upper bound the transferability.
Implications. In Theorem
Meanwhile, when the model becomes smoother (i.e., 3 decreases), the transferability upper bound decreases and becomes tighter. This implication again motivates us to constrain the model smoothness. We further observe that smaller magnitude of gradient, i.e., B, also helps to tighten the upper bound. We will regularize both B and 33 to increase the effect of constraining loss gradients similarity.
Note that the lower bound and upper bound jointly show smaller / leads to reduced gap be- tween lower and upper bounds and thus a stronger correlation between loss gradients similarity and transferabiltiy. Therefore, it is important to both constrain gradient similarity and increase model smoothness (decrease 3) to reduce model transferability and improve ensemble robustness.
3 Improving Ensemble Robustness via Transferability Minimization
Motivated by our theoretical analysis, propose lightweight yet effective robust ensemble training approach, Transfcrability Reduced Smooth (TRS), to reduce the transferability among base models by enforcing low loss gradient similarity and model smoothness the same time.
3.1 TRS Regularizer
In practice, it is challenging to directly regularize the model smoothness. Luckily, inspired from deep learning theory and optimization [14] 37 45), succinct &2 regularization on the gradient terms gradients and thus improve model smoothness. For example, for common neural networks, the smoothness can be upper bounded via bounding the l2 magnitude of gradients [45] Corollary 4J. An intuitive explanation is that, the & regularization on the gradient terms reduces the magnitude of model's weights, thus limits its changing rate when non-linear activation functions are applied to the neural network model. However, we find that directly regularizing the loss gradient magnitude with & norm is not enough, since vanilla lz regularizer such as /ullll will only focus on the local region at data point æ. while it is required to ensure the model smoothness over large decision region control the adversarial transferability based on our theoretical analysis.
[|V2&=ll2 and ||V can reduce the magnitude
To address this challenge, propose a min-max framework to regularize the "support" instance ă with "worst" smoothness in the neighborhood region of data point x, which results in the following model smoothness loss: Lsmorth (F,G,I,))
max
|V + VV
(1)
| $8
where 5 refers to the radius of the Exc ball around instance within which we aim to ensure the model to be smooth. practice, we leverage projection gradient descent optimization to search for support instances á for optimization. This model smoothness loss can be viewed promoting margin-wise smoothness, i.e., improving the margin between nonsmooth decision boundaries and data point Another option is to promote point-wise smoothness that only requires the loss landscape

---

at data point z itself to be smooth. We compare the ensemble robustness of the proposed min-mat framework which promotes the margin- sarg wise smoothness with the naive baseline which directly applies l2 regularization on each model loss gradient terms to promote the point-wise smoothness (i.e. Cos-l2) in Section
Given trained "smoothed" base models, we also decrease the model loss gradient similarity to reduce the overall adversarial transferability between base models. Among various metries which measure the similarity between the loss gradients of base model F and G. we find that the vanilla cosine similarity metric, which is also used in 23] may lead to certain concerns. By minimizing the cosine similarity between V.L5 and V Eg. the optimal case implies &zfe = - -v,kg which means two models have contradictory (rather than diverse) performance on instance x and thus results in turbulent model functionality. Considering this challenge, we leverage the absolute value of cosine similarity between v,tr and V. as similarity loss f.sm and its optimal case implies orthogonal loss gradient vectors. For simplification, we will always use the absolute value of the gradient cosine similarity as the indicator of gradient similarity our later description and evaluation.
Based on our theoretical analysis and particularly the model loss gradient similarity and model smoothness optimization above, we propose TRS regularizer for model pair (F,9) on input as:
(ons(F,,,,, 6) da= Lsim Ao= Lamooh
(v+l=) (Volg)
+ >b
max ||0- Só
| |1V
| 1V2lgl22
Here &a and Eglg refer to the loss gradient vectors of base models F and 9 on input x. and Aa: As the weight balancing parameters.
In Section 4 backed up by extensive empirical evaluation, we will systematically show that the local min-max training and the absolute value of the cosine similarity between the model loss gradients significantly improve the ensemble model robustmess with negligible performance drop on benign accuracy, as well as reduce the adversarial transferability among base models.
3.2 TRS Training
We integrate the proposed TRS regularizer with the standard ensemble training loss, such as Ensemble Cross-Entropy (ECE) loss, to maintain both ensemble model's classification utility and robustness by varying the balancing parameter A, and λs Specifically, for ensemble model consisting of N base models { given an input (It, y). our final training loss train is defined as:
Lurin
Σ CIRS(FF, Σ (C)()) N(N 1) i= jmit1 N
1
2
Σ
where LCE (F,(x), y) refers to the cross-entropy loss between F;(x), the output vector of model F; given x. and the ground-trutt -ruth label y. The weight of LIRS regularizer could be adjusted by the tuning da and A internally. We present one-epoch training pseudo code in Algorithm T of Appendix [F] The detailed papper-rrrmmter setting and training criterion are discussed in Appendix F
4 Experimental Evaluation
In this section, we evaluate the robustness of the proposed TRS-ensemble model under both strong whitebox attacks, as well as blackbox attacks considering the gradient obfuscation concern (T] We compare TRS with six sate-oof-the-ar ensemble approaches. In addition, we evaluate the adversarial transferability among base models within an ensemble and empirically show that the TRS regularizer can indeed reduce transferability effectively. We also conduct extensive ablation studies to explore the effectiveness of different loss terms in TRS, as well as visualize the trained decision boundaries of different ensemble models to provide intuition on the model properties. We open source the code and provide a large-scale benchmark.
4.1 Experimental Setup
Datasets. We conduct our experiments widely-used image datasets including hand-writte dataset MNIST 29 ; and colourful image datasets CIFAR-10 and -1FAR-100 26
http:////gthhoooo..aaaeeetataaas Smooth Ensemble
e

---

Table 1: Robust accuracy (%) of different ensembles against whitebox attacks MNIST/CIFAR-ID 10. "para." refers to the attack parameter (E is the Lo perturbation budget for the attack and the constant to balance the attack stealthiness and effectiveness). The first 6 methods are baseline ensembles, and the last 3 columns (Cos-only, Cos- f2. TRS) the variants of TRS-ensemble
MNIST
par.
AdaBooal
GoodennBosst
CKAE
ADP
GAL
DVERGE
Cos-anly
Cos-f2
TRS
FGSM
022
70.2
39.4
34.2
73.2
42.5
72.6
71.7
20.0
35.7
7.8
91.6
95.8
66.2
50.7
72.5
91.2
91.7
95.6
BIM (50)
0.15
0.1
0.0
2.6
2.4
4.2
1.7
4.6
2.5
47.7
74.1
04
0.0
47.9
76.2
93_11
85.7
PGD (50)
- 0.15
0.0
0.0
0.5
1,0
0.6
4.1
28.8
0.0
30.2
85.1
MIM (50)
0.15
0.1
0.0
1.9
0.1
1.6
033
1.2
13.8
1.0
0.2
0.8
44.6
75.3
0.4
0.0
74.1
35.5
85.1
92.9
66.3
65.8
69.5
90.1
68.3
79.2
58.6
54.4
92.6
EAD
e= 5.0
10.0
0.2
0.0
15.T
00.0
15.
00.0
0.0
2.2
0.2
0.2
D30
0.0
4.1
0.5
6.9
0.8
2.3.3
1.4
APGD-DLR
0.5
2.1
27.4
26.3
70.6
92.1
APGD-CE
0.15
0.1
0.0
0.2
0.2
0.00
0.1
0.1
0.4
1.4
0.2
1.2
26.1
63.2
0.0
0.0
25.4
69.8
91.7
82.8
CIFAR-10
para.
AdaBoost
GradiemBoost
CKAE
ADP
GAL
DVERGE
Cos-only
Cos-é,
TRS
FGSM
0,02
28.2
15.4
30.4
15.2
18.5
54.1
58.8
39.4
12.6
19.2
63.8
53.4
56.1
35.0
25.9
35.8
24.9
44.2
BIM (50)
0.02
0.2
0.1
0.2
13.8
0.9
2.5
39.1
13.0
0.0
1.2
50.6
15.8
PGD (50)
0.01
2.1
1.9
1.9
9.0
8.3
37.1
0.0
15.7
50..5
0.02
0.0
0.2
0.1
10.5
0.0
0.5
15.1
51.5
MIM (50)
0.02
0.1
1.7
0.8
14.4
0.0
0.5
17.2
CW
0.01
36.2
18.1
26.2
35.2
23.0
35,4
55.8
25.9
06.3
28.3
75.1
57.4
36.6
17.6
30.7
67.3
58.1
77.2
RAD
c
0.0
0.0
0.0
0.2
0.0
0.0
11.7
0.1
APGD-DER R
= 0.01
0.02
0.0
1.2
0.9
0.0
1.1
5.5
0.2
2.2
37.6
10.2
0.0
16.1
0.5
50.2
15.1
APGD-CE
37.5
48.0
0.02
0.0
0.0
0.0
0.1
0.0
10.2
0.0
0.5
15.0
Baseline ensemble approaches. We mainly consider the standard ensemble, well the state-of- the-art robust ensemble methods that claim to be resilient against adversarial attacks. Specifically, we consider the following baseline ensemble methods which aim to promote the diversity between base
models: AdaBoost [19; GradientBoost [16]: CKAE [25 ADP 38); GAL [23]: DVERGE (56).
The detailed description about these approaches are Appendix E DVERGE, which has achieved the statc-of-the-an ensemble robustness to our best knowledge, serves as the strongest bascline.
Whitebox robustness evaluation. We consider the following adversarial attacks measure en- sembles' whitebox robustness: Fast Gradient Sign Method (FGSM) [17]; Basic Iterative Method (BIM) 34]; Momentum Iterative Method (MIM); Projected Gradient Descent (PGD); Auto-PGD (APGD); Carlini & Wanger Attack (CW); Elastic-net Attack (EAD) g and we leave the detailed description and parameter configuration of these attacks in Appendix our evaluation metric for the whitebox setting, defined as the ratio of correctly predicted adversarial examples generated by different attacks among the whole test dataset.
B We use Robust Accuracy as
Blackbox robustness evaluation. We also conduct blackbox robustness analysis in our evaluation since recent studies have shown that robust models which obfuscate gradients could still be fragile under blackbox attacks [T] In the blackbox attack setting, we assume the attacker has no knowledge about the target ensemble, including the model architecture and parameters. In this case, the attacker is only able to craft adversarial examples based on several surrogate models and transfer them to the target victim ensemble. We follow the same blackbox attack evaluation setting in (56]: We choose three ensembles consisting of 3, 5,8 base models which are trained with standard Ensemble Cross-Entropy (ECE) loss our surrogate models. We apply 50-steps PGD attack with three random starts and two different loss functions (CrossEntropy and CW loss) on each surrogate model to generate adversarial instances (i.e. for each instance we will have 18 attack attempts). For each instance, among these attack attempts, as long as there is one that can successfully attack the victim model, we will count metric, defined as the number of unsuccessful attack attempts divided by the number of all attacks. We also consider additional three strong blackbox attacks targeting on reducing transferability (i.c.. ILA 21. DI2-SGSM [55], IRA (50) in Appendix J which leads to similar observations.
successful attack. In this case, we use Robust Accuracy as our evaluation
8

---

Stockoo accuracy spaist trarffer attace MNITT
Blackhor accurscy against transfer stback on OFAR-10
143
-o- Vell
-0-
-0- 115
192
pertu botennnonnnno scale
porturbation scalo
Figure 2: Robust accuracy under blackbox attacks with different Eor perturbation budget e. (Leff): MNIST;
(Right): CIFAR-10.
4.2 Experimental Results
In this section, we present both whitebox and blackbox robustness evaluation results, examine the adversarial transferability, and explore the impacts of different loss terms in TRS. Furthermore, in Appendix L.I we visualize the decision boundary; in Appendix [.2 we show results of further improving the robustness of the TRS ensemble by integrating adversarial training; in Appendix [.3] we study the impacts of each of the regularization term Lsim and Lamooth; in Appendix [.4 we show the convergence of robust accuracy under large attack iterations to demonstrate the robustness stability of TRS ensemble; in Appendix [.5] we analyze the trade-off between the training cost and robustness of TRS by varying PGD step size and the total number of steps within smooth approximation.
Whitebox robustness. Table presents the Robust Accuracy of different ensembles against range of whitebox attacks on MNIST and CIFAR-10 10 dataset. We defer results on CIFAR- 100 in Appendix K and measure the statistical stability of our reported robust accuracy in Appendix H Results shows that the proposed TRS ensemble outperforms other baselines including the state-of-the-ar DVERGE significantly, against a range of attacks and perturbation budgets, and such performance gap could be even larger under stronger adversary attacks (e.g. PGD attack). We note that TRS ensemble is slightly less robust than DVERGE under small perturbation with weak attack FGSM. We investigate this based on the decision boundary analysis in Appendix L.I. and find that DVERGE tends to be more robust along the gradient direction and thus more robust against weak attacks which only focus on the gradient direction (c.g., FGSM): while TRS yields a smoother model along different directions leading more consistent predictions within a larger neighborhood of an input, and thus more robust against strong iterative attacks (e.g.. PGD). This may be due to that DVERGE is essentially performing adversarial training for different base models and therefore it protects the adversarial (gradient) direction, while TRS optimizes to train smooth ensemble with diverse base models. We also analyze the convergence attack algorithms Appendix [.4] showing that when the number of attack iterations is large, both ADP and GAL ensemble achieve much lower robust accuracy against such iterative attacks; while both DVERGE and TRS remain robust.
Blackbox robustness. Figure 2 shows the Robust Accuracy performance of TRS compared with different baseline ensembles under different perturbation budget e. As we can see, the TRS ensemble achieves competitive robust accuracy with DVERGE when e is very small, and TRS beats all the baselines significantly when c large. Precisely speaking. TRS ensemble achieves over 85% robust accuracy against transfer attack with c = 0.4 on MNIST while the second-best ensemble (DVERGE) only achieves 20.2%. Also on CIFAR- 10, TRS ensemble achieves over 25% robust accuracy against transfer attack when 0.06, while all the other baseline ensembles achieve robust accuracy lower than 6%. This implies that our proposed TRS ensemble has stronger generalization ability in terms of robustness against large adversarial attacks compared with other ensembles. We also put more details of the robust accuracy under blackbox attacks in Appendix G
Adversarial transferability. Figure 3 shows the adversarial transferabilityy matrix of different ensembles against 50-steps PGD attack with e 0.3 for MNIST and 0.04 for CIFAR-10. Cell (i,j) where i * represents the transfer attack success rate evaluated on j-th base model by using the i-th base model as the surrogate model. Lower number in each cell indicates lower transferability and thus potentially higher ensemble robustness. The diagonal cell (i. i) refers to i-th base model's 's attack success rate, which reflects the vulnerability of a single model. From these figures, we can see that while base models show their vulnerabilities against adversarial attack, only DVERGE and TRS ensemble could achieve low adversarial transferability among base models. We should also notice that though GAL applied similar gradient cosine similarity loss our loss term Csims GAL still can

---

1823
29.00
1:0-0
19.2
15.5
1000
100.0
66.2
100.3
37.6
C1.
30002
11.0
66.3
24.8
90.4
95.5
69
J0M
21-8
85.3
19
174
100
45.4
20:3
59
Base modell
Boet mrde
Den nodel
Cest nodd
OM
68.5
50.3
1020
92.9
33.0
90.7
62.5
60.4
59.5
70.6
14.6
1000
24.2
24.8
2002
15.2
Figure 3: Transfcrability analysis of PGD attack on MNIST (top) and CIFAR-10 10 (bottom). Each cell (i., j) shows the attack success rate of i-th model on the adversarial examples generated against the j-th model. We 0.3 for MNIST and 0.04 for CIFAR-10.
not achieve low adversarial transferability due to the lack of model smoothness enforcement, which one of key contributions in this paper.
Gradient similarity only vs. TRS. To further verify our theoretical analysis on the sufficient condi- tion of transferability as model smoothness, consider only applying similarity loss Lsim without model smoothness loss Smooth in TRS (i.e. A = 0). The result is shown as "Cos-only" method of Table I We observe that the resulting whitebox robustness is much worse than standard TRS. This matches our theoretical analysis that only minimizing the gradient similarity cannot guarantee low adversarial transferability among base models and thus lead low ensemble robustness. In Appendix [.3] we investigate the impacts of Lsim and Lamooth thoroughly, and we show that though mmooth contribute slightly more, both terms are critical to the final ensemble robustness.
l2 regularizer only vs. Min-max model smoothing. To emphasis the importance of our proposed min-max training loss promoting the margin-wise model smoothness, train variant of TRS ensemble Cos-Ez, where we directly apply the &g regularization on | and || V V l|2. The results are shown "Cos-lz" in Table We observe that Cos-èz achieves lower robustness accuracy compared with TRS, which implies the necessity of regularizing the gradient magnitude on not only the local training points but also their neighhorhood regions ensure overall model smoothness.
5 Conclusion
In this paper, we deliver an in-depth understanding of adversarial transferability. Theoretically, we provide both lower and upper bounds on transferability which shows that smooth models together with low loss gradient similarity guarantee low transferability. Inspired by our analysis, we propose TRS ensemble training to empirically reduce transferability by reducing loss gradient similarity and promoting model smoothness, yielding significant improvement ensemble robustness.
Acknowledgments and Disclosure of Funding
This work is partially supported by the NSF grant No. 1910100, NSF CNS 20-46726 CAR, the Amazon Research Award, and the joint CATCH MURI-AUSMURI.
References
[1] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning. pages 274 283. PMLR, 2018.
[2] Jonathan Borwein and Adrian S Lewis. Convex analysis and nonlinear optimization: theory and examples. Springer Science & Business Media, 2010.
[3] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
10

---

[4] Wieland Brendel, Jonas Rauber, and Matthias Bethge. recision-basee adversarial attacks: Reliable attacks against black-box box machine learning models. In International Conference on Learning Representations, 2018.
[5] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In
2017 IEEE Symposium on Security and Privacy (SP), pages 39-57. IEEE, 2017.
[6] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: /902. 2019.
[7] Antonin Chambolle. An algorithm for total variation minimization and applications. Journal of Mathematical imaging and vision, 20(1):89-97, 2004.
[8] Pin-Yu Chen, Yash Sharma, Huan Zhang. Jinfeng Yi, and Cho-Jui Hsieh. Fad: elastic-net attacks to deep neural networks via adversarial examples. Thirty-second AAAI conference on artificial intelligence, 2018
[9] Shuyu Cheng, Yinpeng Dong, Tianyu Pang. Hang Su, and Jun Zhu. Improving black-box adversarial attacks with transfer-basee prior. In Advances in Neural Information Processing Systems, pages 10932-10942, 2019.
[10] Sunoh Choi, Jangseong Bae, Changki Lee, Youngsoo Kim, and Jonghyun Kim. Attention-based automated feature extraction for malware analysis. Sensors, 20(1))28893, 2020.
[11] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, PMLR, 2020.
pages
[12] Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In 28th USENIX Security Symposium (USENIX Security 19), pages 321-338, 2019.
[13] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9185-9193, 2018.
[14] Harris Drucker and Yann Le Cun. Improving generalization performance using double back- propagation. IEEE Transactions on Neural Networks, 3(6):991-997, 1992.
[15] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-worke attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1625-1634, 1634, 2018.
[16] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
[17] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint 1rXiv:1412.6572, 2014.
[18] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv: 2014.
[19] Trevor Hastie, Saharon Rosset, Ji Zhu, and Hui Zou. Multi-class adaboost. Statistics and its Interface, 2(3):349-365 -300, 2009.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
11

---

[21] Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. In Proceedings of the IEEE International Conference on Computer Vision, pages 9733-4742,, 2019.
[22] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pages 125-136 2019.
[23] Sanjay Kariyappa and Moinuddin K Qureshi. Improving adversarial robustness of ensembles with diversity training. arXiv preprint arXiv:1901.00 .09981, 2019.
[24] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of 3rd International Conference for Learning Representations, 2015.
[25] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pages 3519-3529. PMLR, 2019.
[26] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009.
[27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[28] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world, arXiv preprint arXiv: 1607.0050.. 2016.
[29] Yann LeCun. The mnist database of handwritten digits. http://yaam.lccu..com/xx//mmmii 1998.
[30] Huichen Li, Xiaojun Xu, Xiaolu Zhang, Shuang Yang, and Bo Li. Qeba: Query-efficient boundary-based blackbox attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1221-1230, 2020.
[31] Huichen Li, Linyi Li, Xiaojun Xu, Xiaolu Zhang. Shuang Yang, and Bo Li. Nonlinear projection based gradient estimation for query efficient blackbox attacks. In International Conference on Artificial Intelligence and Statistics, pages 31 3142-3150 PMLR, 2021.
[32] Yen-Chen Lin, Zhang- Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming- Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 3756-3762, 2017.
[33] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. arXiv preprini 1rXiv:161..02770 2016
[34] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
[35] Llew Mason, Jonathan Baxter, Peter Bartlett, and Mareus Frean. Boosting algorithms as gradient descent. Advances in neural information processing systems, 12:512-518, 1999.
[36] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Univer- sal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765-1773, 2017.
[37] Adam M Oberman and Jeff Calder. Lipschitz regularized deep neural networks converge and generalize. arXiv preprint arXiv:1808. 0000000000 2018
[38] Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning, pages 4970-4979. PMLR, 2019.
12

---

[39] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learn- ing: from phenomenn to black-box attacks using adversarial samples. arXiv preprint arXiv: 1605.007777 2016.
[40] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pages 372-387. IEFE, 2016.
[41] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Anan- thram Swami. Practical black-box attacks against deep learning systems using adversarial examples. In Proceedings of the 2017 ACM Asia Conference on Computer and Communications Security, 2017.
[42] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Anan- thram Swami. Practical black- box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pages 506-519. ACM, 2017.
[43] Robert E Schapire. The strength of weak learnability. Machine learning, 5(2):197-227, 1990.
[44] Sahil Singla and Soheil Feizi. Second-order provable defenses against adversarial attacks. In International Conference Machine Learning, 2020.
[45] Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018.
[46] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3 10443112, 2014.
[47] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
[48] Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint 1rXiv:1704.000 2017.
[49] Florian Tramer, Nicholas Carlini, Wicland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems, 33, 2020.
[50] Xin Wang. Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. unified approach to interpreting and boosting adversarial transferability. In International Conference on Learning Representations, 2020.
[51] Chaowei Xiao, Bo Li, Jun- Zhu, Warren He, Mingyan I Liu, and Dawn Song. Generating adversarial examples with adversarial networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 3905-3911 1, 2018.
[52] Chaowei Xiao, Bo Li, Jun- Yan Zhu, Warren He, Mingyan I Liu, and Dawn Song. Generating adversarial examples with adversarial networks. In Proceedings of the 27th International Joint Conference Artificial Intelligence, pages 3905-3911, 2018.
[53] Chaowei Xiao, Jun- Yan Zhu. Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In International Conference on Learning Representations, 2018.
[54] Chaowei Xiao, Jun Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In 6th International Conference Learning Representations, ICLR 2018, 2018
[55] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L. Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2730-2739, 2019.
13

---

[56] Huanrui Yang. Jingyang Zhang, Hongliang Dong, Nathan Inkawhich, Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. Dverge: Diversifying vulnerabilities for enhanced robust generation of ensembles. Advances in Neural Information Processing Systems, 33, 2020.
[57] Jiawei Zhang, Linyi Li, Huichen Li, Xiaolu Zhang, Shuang Yang, and Bo Li. Progressive boundary blackbox attack via projective gradient estimation. arXiv preprint arXhv:2106.06055 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(C) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you'ne using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "roaWgwZSaS7", "reviewer": "Reviewer_ABqv", "summary": "This paper proposes to enforce the smoothness of the base models in addition to reducing the loss gradient similarity between them to improve ensemble model robustness. This is motivated by theoretical analysis for adversarial transferability and verified by extensive experiments. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To the best of my knowledge, the theoretical analysis and proposed method are novel. The experiments also support the central claim about improving ensemble model robustness. \n\nHowever, the central concern I have is that: Is the proposed method essentially trading model performance for the \"robustness\"? After all, two completely different random guess models probably cannot transfer to each other, but such models are also useless. The proposed objective optimizes the base models to have smooth orthogonal decision boundaries, so that they are less adversarial examples are transferable between them. But this probably also makes at least one of them have much worse performance with such different decision boundaries as shown in figure 1. Then how can the ensemble still maintain the high performance?  On the other hand, previous works on promoting base model diversity to improve robustness still maintain that base models make the same predictions, i.e. having similar decision boundary, but potentially different confidence score distribution, such as in Improving Adversarial Robustness via Promoting Ensemble Diversity. \n\nThere are also several other issues\n- It would be more convincing to test against stronger black-box attacks, such as \n    - ILA (Enhancing adversarial example transferability with an intermediate level attack)\n    - DI2-FGSM (Improving Transferability of Adversarial Examples with Input Diversity)\n    - Interaction-Reduced attack (A Unified Approach to Interpreting and Boosting Adversarial Transferability)\n- Standard deviations are not reported in table 1, but the error bar is claimed to be reported in the checklist.\n- In Definition 4, the text on 159 suggesting that x and y should be paired input and true labels. But the notation in the formula could be interpreted otherwise. This should be clarified, maybe by qualifying (x,y) as a pair under the inf and sup. Similarly for other places applicable. In general, it would be better to use a different symbol for the qualified y that is potentially not the same as the ground truth label, such as the ones in Theorem 1.\n-  On line 326, DIVERGE -> DVERGE\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author pointed out the limitation that \"TRS ensemble is slightly less robust than DIVERGE under small perturbation with weak attack FGSM\". If the method indeed significantly decreased the model performance over the normal dataset, it should also be noted as a limitation and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "WY7xQPIoBuL", "reviewer": "Reviewer_zHq1", "summary": "The manuscript describes the problem of adversarial transferability, the process of adversarial attacks that can be transferred across models, which can help scale adversarial attack strategies. As a guardrail against such attacks, it is desired that models are robust and protected against such attacks. Towards solving this, the authors first derive a lower and upper bound on a transferability metric that they define. The similarity between gradient losses was found to be the salient cross-model term affecting adversarial transferability. Subsequently, this pointed to a strategy of reducing model smoothness and loss gradient similarity across models as a way to reduce adversarial transferability.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Motivated by reducing model smoothness, the authors propose a Transferability Reduced Smoothness (TRS) approach. The authors define adversarial attacks, both untargeted (goal is to mislabel) and targeted (goal is to specify an incorrect label). The definitions 1-5 and derivations are clear. The two central metrics driving the developments in the paper are the loss gradient similarity, which describes similarity between the gradient of losses for two models, and a beta-smoothness, which defines when a model is beta-smooth. \n\nThe two loss functions proposed, for smoothness and similarities, are fairly novel and appear to work well for the benchmarks. My two big concerns are that the experiments do not have the same level of clarity and detail as the theoretical counterpart, and the real-world use case of preventing adversarial attacks from an unknown source model are not addressed. My detailed comments below:\n\nOne key weakness in the utility of such a framework is that the training is across pairs of models while in reality the adversarial transferability has to be against unknown source models. I believe the authors should also look at source model agnostic solutions to make their proposed work useful in real-world settings.\n\n“Luckily, inspired from deep learning theory and optimization [14, 36], succinct l2 regularization on the gradient terms can reduce the magnitude of gradients and thus improve model smoothness.“ This statement is not well supported and appears to be a very crude approximation to the beta-smoothness metric (Definition 4). Equation 1 only has weak similarity between the two models F and G. \nWhich of Lsim or Lsmooth is contributing more to the losses?\n\nThe real meat of the paper starts only on page 5. I think the authors should condense pages 1-4 to within three pages to get to the new developments of the paper faster.\n\nExperimental results section is rushed and comprises only 1 page. It is concerning that in Fig. 3, the results of the proposed method for some model pairs are significantly lower than the DVERGE baseline method.\n\nI have read the authors' rebuttals as well as the other reviews.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss limitations of their approach. Is truncating the Taylor series with the first term correct? What about higher order terms, which might come into play when source models are unknown. The methods developed in general will have a positive societal impact by reducing adversarial attacks.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "PP5Xq2AHjNp", "reviewer": "Reviewer_73Cj", "summary": "1. This paper provides theoretical analysus on the sufficient condition on the adversarial transferability between ML models, and identify the importance of both gradient orthogonality and model smoothness\n\n2. This paper proposes a simple smoothness regularization term and merge it with gradient diversity objective to form TRS robust ensemble training algorithm\n\n3. Thorough experiments are provided to show TRS achieves state-of-the-art ensemble robustness", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides solid theortical analysis on the transfer robustness of deep learning models, and accordingly propose ensemble training methods to reduce the transferability between sub-models and achieve robust ensemble model. Overall I enjoy reading this paper and find it interesting. Here are the strength and weakness of the paper:\n\nStrength:\n1. The theortical analysis on the transferability bound is solid, and its the first work to identify the importance of model smoothness in diverse ensemble training, making the proposed TRS training method well motivated.\n2. The experiment results support the theortical claims by showing the ensemble trained with TRS can achieve SOTA transferability robutness and ensemble robustness.\n3. The paper is well written and easy to follow.\n\nWeakness:\n1. There lacks a discussion on the additional training cost introduced by optimizing the smoothness loss. See the limitation section for details.\n2. It should be noted that model smoothness has already been introduced in previous works on improving the robustness of deep learning model. One representative work is CURE [1], which also use a form of gradient regularization to minimize local curvature of deep learning model for robusntess improvement. I think it would be important for the author to cite this paper and discuss the difference in the smoothness objective.\n3.  Though I appreciate the finding of the importance of model smoothness , how exactly the smoothness regularization is contributing to the overall ensemble robustness is unclear. The author suggest the smoothness is helpful on achieving better model diversity. However, as shown in CURE, adding smoothness regularization is also an effective way to increase the robustness of individual model. So it's likely that TRS is achieving higher robustness against larger perturbation due to each sub-mode is more robust. This hypothesis could also be supported by the fact that the the curve of TRS in Fig 2 looks more like DVERGE+ADVT (lower in low perturbation and larger in strong perturbation, see Fig 5 in DVERGE paper). This may also explain why higher transferability between sub-models is shown in Fig 3 as all of them learn robust features. Redoing Fig 3 with a smaller perturbation strength may unveil the difference in sub-model robustness between TRS and DVERGE, also a result with only the smoothness loss but not the cos loss would be helpful.\n\n[1] Moosavi-Dezfooli, Seyed-Mohsen, et al. \"Robustness via curvature regularization, and vice versa.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n\nIn summary, I think this is a novel and solid theortical work supported by thorough experimental evaluation. Though there are concerns on whether the proposed smoothness objective is optimally designed and how the smoothness is contributing to the overall robustness exctly, I do think this can be a good starting point with high potential of inspiring future works on investigating how smoothness can help on achieving more robust ensemble models. So I would suggest accepting this submission, with the expectation that the authors can respond to my concerns in the discussion period.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One potential limitation of the proposed method is the computation cost of optimizing the min-max smoothness objective in Equ. (1). Given the large performance margin between using the min-max objective vs. using vanilla L2, it is likely that the perofrmance of TRS is determined by how well the inner maximization is optimized, yet the discussion on this part is lacking in the paper. I would suggest the author to\n1. Provide detailed information on the hyperparameter (i.e. step size and number of steps for the inner maximization) of the smoothness loss and discuss its impact on the overall training cost\n2. If possible provide an ablation study on how these hyperparameters affect the final performance of TRS, is there a tradeoff in the training cost vs. robustness? ", "review_text": "", "extra_scores": {"ethical_concerns": "There's no ethical concern related to this submission", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "8ac09C05GxG", "reviewer": "Reviewer_gXTn", "summary": "This work first theoretically analyzes and outlines sufficient conditions for adversarial transferability between models, then proposes a practical algorithm to reduce the transferability between base models within an ensemble to improve its robustness. They also provide the lower and upper bounds of adversarial transferability under certain conditions and propose an effective Transferability Reduced Smooth (TRS) ensemble training strategy to train a robust ensemble with low transferability. Extensive experiments demonstrate that the proposed TRS outperforms all baselines significantly.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n\t+They make the first attempt towards a theoretical understanding of adversarial transferability, and provide an approach for developing robust ML ensembles.\n\t+A theoretical analysis is provided, which helps us to understand the transferability of adversarial examples between different models.\n\t+Extensive experiments demonstrate that the proposed TRS outperforms all baselines significantly. Ablation studies are provided.\n\n[Weaknesses]\n\t-“white-box and blackbox attacks” is used in line 62, but “whitebox and blackbox attacks” is used in line 68, please write in the same way. “theoretical understanding” -> “a theoretical understanding”.\n\t-TRS requires smaller or comparable training time (line 65). But there is no statistical comparison of training times between different methods.\n\t-∥∇_x ˆ  l_F ∥_2 and ∥∇_x ˆ  l_G ∥_2 and used In Eq. (1). As I understand, it means for training a robust model B, model A is needed to calculate the smooth loss. In the end, model B may be “robust” for transferable adversarial examples from model A, but vulnerable for adversarial examples generated from other models. This type of training is limited in practical application.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The theoretical analysis is provided but the practicability maybe weak.  I suggest to test the avaiablity in the real application.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "five"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the reason for adversarial transferability between models, and proposes an algorithm to reduce the transferability between base models. All reviewer agree that this is an interesting topic and the paper provides theoretical insight on the transferability problem. On the other side, the experimental analyses need to be strengthened. What are the additional costs for the reduction of adversarial transferability? Overall, this is an interesting paper. I recommend accept.
Decision: Accept (Poster)