A Unified Approach to Interpreting and Boosting Adversarial Transferability
Authors: Xin Wang
Jie Ren
Shuyun Lin
Xiangming Zhu
Yisen Wang
Quanshi Zhang
Venue: ICLR 2021 Poster (2021)
Keywords: Adversarial Learning
Interpretability
Adversarial Transferability

Abstract: In this paper, we use the interaction inside adversarial perturbations to explain and boost the adversarial transferability. We discover and prove the negative correlation between the adversarial transferability and the interaction inside adversarial perturbations. The negative correlation is further verified through different DNNs with various inputs. Moreover, this negative correlation can be regarded as a unified perspective to understand current transferability-boosting methods. To this end, we prove that some classic methods of enhancing the transferability essentially decease interactions inside adversarial perturbations. Based on this, we propose to directly penalize interactions during the attacking process, which significantly improves the adversarial transferability. We will release the code when the paper is accepted.

Published as a conference paper at ICLR 2021
A UNIFIED APPROACH TO INTERPRETING AND BOOST- ING ADVERSARIAL TRANSFERABILITY
Xin Wang"*, Jie Shuuun Lin", Xiangming Zhu", Yisen Wang", Quanshi Zhang "Shanghai Jiao Tong University
ò Keyy Lab. of Machine Perception (MoE), School of EECS, Peking University, Beijing, China
ABSTRACT
In this paper, we use the interaction inside adversarial perturbations to explain and boost the adversarial transferability. We discover and prove the negative cor- relation between the adversarial transferability and the interaction inside adver- sarial perturbations. The negative correlation is further verified through different DNNs with various inputs. Moreover, this negative correlation can be regarded as unified perspective to understand current transferability- tinbsoorrbblltyy-oootnnggg methods. To this end, we prove that some classic methods of enhancing the transferability es- sentially decease interactions inside adversarial perturbations. Based on this, we propose to directly penalize interactions during the attacking process, which sig- nificantly improves the adversarial transferability. Our code available online
INTRODUCTION
Adversarial examples of deep neural networks (DNNs) have attracted increasing attention in re- cent years (Ma et al.) 2018 Madry al., 2018: Wang al. 2019 Ilyas al.) 2019 Duan al.) 2020 Wu et al. 2020b; Ma et al. (021). Goodfellow et al. (2014) found the transferability of ad- versanial perturbations, and used perturbations generated on source DNN to attack other target DNNs. Although many methods have been proposed to enhance the transferability of adversarial
perturbations (Dong et al., 2018 Wu et al. 2018 2020a), the essence of the improvement of the transferability is still unclear.
This paper considers the interaction inside adversarial perturbations as new perspective interpret adversarial transferability. Interactions inside adversarial perturbations are defined using the Shapley interaction index proposed in game theory (Michel & Mare 1999. Shapley] (953). Given an input sample.r r E I" the adversarial attack aims to fool the DNN by adding an imperceptible perturbation R" on x. Each unit in the perturbation map is termed a perturbation unit. Let Ø; denote the importance of the i-th perturbation unit d: attacking. Φ, is implemented as the Shapley value, which will be explained later. The interaction between perturbation units 5; og is defined as the change of the i-th unit's importance di when the j-th unit is perturbed w.r./ the case when the j-th unit is not perturbed. If the perturbation ó, on the j-th unit increases the importance Φ; of the i- th unit, then there is a positive interaction between ó; and 6y. If the perturbation d, decreases the importance Φ;. it indicates a negative interaction.
In this paper, we discover and partially prove clear negative correlation between the transferability and the interaction between adversarial perturbation units, i.e. adversarial perturbations with lower transferability tend to exhibit larger interactions between perturbation units. We verify such cor- relation based both the theoretical proof and comparative studies. Furthermore, based on the correlation, we propose to penalize interactions during attacking to improve the transferability.
Equul contribution
1 'Correspondence. This study is conducted under the supervision of Dr. Quanshi Zhang zqs 022 juu. edu. en. Quanshi Zhang is with the John Hopcroft Center and the MoB Key Lab of Arti- ficial Intelligence, AI Institute, at Shanghai Jiao Tong University, China.
/ thub. com htt.://wwwtwweeeeeeeee wttp://hherran//aaaaa fied -ttopppchhoo Incerpreting-and
Boost loosting-AAversaaiiaaa

---

Published as a conference paper at ICLR 2021
In fact, our research group led by Dr. Quanshi Zhang has proposed game-theoretic interactions, including interactions of different orders (Zhang et al. 2020) and multivariate interactions (Zhang et al. 2021c). As basic metric, the interaction can be used to explain signal processing in trained DNNs from different perspectives. For example, we have build up a tree structure to explain the hierarchical interactions between words in NL.P models (Zhang al.] (021a). We have also used interactions to explain the generalization power of DNNs (Zhang et al. (021b). The interaction can also explain the utility of adversarial training (Ren et al. (021). As extension of the system of game- game-theoretic interactions, in this study, we explain the adversarial transferability based on interactions.
In this paper, the background for us to investigate the correlation between adversarial transferability and the interaction as follows. First, we prove that multi-step attacking usually generates pertur- bations with larger interactions than single-step attacking. Second, according to (Xie et al.) 2019), multi-step attacking tends to generate more over-fitted adversarial perturbations with lower trans- ferability than single-step attacking. We consider that the more dedicated interaction reflects more over- over-fitting towards the source DNN, which hurts adversarial transferability. In this way, we propose the hypothesis that the transferability and the interaction negatively correlated.
Comparative studies are conducted to verify this negative correlation through different DNNs.
Unified explanation. Such negative correlation provides unified view to understand current -booooiiiitty methods. We theoretically prove that some classic transfcrability methods (Dong al. [018]] Wu et al., 2018, 2020a) essentially decrease interactions between per- turbation units, which also verifies the hypothesis of the negative correlation.
transferability- bbosting
Boosting adversarial transferability. Based above findings, we propose loss to decrease interactions between perturbation units during attacking, namely the interaction loss, in order to enhance the adversarial transferability. The effectiveness of the interaction loss further proves the negative correlation between the adversarial transferability and the interaction inside adversarial perturbations. Furthermore, we also try to only use the interaction loss to generate perturbations without the loss for the classification task. We find that such perturbations still exhibit moderate adversarial transferability for attacking. Such perturbations may decrease interactions encoded by the DNN, thereby damaging the inference patterns of the input.
Our contributions are summarized as follows. (1) We reveal the negative correlation between the transferability and the interaction inside adversarial perturbations. (2) We provide unified view to understand current wannsfrabillity-boottin methods. (3) We propose new loss to penalize interac- tions inside adversarial perturbations and enhance the adversarial transferability.
RELATED WORK
Adversarial transferability. Attacking methods can be roughly divided into two categories, i.e.
white-box attacks (Szegedy al., 2013: Goodfellow al. 2014 Papernot al. 2016: Carlini & Wagner [017] Kurakin et 2017 Su et al. 2017 Madry ct al. 2018) and black- box attacks (Liu al..|2016] Papernot et al. 2017 Chen et al. 2017a Bhagoji et al. 2018 Ilyas et al. 2018 Bai al.
2020 A specific type of the black-box attack is based on the adversarial transferability (Dong et al,
2018 Wu et al.) 2018 Xie et al.
20199 Wu al. (020a), which transfers adversarial perturbations
on a surrogate/source DNN to target DNN.
Thus, some previous studies focused on the transferability of adversarial attacking. Liu et al. (2016) demonstrated that non-targeted attacks were easy to transfer, while the targeted attacks were difficult to transfer. Wu al. (2018) and Demontis et al. (2019) explored factors influencing the transferabil- ity, such as network architectures, model capacity, and gradient alignment. Several methods have been proposed to enhance the transferability of adversarial perturbations. The momentum iterative attack (MI Attack) (Dong et al. 2018)) incorporated the momentum of gradients to boost the trans- ferability. The variance-reduced attack (VR Attack) (Wu et al., 2018) used the smoothed gradients to craft perturbations with high transferability. The diversity input attack (DI Attack) (Xie et al.) 2019) applied the adversarial attacking to randomly transformed input images, which included ran- dom resizing and padding with certain probability. The skip gradient method (SGM Attack) (Wul ct al. 2020a) used the gradients of the skip connection to improve the transferability. Dong et al. (2019) proposed the ianislationinvvriinn attack (TT Attack) to evade robustly trained DNNs. Li
2

---

Published as a conference paper at ICLR 2021
2020) used the dropout erosion and the skip connection erosion to improve the transferability. In comparison. we explain the transferability based on game theory, and discover the negative correla- tion between the transferability and interactions as unified explanation for some above methods.
Interaction.
The interaction between input variables has been widely investigated. Michel & Marc 1999) proposed the Shapley interaction index based on the Shapley value (Shapley 1953) in game
theory. [Daria Sorokina (2008) defined the interaction of K input variables of additive models.
Scott Landberggg quantified interactions between each pair of input variables for tree-ensemble models. Some studies mainly focused on interactions to analyze DNNs. |Tann et al. (2018) mea- sured statistical interactions based on DNN weights. Murdoch
al. (2018) proposed to extract interactions in LSTMs by disambiguating information of different gates, and Singh et al. (2019) extended this method to CNNs. Jii et al. (2020) quantified the contextual independence of words to hierarchically explain the LSTMs. Janizek et al. (2020) extended the method of Integrated Gra- al. 2017) to quantify pairwise interactions of input features based on the Hessian matrix, which required the DNN to use the SoftPlus operation replace the ReLU operation. Chen et al. 2020)) extended the attribution in (Chen & Ji 2020) to use the Shapley interaction index the Shapley interaction index to explain and improve the transferability of adversarial perturbations.
dients (Sundararajan
to generate hierarchical explanations of NLP tasks. In comparison, in this study,
3 THE RELATIONSHI BETWEEN TRANSFERABILITY AND INTERACTIONS
Preliminaries: the Shapley value. The Shapley value was first proposed in game theory (Shapley
1953). Considering multiple players in a game, each player aims win high reward. The Shapley value is considered as a unique and unbiased approach to fairly allocating the total reward gained by all players to each player (Weber 1988). The Shapley value satisfies four desirable properties, i.e. the linearity, dummy, symmetry, and efficiency (please see the Appendix [A.] for details). Let S = (1.2,...,nI denote the set of all players, and let v(-) denote the reward function. vSS) represents the reward obtained by a set of players S C Ω. The Shapley value ((i[() unbiasedly measures the contribution of the i-th player to the total reward gained by all players in Q as follows.
|S (n - 1 (v(SU n!
= =()) w(22)vv(0))) - v0),
(11(2) = Σ SCAV())
(1)
Adversarial attack. Given input sample E 1]" with the true label y E 21, C}, we use h(z) E RC to denote the output of the DNN before the softmax layer. To simplify the story, in this study, we mainly focus on the untargeted adversarial attack. The goal of the untargeted adversarial attack is to add a human-impercepublle perturbation 5 E IR" on the sample x, and make the DNN classify the perturbed sample x' + 5 into incorrect category, Lee. arg maxy by (x') # y. The objective of adversarial attacking is usually formulated as follows.
maximize E(h(x + 8).
s.t. ||01| c, x + E (0,1))
(2)
where f(h(x + 8). y) is referred to the classification loss, and is a constant of the norm constraint.
Please see Appendix for technical details of solving Equation
3.1 THEORETICAL UNDERSTANDING OF THE ADVERSARIAL ATTACK IN GAME THEORY.
In adversarial attacking, given the perturbation d R" we use Z {1, n) to denote all units/dimensions in the perturbation. We use the Shapley value in Equation (T) to measure the contribution of each perturbation unit i E Ω the attack. To this end, it requires us define the utility of subset of perturbation units S C Ω for attacking, which can be formulated as u(S) maxy'y by (x + 5(5))- - hy(l + 5(5)), according to Equation (2). (.)()) is the value of the y-th element of l((-) E RC. ((5) E RD is the perturbation which only contains perturbation units in S, L.e. Vi E S, (5) = d,; Vi $ S. 8
0. In this way, v(2)) = max, fyy + 6) - h, : + 6) denotes the utility of all perturbation units, and v(0) maxy' (x)- denotes the baseline score without perturbations. Thus, the overall contribution of perturbation units can be measured as v(2) - ((0). We apply the Shapley value in Equation (I) to assign the overall contribution to each perturbation unit as Σ. (i()) = (S2) v0)). where (i([() denotes the contribution the i-th perturbation unit.
3

---

Published as a conference paper at ICLR 2021
Interactions. Perturbation units do not contribute to the adversarial utilityy independently. For ample, perturbation units may form a certain pattern, e.g. an edge in the image. Thus, perturbations units in the edge must appear together. The absence of few units in the pattern may invalidate this pattern. Let us consider two perturbation units j. According to (Michel & Marc 1999). the Shapley interaction index between units j, is defined as the additional contribution follows.
1gg (6) = ((5,)19)) [0(1 ()) + 6012 (i)),
where (d][S \ (i)) and Φ0i {it) represent the individual contributions of units i and j. respec- computed in the tively, when the perturbation units i, work individually. Note that @(|[( \ {j})
scenario of considering the unit always absent. Σ: ø(i[ \ (i3) a(2 (G) v(0), due to the ab- sence of perturbation unit j. O(S.y|12) denotes the joint contribution of j. when perturbation units i,j are regarded as a singleton unit Sij = {i.j}. In this case, umits j are supposed to be always perturbed or not perturbed simultaneously, and we can consider that there are only n players in the game. Thus, the set of all perturbation units is considered as D = Ω \ {i.j) } U Sty. The joint contribution of Sig is denoted by ((S.g(22), s.t. EMEr((s,)) ((' [52') + ((S, (52') = v(2') - (0).
The interaction defined in Equation (3) is equivalent to the change of the i-th unit's importance O when the unit; exists W.E. the case when the unit, is absent. Please see Appendix for details.
If (o) > O, it means 6 and 0j cooperate with each other, ie. the interaction is positive; T,,(6) O, it means 6, and by conftict with each other, i.e. the interaction is negative. The absolute value of [4: (6) indicates the interaction strength. The interaction is symmetric that (o) = (o)(
We are given an input sample x E R and a DNN iu(-) trained for classification. With the definition of interactions, in adversarial attacking, we have the following propositions:
Proposition 1. (Proof in Appendix Ep The adversarial perturbation generated by the multi-step attack via gradient descent is given 6innn = Et=g _ze(h(x 8mmta) y), where 8mmta denotes the perturbation after the t-th step of updating, and m is referred to as the total number of steps. The adversarial perturbation generated by the single-step attack is given dsingle = om V e((h()), y). Then, the expectation of interactions between perturbation units in 8y Ea (5mmat mmttaal. is larger than Ea, Lat (ōsingte]] )[, i.e. Eo, Tats (6ma))))))))) ) Z Ea, aat (as(osingic)): )-
Note that when we compare interactions inside different perturbations, magnitudes of these pertur- bations should be similar. It is because the comparison of interactions between adversarial perturba- tions of different magnitudes is not fair. Therefore, we use the step size am in the single-step attack to roughly (not accurately) balance the magnitude of perturbations. The fairness is further discussed in Appendix E.
Proposition shows that, in general, adversarial perturbations generated by the multi-step attack tend to exhibit larger interactions than those generated by the single-step attack. In addition, Ap- pendix E.4 shows that the multi-step attack usually generates perturbations with larger interactions than noisy perturbations of the same magnitude. Besides, Xie et al. (2019) demonstrated that the multi-step attack tends over-fit the source DNN, which led low transferability. Intuitively, large interactions mean a strong cooperative relationship between perturbation units, which indicates the significant over- wer-fitting towards adversarial perturbations oriented the source DNN. In this way, we propose the hypothesis that the adversarial transferability and the interactions inside adversar- ial perturbations are negatively correlated.
3.2 EMPIRICAL VERIFICATION OF THE NEGATIVE CORRELATION
To verify the negative correlation between the transferability and interactions, we conduct experi- ments to examine whether adversarial perturbations with low transferability tend to exhibit larger in- teractions than those perturbations with high transferability. Given a source DNN and an input sam- ple xr, we generate the adversarial example x' = I + 5. Then, given a target DNN 1,0), we measure the transfer utility of 8 as Transfer Utility = [maxy'#v / (x+6)- (4) (x+ [maxy'#y h (2) The interaction is given Interaction Ex, [6)] which is computed on the source DNN. Note that the computational cost of (8) is NP-hard. However, prove that we can simplify the computation of the average interaction over all pairs of units as
(t) (x) as mentioned in Section 3.1
4

---

Published as a conference paper at ICLR 2021
1224
Setinn
Inasadion
Hescenn
Intercttoon
Inttrcciion
Setting
Mar 136 313
22F
XX
Figure 1: The negative correlation between the transfer utility and the interaction. The correla- tion is computed as the Pearson correlation. The blue shade in each subfigure represents the 95% confidence interval of the linear regression.
follows, which significantly reduces the computational cost. Please see Appendix F for the proof.
RJ, (8, 72 E& [e( - v(2 (i)) - ((()) + v()
Using 50 images randomly sampled from the validation set of the ImageNet dataset (Russakovsky jet al., 2013), we generate adversarial perturbations on four types of DNNS, including ResNet- 34/152(RN-34/152 (He al.] (016) and
(Huang al.) 2017).
We transfer adversarial perturbations generated on each ResNet to DenseNets. Similarly, ly, we also transfer adversarial perturbations generated on each DenseNet to ResNets. Figure D shows the negative correlation between the transfer utility and the interaction. Each subfigure corre- sponds to a specific pair of source DNN and target DNN. In each subfigure, each point repre- sents the average transfer utility and the average interaction of adversarial perturbations through all testing images. Different points represent the average interaction and the average transfer utility computed using different hyper-parameters bations are generated by solving the relaxed form of Equation 2 via the gradient descent, i.e. ming- -(h(x + 5). v) c [o0l s.t. + 5 E 1]", where E R is scalar. In this way, we grad- ually change the value of e and set different values of e as different hyper-parameters to generate different adversarial perturbations, thereby drawing different points in each subfigure. Fair compar- isons require adversarial perturbations generated with different hyper-parameters e to be comparable with each other. Thus, we select constant and take ||ō||2= T as the stopping criteria of all adver- sarial attacks. Please see Appendix for more details.
Given input image adversarial pertur-
4 UNIFIED UNDERSTANDING OF TRANSFERABLIIT ATTACKS
In this section, we prove that some classical methods of improving the adversarial transferability essentially decrease interactions between perturbation units, although these methods are not origi- nally designed to decrease the interaction. Without loss of generality, let us be given an input sample E R" and DNN h(.)) trained for classification.
VR Attack (Wu et al., 2018 smooths the classification loss with the Gaussian noise during attacking. In the VR Attack, the gradient of the input sample is computed as follows. g Bg [V ((((z + 8' + +),,)) y)]. The following proposition proves that the VR Attack is prone to decrease interactions inside perturbation units.
Proposition 2. (Proof in Appendix H The adversarial perturbation generated by the multi-step attack is given as 6mm a Em- vel((x 8t 2 v). The adversarial perturbation generated by the VR Attack is computed Eg~N(00.02) 1) [E(h(x + +5,, + E), 8)]. Perturbation units of 8m tend to exhibit smaller interactions than 8mutti i.e. E,E.,, Lat (8 S En,.EE, ( m))
V. > Val(h(x + St, 6tr), y). where ε(h(x + (tr), = 8th a ===
Besides the theoretical proof, we also conduct experiments to compare interactions of perturbation units generated by the baseline multi-step attack (implemented as (Madry et al. 2018)) with those
We set p- = 2 as the setting 1, and = as the setting 2. To this end, the performance of adversarial perturbations is not the key issue in the experiment. Instead, we just randomly set the value to examine the trustworthiness of the negative correlation under various attacking conditions (even in extreme attacking conditions).
5

---

Published as a conference paper at ICLR 2021
of perturbation units generated by the VR Attack. T shows that the VR Attack exhibits lower interactions between perturbation units than the baseline multi-step attack.
MI Attack (Dong et al. 2018) incorporates the momentum of gradients when updating the ad- versarial perturbation. In the MI Attack, the gradient used in step t is computed as follows. g' -" V (x: + 8"-") ").)) .v) // (h (z + 8(-1) l|.
Note that the original MI Attack and the multi-step attack cannot be directly compared, since that magnitudes of the generated perturbations cannot be fairly controlled. The values of interactions are sensitive to the magnitude of perturbations. Comparing perturbations with different magnitudes is not fair. Thus,
slightly revise the MI Attack as 1I 0. 0. -'gn + - V.l(((ar +
" y); gmi = O, where μ = (t 1)/l. We investigate the interaction of adversarial perturbations generated by the original multi-step attack and the MI Attack. We prove the following proposition, which shows that the MI Attack decreases the interaction between perturbation units in most cases.
Proposition 3. (Proof in Appendix ! The adversarial perturbation generated by the multi-step Ere (2((h((+mmmtt)) y)- The adversarial perturbation generated by the multi-step attack incorporating the momentum is computed as 6m = a m- Pmi- Perturbation units of 8mi exhibit smaller interactions than 6mh mutit i.e. Ea,b (Tab([at(6m] (6m) I E., Fab (8
attack is given as multi
SGM Attack (Wu et al. 2020a) exploits the gradient information of the skip connection in ResNets to improve the transferability of adversarial perturbations. The SGM Attack revises the gradient in the backpropagation, which can be considered as to add a specific dropout operation in the back- propagation. We notice that /hang et al. (202Tb) has proved that the dropout operation can decrease the significance of interactions, so to decrease the significance of the wver-fitting of DNNs. Thus, this also proves that the SGM Attack decreases interactions between perturbation units.
Besides the theoretical proof, we also conduct experiments to compare interactions of perturbation units generated by the baseline multi-step attack (implemented as Madry et al. (2018)) with those of perturbation units generated by the SGM Attack. Table 5 shows that the SGM Attack exhibits lower interactions than the baseline multi-step attack.
5 THE INTERACTION LOSS FOR TRANSFERABILITY ENHANCEMENT
Interaction loss. Based on findings in previous sections, we propose a loss to directly penalize interactions during attacking, in order to improve the transferability of adversarial perturbations. Based on Equation (2). we jointly optimize the classification loss and the interaction loss to generate adversarial perturbations. This method is termed the interaction-reducee attack (IR Attack).
max [e(h(z + 8). v)- - MMmmeraction):
Enteraction = E., (5) s.t. ||01| e, x + E [], (5)
where Linteraction is the interaction loss, and À is a constant weight for the interaction loss. Although the computation of the interaction loss can be simplified according to Equation (4), the computa- tional cost of the interaction loss is intolerable, when the dimension of images is high. Therefore, as trade-off between the accuracy and the computational cost, we divide the input image into 16 16 grids. We measure and penalize interactions at the grid level, instead of the pixel level. Moreover, we apply an efficient sampling method approximate the expectation operation during the compu- tation of interactions in Equation (4). Figure visualizes interactions between adjacent perturbation units at the grid level generated with and without the interaction loss.
Experiments. For implementation, we generated adversarial perturbations on six different source DNNs, including Alexnet (Krizhevsky al.) 2012). VGG-16 (Simonyan & Zisserman 2015). ResNet-34/152 (RN-34/152) (He et al. 2016) and DenseNet-121/20 (DN-121/201) (Huang et al.) 2.17 For each source DNN, we tested the transferability of the generated perturbations on seven target DNNs, including VGG- 16, ResNet- M 152 (RN-152), DenseNet-201 (DN-201). SENet- 154 (SE- 154) (Hu et al., 2018), Inception V3) (Szegedy et al. 2016), Inception V4 (Inc (Szegedy) et al. 2017), and Inception-ResNervV. (IncResV2) (Szegedy et al. (017). In addition, three state- of-the-art art DNNs, including the Dhall-PathNNewoor (DPN-68) (Chen et al. 2017b), the NASNet- LARGE (NASN-L) (Zoph et al. 2018), and the Progressive NASNet (PNASN) (Liu et al.) 2018), were used as target DNNs to evaluate the ensemble source model (will be introduced in the next paragraph). Besides unsecured target DNNs mentioned above, also used three secured target

---

Published as a conference paper at ICLR 2021
Image
Interactions
Image
Interaciion
Image
Interactiona
w/l intecaction lose with ineactioon lose
wi internction lose
with inveaction Joes
We inaraction loss with interection losa
Figure 2: Visualization of interactions between neighboring perturbation units generated with and without the interaction loss. The color in the visualization is computed as colorli, x EjEN, [7y (8)), where N; denotes the set of adjacent perturbation units of the perturbation unit i. Here, we ignore interactions between non-adjacent units to simplify the visualization. It is because adjacent units usually encode much more significant interactions than other units. The interaction loss forces the perturbation to encode more negative interactions.
Table 1: The success rates of Lxx and L2 black-box attacks crafted on six source models, includ- ing AlexNet, VGG16, RN-34/152, DN-121/201, against seven target models. Transferability of adversarial perturbations can be enhanced by penalizing interactions.
Source
Method
VGG-16
RN152
DN-201
SE-154
IncV3
IncV4
IncResV2
AlexNet
PGD Loc
67.0=1.6 27.8111
32.310.4
28.210.7
29.11:1.
23.010.4
18.6.1.55
PGD Los +R
7..7=110
4..0:115
50.340.4
41.24:0.6
43.7:0.5
364:1.5
29.0:-10
VGG-16
PGD Tow
43.0+18
48.3422.0
52.9+2.7
39.3+007
49.3+1111
29.742.0
PGD Loe
63.1:116
70.0:111
71,241.5
57.6+10
6864332
49.2::12
PGD
65.44229
59.2422.
63,5333
33.1422.
27.4+3.
23.94117
21.1+111
RN-34
PGD Los +1R
84.0.10.5
84.7.1233
$8.5.00.
64.411.
56.91311
59.314,3
49.21111
RN-152
PGD Li
51.6:23.2
61.54224
33.9215
28.1209
25.0:112
22.41100
PGD Loc +1
72.3::12
82.1113
61.1109
53.61088
50.613.5
46.01123
PGD Loxe
68.6:1111
63.62t3.
86.921
46.12:1
57.3+16
37.1#2.1
28.95288
DN-121
T.
85.0=0.3
848+004
95.1-1..
70.3+1.7
61.1425
62.14-.0
53.5+0.3
DN.201
PGD
64.411:.4 67,8102
50.108
39.5133
36.510.9
34.1104
PGD Iag 1R
78.6--2.5 85.0+1111
73.900.5
61.6+188
63.7+0.6
56.442.1
AlexNet
PGD La
85.1:115
58BELO 60.24.1
55.111.
$6.023.7
49.615.
44.6133
PGD L2+1R
91.6.1111
720116
76.811
69.0110
73.010.8
63.112.11
594119
PGD
76,72099
52.3429 83,5419
69.42211
VGG-16
PGD Ia-iR
86.5+0.99
88.9+1.5 89.6+1.2
85.2+1.1
88.3+144
80.4+0.4
RN-34
PGD L2
88.2.11.4
86.210.4
89.6.13 66.91111
642.12.
60.0.1.9
58.21188
PGD Ia=IR
95.2--02
95.4+10.1
96.7+90.6
86.7+1.2
84.3+0.6
818:+199
80.4+-199
PGD Lg
89.41111 86.8.10
97.6.L. 75.611.7
70.1129
70.414.4
66.514.7
DN-121
PGD L,+IR
94.2::0.1
93.3:10.8
97.7.003 8788:0.7 84.5:0.7 84.2:0.1 82.410.1
The success rates 4. black-box attacks crafted on the ensemble model (RN-34+RN- 152+DN-121 against nine target models.
Table 2:
Source
Method
VGG-16
RN-152 DN-201 SF.154
IncV3
IncV4
TncResV2 DPN-68 NASN-L PNASN
Ensemble
PGD Loo
72.310.3 91.5.0.11 92.4.11.6 92.1.11.7 16.1 8..6 1.7 784113 82.5.100 82.3116 15.6
86.61112
99.910.1 97.010.77
70.721.6
64.220.3
57.7124
53.110.7 61.610.5 59.6:0044
PGD LI
models for testing, which were learned via ensemble adversarial training: IncV3ensa (ensemble of three V3 networks), Inc (ensemble of four IneV3 networks), and IncRes V (ensemble of three IncRes networks), which were released by Tramèr et al. (2017).
Ensemble source model: Besides above adversarial transferring from a single-source model, we also conducted the proposed IR Attack in the scenario of the ensemble- attacking (Liu al. 2016). in order to generate adversarial perturbations on the ensemble of RN-34, RN-152, and DN- 121.
Baselines. The first baseline method, the PGD Attack (Madry al., 2018). directly solved the Equation which was widely used for adversarial attacks. Besides this baseline attack, the other four baselines were the MI Attack (Dong al.) (2018), the VR Attack (Wu et al. 2018). the SGM Attack (Wu et al.) 2020a). and the TI Attack (Dong et al.) 2019). Our method was implemented according to Equation (5). namely the Attack. Because the SGM Attack was one of the top-ranked methods of boosting the adversarial transferability, we further added the interaction loss fitteraction to the SGM Attack as another implementation of our method (namely the SGM+IR Attack). We also used the interaction loss to boost the performance of the MI Attack and the VR Attack (namely MI+IR and VR+IR, respectively). Please see Appendix M. for details. Moreover, as Section 4 states, the MI Attack, VR Attack, and SGM Attack also decrease interactions during attacking. Thus, we combined the IR Attack with all these interaction-reducing techniques together as a new implementation of our method, namely the HybridIR Attack. All attacks were conducted with 100
7

---

Published as a conference paper at ICLR 2021
Table 3: Transferability against the secured models: the success rates of Toco black-hox attacks crafted on RN-34 and DN-121 source models against three secured models.
Source
Method
IncV3
IncV3rmat IncRes
Source
Method
IncV3rma3
IncV3
IncRes
PGD Loz
9.811.11
10.0:0.55
5.740.3
PGD Lox
12.8:0.1
11,22117
6,941.0
PGD +I +IR
26.
2,31+113
1..3::0.4
PGD T IR
28.0++1.8
26,5++2.1
17,4+1.3
RN-34
DN-121
21.42:0.8
20.940.9
14.9::14
T
1..
26.1
19.4=1.6
IR
33.6.10.4
33.2.10.3
24.0.1..55
T IR
38.0.1..5
42.2.17.7
29.0.11.4
Table 4: The success rates of Loc black-box attacks crafted by different methods on four source models (RN-34/152, DN- DN-121/201) against seven target models, Transferability of adversarial per- turbations can be enhanced by penalizing interactions.
Source
Method
VGG-16
RNI52 DN-201
SE-154
IncV3
1ocV4
IncResV2
80.1
73.0.2.11
77.710.5
48.91088
46.2112
3.59-0.5
34812.
VR
88.8:0022 86.4=166
87,942,4
62.111.5
58.413.0
56.3:233
49.710.9
RN-34
SGM
91.8+0.66 89.0=0.9 90.0=0.4
68.0+144
63.940.3
5822-111
14.6
SGM+IR
94.74006
91.7=006
93.420.8
72.74:0.
68.94099
64.1113
61.3+100
HybridIR
96.5+0.11
949-0.3
95.6=006
79.7+1.0
77.1+08
738-0.1
70.2:-0.5
MI
70.3+0.6
748=14
51.74088
4:11009
40.5:1
3684277
VR
83.9:3.4
91.1209
70.013.7
63.1
588=0011
56.211.3
RN-152
SGM
88.240..5
90.2:033
72.7++14
63.240.7
59.1=15
58.1+112
SGM+IR
92.0++1.0
92.5=0.4
79.3+0.1
69.6+088
6622110
63.62:09
HybridTR
95.3+-0.4
96.9-002
84.74-0.7
80.0+112
77.5-008
75.6+-0.6
MI
83.0=b4.9
72.0=0.7
91.5+002
58.442.6
54.6+1..6
49.2-244
43.9+1.5
VR
91.5+005
88.7=0.5
98.8=02
75.1+11
74.3+117
75.6=3.0
09.8+13
DN-121
SGM
88.7+00.9
88 1=100
98.0=04
78.0+0,9
64.742..
65.4-2.3
59.7+1.7
SGM+IR
91.7.10.2
90.4.0.4
0.4 94.3.0.11
87.0.0..4
78.8.1..3
79.5.0.22
75.8122.7
HybridIR
96.9400.4
96.8 0.4 9..1
90.9:0.5
88.41088
8788-08
87.1
MI
77.340.8
74.8=1A
64.6ET.0
56.542.5
51.-2.11
47.8+19
VR
87.3.1..1 90.4.1.2
78.011.5
75.812.1
75.8.1.
71.311.2
DN-201
SGM
87.34-0.3
92.4-100
82.94-022
72.340.3
71.3-0.6
68.8+0055
SGM+IR
89.5:10.9
91.8-007
87.311.2
82.51088
80.310.3
81.510.5
HybridIR
94.4.00.1
91.710.2 89.610.6 0.3 87.310.7
steps on randomly selected 1000 images of the validation set in the ImageNet dataset. We set T6/255 for the T attack, and set e = 16/255// following the setting in (Dong et al. 2018) for the L2 attack. The step size was set 2/ 2/255 for all attacks. Considering the efficiency of signal processing in DNNs with different depths, we set À = 1 for the IR Attack, when the source DNN was ResNet. We set À 2, for other source DNNs. To enable fair comparisons, the transferability of each baseline was computed based on the best adversarial perturbation during the 100 steps via the leave-one-out (LOO) validation. Please see the Appendix for the motivation and the evidence of the LOO evaluation of transferability. All attacks were conducted with three different random samplings of grids or different initial perturbations.
T the success rates of the baseline attack (PGD (Madry et al., 2018) and the IR Attack, namely PGD +IR of Laz attacks and PGD II of L2 attacks. Compared with the baseline attack, the transferability was significantly improved by the interaction loss on various source mod- els against different target models. Let us focus on the Los attack. For most source models and target models, the transferability enhancement brought by the interaction loss more than 10%. In particular, when the source DNN and the target DNN were DN-201 and Inc, respectively, the baseline attack achieved the transferability of 36.5%. With the interaction loss, the transferability was improved to 63.7% (> 27% gain). As Table P shows, in most cases, the IR Attack on the ensemble model generated more transferable perturbations than the PGD Attack. Besides, as Ta- ssows, our interaction loss also improved the transferability against the secured target DNNs Such improvement further verified the negative correlation between transferability and interactions. Note that we did not use the LOO in Table 3 in order to make experimental settings in this table consistent with the evaluation used by Tramer et al. (017). Table shows the improvement of the transferability obtained by the interaction loss on other attacking methods. The interaction loss could further boost the transferability of state-of-the-ar transfer attacks. Without the interaction loss, the highest transferability made by the SGM Attack against the V2 was 68.8% (when the source is 2012 When the interaction loss was added, the transfcrability was improved to 81.5% (>
Prrevoous studies usually the number of steps 10 or 20. Here, we set the number of steps to 100 together with the leave-one-out validation for fair comparisons of different attacks.
"The TI Attack was designed oriented to the secured DNNs which were robustly trained via adversarial training. Thus, we applied the TI Attack to the secured models in Table
8

---

Published as a conference paper at ICLR 2021
Source: R9,34
Source: DN121
Souroe: RR434
Sourre: DN121
Trandferabil Bradeerbbiiyy
Tanntorabily
Transferrbiilly
RN152
(M0011
63-
51354
Ind/3
00
02
58
10
12
02
34
64
03
12
20
step
50ap
(a)
(b)
Figure 3: (a) The success rates of black-box attacks with the IR Attack using different values of X. The success rates increased, when the value of À increased. (b) The transferability of adversarial perturbations generated by only using the interaction loss (without the classification loss). Such adversarial perturbations still exhibited moderate adversarial transferability. Points localized at the last epoch represent the transferability of noise perturbations the baseline.
12% gain). Moreover, the HybridIR Attack, which combined all methods of reducing interactions together, improved success rates from the range of 54.6%~~98.8 to the range of 70.2%~~99. 1%.
We can understand behaviors of the proposed interaction loss as follows, Different methods generate adversarial perturbations in different manifolds, thereby exhibiting different transferabilitty Based on the current perturbation, the interaction loss can point out the optimization direction towards further decrease of interactions in local manner due to its optimization power. Thus, the interaction loss further boosts the transferability.
To further demonstrate the broad applicability of the interaction loss, besides untargeted attacks on the ImageNet dataset, also conducted targeted attacks the CIFAR-I0 dataset (Krizhevsky & Hinton 2009). Experimental results consistently showed that the adversarial transferability can be enhanced by reducing interactions in targeted attacks. Please see Appendix. M.2 for details.
Effects of the interaction loss. We tested the transferability of perturbations generated by the IR Attack with different weights of the interaction loss X. In particular, the baseline attack (PGD) can be considered as the IR Attack when = O. We conducted attacks on two source DNNs (RN-34, DN-121), and transferred adversarial perturbations to seven target DNNs (VGG16 RN-152, DN- 201, SE-154, V3c V4, In)ResV2). The attacks were conducted with 100 steps on validation images ImageNet. Figure 3 (a) shows the black-box success rates with different values of X. The transferability of the IR Attack increased along with the increase of the weight X.
Attack only with the interaction loss. To further understand the effects of the interaction loss, we generated perturbations by exclusively using the interaction loss (without the classification loss). We used the RN-34 and DN-121 as source DNNs and tested the transferability on seven target DNNs. The attacks were conducted with 100 steps on ImageNet validation images. Figure 3 (b) shows the curve of the transferability in different epochs. We compared such adversarial perturbations with noise perturbations generated as sinnnnoise), where noise ~ NN(0, I²1), and 16/255, which was the same as the value used in the Loo attack. We found that perturbations generated by only using the interaction loss still exhibited moderate adversarial transferability. This phenomenon may be explained as that such perturbations decrease most interactions in the DNN, thereby damaging the inference patterns in the input image.
6 CONCLUSION
In this paper, we have analyzed the transferability of adversarial perturbations from the perspective of interactions based on game theory. We have proved that the multi-step attack tends to gener- ate adversarial perturbations with large interactions. We have discovered and partially proved the negative correlation between the transferability and interactions inside adversarial perturbations. Le. adversarial perturbations with higher transferability usually exhibit more negative interactions. We have proved that some classical methods of enhancing the transferability essentially decrease interactions between perturbation units, which provides a unified view to understand the enhance- ment of transferability. Moreover, we have proposed new loss to directly penalize interactions between perturbation units during attacking, which significantly improves the transferability of pre- vious methods. Furthermore, we have found that adversarial perturbations generated only using the interaction loss without the classification loss still exhibited moderate transfcrability, which provides new perspective understand the transferability of adversarial perturbations.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
All members in Shanghai Jiao Tong university, including Xin Wang, Jie Ren, Shuyun Lin, Xiang- ming Zhu, and Dr. Quanshi Zhang are supported by National Natural Science Foundation of China (61906120 and U19B2043) and Huawei Technologies. Dr. Yisen Wang is partially supported by the National Natural Science Foundation of China under Grant 62006153, and CCF-Baidu Open Fund (OF2020002). Xin Wang is supported by Wu Wen Jun Honorary Doctoral Scholarship, Al Institute, Shanghai Jiao Tong University.
REFERENCES
Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a poly- nomial time algorithm for shapley value approximation. In International Conference on Machine Learning, pp. 272-281, 2019.
Yang Bai, Yuyuan Zeng, Yong Jiang. Yisen Wang, Shu-Tar Xia, and Weiwei Guo. Improving query efficiency of black-box adversarial attack. In ECCV, 2020.
Arjun Nitin Bhagoji. Warren He, Bo Li, and Dawn Song. Practical black-box attacks on deep neural networks using efficient query mechanisms. In European Conference Computer Vision pp. 158-174 Springer, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. IEEE, 2017.
Hanjie Chen and Yangfeng Ji. Learning variational word masks to improve the interpretability of neural text classifiers. In EMNLP, 2020.
Hanjie Chen, Guangtao Zheng, and Yangfeng Ji. Generating hierarchical explanations text clas- sification via feature interaction detection. In ACL, 2020.
Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. L-shapley and c-shapley: Efficient model interpretation for structured data. In arXiv:1808.02611 201 8a.
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order opti- mization based black-box attacks to deep neural networks without training substitute models. In arXiv:1708.00 03999, 2017a.
Pin-Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Ead: Elastic-net attacks to deep neural networks via adversarial examples. In AAAI, 2018b.
Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In Advances in neural information processing systems, pp. 1467-4475. 2017b.
Mirek Riedewald Daria Sorokina, Rich Caruana. Detecting statistical interactions with additive groves of trees. In ICML, 2008.
Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transfer- ability of evasion and poisoning attacks. In 28th USENIX Security Symposium USENIX Security 19), pp. 321-338, 2019.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Xiaolin Hu, Jianguo Li, and Jun Zhu. Boosting adversarial attacks with momentum. In CVPR, 2018.
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invann invariant attacks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4312 2019.
Ranjie Duan, Xingjun Ma. Yisen Wang, James Bailey, A Kai Qin, and Yun Yang. Adversarial camouflage: Hiding physical-wond world attacks with natural styles. In CVPR, 2020.
10

---

Published as a conference paper at ICLR 2021
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Qian Huang. Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. Proceedings of the IEFE International Conference Computer Vision, pp. 4733-4742, 2019.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In ICML, 2018.
Andrew Ilass, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Alcksander Madry. Adversarial examples not bugs, they are features. In NeurlPS, 2019.
Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7066-7074, 2019.
Nathan Inkawhich, Kevin Liang, Lawrence Carin, and Yiran Chen. Transferable perturbations of deep feature distributions. In International Conference on Learning Representations, 2020.
Joseph Janizek, Paseal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. arXiv preprint arXiv:2002.04138, 2020.
Xisen Jin, Zhongyu Wei, Junyi Du, Xiangyang Xue, and Xiang Ren. Towards hierarchical im- portance attribution: Explaining compositional semantics for neural sequence models. In ICLR, 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, 2009.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. pp. 1097-1105, 2012.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In arXiv:1607.02533, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-bart based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 (111))2222222222 2324,, 1998.
Yingwei Li, Song Bai, Yuyin Zhou, Cihang Xie, Zhishuai Zhang, and Alan Yuille. Learning trans- ferable adversarial examples via ghost networks. In Proceedingy of the AAAI Conference Artificial Intelligence, volume 34, 2020.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang. and Kevin Murphy. Progressive neural architecture search. Proceed- ings of the European Conference Computer Vision (ECCV), pp. 19-34, 2018.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial exam- ples and black-box attacks. ICLR, 2016.
11

---

Published as a conference paper at ICLR 2021
Xingjun Ma, Bo Li, Yisen Wang. Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu. Under- standing adversarial attacks on deep learning based medical image analysis systems. Pattern Recognition, 100:10733,, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Grabisch Michel and Roubens Marc. An axiomatic approach to the concept of interaction among players in cooperative games. In International Journal of Game Theory, 1999.
W. James Murdoch, Peter J. Liu, and Bin Yu. Beyond word importance: Contextual decomposition to extract interactions from Istms. In ICLR, 2018.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In IEEE European Symposium on Security & Privacy, 2016.
Nicolas Papemot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In 16Xiv:160..02697, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 8026- 8037. Curran Associates, Inc., 2019.
Jie Ren, Die Zhang, Yisen Wang. Lu Chen, Zhanpeng Zhou, Xu Cheng, Xin Wang, Yiting Chen, Jie Shi, and Quanshi Zhang. Game-theoretic understanding of adversarially learned features. arXiv preprint arXiv:2 103.0 003.07364 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. In International Journal of Computer Vision, 115(3):211 12121122555 1-252, 2015.
Su-In Lee Scott Lundberg. Consistent feature attribution for tree ensembles. In ICML WHI Work- shop, 2017.
Lloyd S Shapley. A value for n-person games. In Contributions to the Theory of Games, 2(28): 307-317, 1953.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Chandan Singh, W. James Murdoch, and Bin Yu. Hierarchical interpretations for neural network predictions. In ICLR, 2019.
Jiawei Su, Danilo Vasconcellos Vargas, and Sakurai Kouichi. One pixel attack for fooling deep neural networks. In arXiv: 088644 2017.
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. arXiv preprint arXivv/908.000 00777,201 2019.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pp. 3319-3328, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 23111699999999 2013.
12

---

Published as a conference paper at ICLR 2021
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-forst AAAI con- ference on artificial intelligence, 2017.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv:1705.0773 2017.
Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In ICLR, 2018.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML, 2019.
Robert J Weber. Probabilistic values for games. The Shapley Value. Essays in Honor of Lloyd S. Shapley, pp. 101-119, 1988.
Dongxian Wu, Yisen Wang, Shu- Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In International Conference on Learning Representations, 2020a.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. In NeurIPS, 2020b.
Lei Wu, Zhanxing Zhu, and Cheng Tai. Understanding and enhancing the transferability of adver- sarial examples. arXiv preprint arXiv:1802.09700 2018
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L. Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2730-2739, 2019.
Die Zhang, Huilin Zhou, Hao Zhang, Xiaoyi Bao, Da Huo, Ruizhao Chen, Xu Cheng, Mengyue Wu, and Quanshi Zhang. Building interpretable interaction trees for deep nlp models. In AAAI, 2021a.
Hao Zhang, Xu Cheng, Yiting Chen, and Quanshi Zhang. Game-theoretic interactions of different orders. arXiv preprint arXh:22010.14978 2020.
Hao Zhang, Sen Li, Yinchao Ma. Mingjie Li, Yichen Xie, and Quanshi Zhang. Interpreting and boosting dropout from a game-theoretic view. In ICLR, 2021b.
Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang, Interpreting multivariate interactions in dnns. In AAAI, 2021c.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8697-8710, 2018.
13

---

Published as a conference paper at ICLR 2021
A MOTIVATIONS FOR USING THE SHAPLEY INTERACTION INDEX
In this section, we discuss the motivations of using the Shapley interaction index to define the inter- action.
A.1 FOUR PROPERTIES OF SHAPLEY VALUES
Let = (1, n) denote the set of all players, and the reward function is e. Without ambiguity, use ((i[S2) to denote the Shapley value of the player in the game with all players Ω and reward function v, which is given as follows.
(s (1
(112) = Σ
(6)
n!!
(CAV()
The Shapley value satisfies the following four properties (Weber
1988):
Linearity property: If there are two games and the corresponding reward functions are and w, i.e. e(S) and w(S) measure the reward obtained by players S in these two games. Let (i[2 and & (2) denote the Shapley value of the player in the game and game respectively. If these two games are combined into a new game, and the reward function becomes reward(S)) = v(S) + w(S), then the Shapley value comes be (d[[) (i({2) + (i((2) for each player in Q.
Dummy property: A player E Ω is referred to as a dummy player if VS C (1()), ((SU(() v(S) v(()). In this way, ((i[[) v((+)) v(0), which means that player plays the game independently.
Symmetry property: VS Φ1 (i.j) ((S = (7)), then Shapley values of player and j are equal, i.e. ((i(2) = ((j
Efficiency property: The sum of each individual's Shapley value is equal the reward by the coalition N, i.e. Σ, (((()) v(2) - v(0). This property guarantees the overall reward can be allocated to each player in the game.
A.2 MOTIVATIONS
Theoretical rigor. We use the Shapley interaction index defined based on the Shapley value, be- cause the Shapley value has solid theoretical foundation in the game theory, which the unique attribution satisfying the above four desirable axioms.
Whether the metric depends on network architectures. Because adversarial transferability is general property for the attack, a convincing metric for adversarial transferability is supposed not to be directly related to the network architecture. To this end, the computation of the interaction defined the Shapley value does not depend on the network architecture. In comparison, previous definitions of the interaction are usually oriented to model architectures. For example, the interaction proposed by Tsang al. (2018) requires the DNN to be fully- fully-connected. The two interaction metrics proposed by Murdoch et al. (2018) Jin et al. (2020) are designed for LSTMs. The Hessian based interaction (Janizek al. 2020) requires the DNN to use the softPlus operation to replace the ReL.U operation.
Computational cost. The computational cost of the Shapley-based interaction-reduction loss is relatively low. Because of the efficiency axiom of the Shapley value, we prove that the time cost of computing the interaction loss Linteraction - [v(2) i.e. On),, where is the dimension of features. The linear complexity makes possible to apply the interaction to high-dimensional data and deep neural networks. In contrast, the complexity of computing all possible pairwise interactions defined (Daria Sorokina 2008 is (n
v()) - v( o({i)) + x(0] is linear,
B COMPARISONS BETWEEN INTERACTIONS INSIDE PERTURBATIONS OF DIFFERENT ATTACKS
We have theoretically proved that some classical attacking methods of boosting the adversarial trans- ferability essentially decrease interactions inside perturbations. Besides the theoretical proof in Ap-
14

---

Published as a conference paper at ICLR 2021
Table 5: The average interaction inside adversarial perturbations generated by different attacks.
Method
RN-34
RN-152
DN-121
DNN-201
Baseline (PGD Attack)
0.422
0.926
0.909
0.784
SGM Attack
-0.012
0.037
0.395
0.308
VR Attack
0.097
0.270
0.242
0.137
pendix !] and Appendix H we also conduct experiments to compare interactions of perturbation units when we generate adversarial perturbations with and without these attacking methods. Such experiments further verify that these methods of boosting the transferability essentially decrease in- teractions. We conduct attacks with the validation set the ImageNet dataset on four DNNs, and measure the average interaction inside perturbation units. As Table 5 shows, the SGM Attack and the VR Attack decrease interactions inside perturbations.
C ADVERSARIAL ATTACK
In general, the objective of adversarial attacking can be formulated as the following optimization problem.
maximize (1h(x+6).0) s.t. | e, + 5 [0,
where E(h(x 8), y) is the classification loss. There are many ways solve the above optimization problem under different norm constraints |-Ipp (Goodfellow et al. 2014 Carlini & Wagner 2017
Kurakin et al., 2017] Madry et al., 2018 Chen et 2018b: Wang et al. 2019).
Optimization-bann based approach. One approach to approximately solving Equation is to solve the following relaxed form:
minimize {-((h(x +), + c c001))) s.t. + 8 e [0,
where e >> a scalar constant to balance the classification loss and the norm constraint. Szegedy al. (2013); Carlini & Wagner (2017) have demonstrated the effectiveness of this method.
Projected gradient descent (PGD) (Madry et al., 2018). The PGD Attack is usually considered as one of the simplest and the most widely used baseline for adversarial attacking. In this paper, this method is called the Baseline. The PGD Attack directly optimizes the classification loss in Equation (7p. Considering the norm constraint, after each step of updating, the PGD Attack projects the adversarial perturbation back to the e-ball, if the perturbation goes beyond the ball.
PGD updates adversarial perturbations in each step with the following equation: (e)
(8+ + a sign (Ve (x ()))), p = +o0
8++1
!T??) 5t + (e(h(())))))
ve(h(2++")..)
p= 2,
where denotes the perturbation of the t-th step. II and II! are projection operations, which project the perturbation 5 back the e-ball, if the perturbation goes beyond the ball. is the step size. Given E IR,", we have:
II (1.))( { 6 (6.).
( >
if ||0| > E
(1)2 =
T 8,
(10)
if o. S
if | <
D EQUIVALENT FORMS OF THE INTERACTION
In Section 3.1 the interaction between units i,j is defined as the additional contribution as follows. (6) = (SSy [Syy1')) (Q') - [o{i ( \ (f}) + o(j[2 (i))),
(11)
where ((S,, (2') denotes the joint contribution of i.j, when perturbation units i.j are regarded as singleton unit Syz = {i,j}, ), as follows.
SS (n (1- - 15|-
(S,,12))
Σ (CR)(E)
(w(S (i.5)) =(S)).
(n
15

---

Published as a conference paper at ICLR 2021
where St {i.j} represents the coalition of perturbation units i.j. In this game, because pertur- bation units i, j are regarded as a singleton player, we can consider there are only n 1 players in the game, and consequently the set of players changes to !2' 52 {i.j} U S.y.
( {j}) and 6 \ {i)) represent the individual contributions of units i and j. respectively, when the perturbation units i, work individually. The individual contribution of perturbation unit i, when perturbation unit j is absent, is given as follows.
[s 1S1 (n 1)!
o \ (i) = Σ
(i)) u(S))
SCAVO.))
In this game, because the perturbation unit j is always absent, we can consider there are only n players in the game. Consequently the set of players changes to Ω \ (i).
Similarly, the individual contribution of perturbation unit j. when perturbation unit is absent, is given follows.
SS (n- 1 (((SU (n - 1)1
2010 (1))
E SEAV(S))
In Section 1. the interaction between perturbation units 6,, 6j is defined the change of the impor- tance d, of the i-th unit when the j-th unit 6, is perturbed w.F. the case when the j-th unit 6, is not perturbed. If the perturbation 6, on the j-th unit increases the importance òi of the i-th unit, then there is positive interaction between 6; and ój. the perturbation ój decreases the importance di. it indicates negative interaction. Mathematically. this definition can be written follows.
[6) di,w/ Piww/oj>
(12)
where / represents the importance of 8, when oy is always present; Pi,w/o3 represents the importance of 6; when 0j is always absent. When perturbation unit is always present, the contri- bution of perturbation unit is given as follows.
|S (n 1S1-2)1
Pi,w/j
E ScRVin)
((SSU(i))) -
((S
(n ( 1)!
In this game, because the perturbation unit is always present, consider there are only players.
When perturbation unit j is always absent, the contribution of perturbation unit is given as follows.
|S (n 151- (n - 1)! (v(S (i)) v(S)).
Piww/oj
(C))
In this game, because the perturbation unit j is always absent, consider there only players.
The interaction in Equation is equal to the interaction in Equation 12 i.e.
Iτ,(5) (1,
PROOF OF PROPOSITION
To simplify the problem setting, we do not consider some tricks in adversarial attacking, such as gra- dient normalization and the clip operation. In multi-step attacking, the final perturbation generated after steps is given as follows.
def
V Vel(h(x + 56 Diunta) y),
#=0
where represents the step size, and ((h(x), y) is referred as the classification loss.
16

---

Published as a conference paper at ICLR 2021
To simplify the notation, we use ((x) to denote V. V=((h(a), y). i.e. g(x) def more, we define the update of the perturbation with the multi-step attack at each step t as follows.
g(x)== (x(())()),)) v). Further-
Armult def g(x + 8 )-
(13)
In this way, the perturbation can be written as follows.
mui Ar + Ar
Artm1
(14)
Lemma Given the sample x E IR and the adversarial perturbation å R", we use Q
[1. denote the set of all perturbation units. The score function is denoted by v(S)
L(x + ).)()),, where 8(5) satisfies vi 5,5 ō,;Vi & S. 5,6%.) O. The Shapley interaction between perturbation units a, b can be written as Lub = 8 (a))) + R2(5), where (as(z) = De.dus represents the element of the Hessian matrix, and (6) denotes terms with elements in ō of higher than the second order.
OLiz)
Proof. The Shapley interaction between perturbation units a, is
SS (n ( 2 ((SU(a,b)) - (()) v(8 (a)) v(S),
Ins(ö)=
Σ
(n 1)!
where USS) = L(x + ((5)). Here, the classification loss can be approximated as L(x + 5) = L(x) + g" (z)6 206 (x)6 + R2(6) using Taylor series. Thus, AS C S2
+ 1 Σ (a)a) R (0).
v(S') L(x) + Σ go (x)
"ES'
a.DES'
where R (8) denotes terms with elements in 8(5") of higher than the second order.
In this way, the Shapley interaction Las given as
| (n -
(06(0)
Σ
(n 1)! (u(S U U b3) v(S (b)) eSS (a}) + x(S)
(c
|S| (n 2S)
Σ
[[(())+
(n - 1)!
sC
Σ
Ja (x)0aa + 2
Σ
Ha' + R ((S(a.b))
d'ESUfa,b
a ())
-[((x)+
Σ
Ie (r)
+
Σ
Sa 122 (((((())())
d'ESUIb)
a' d'M'SS((b)
- -[((x) +
Σ
De (1) +
Σ
Ha (x) + n (SSuaa (0)
a'ESU(a)
al a///ES((a)
+ [(xx) Σ
Do (1)
1
Σ
da Haub(o)d R1 (0)
e'ES
|S (n 22)
(5. (a.
=
(n - 1)!
S|| (n |S| 2))!
(5
(6))
(8)) R R(SU(a))( R (8) + R (8)
(S((b))
(n 1)!
(c))(aM)
R2(8)
R-2
(n - 2)
15, Hat
+ R2(0)
(n - 1)!
scnV (a.0). |S]
17

---

Published as a conference paper at ICLR 2021
2 sl (n dd.laa(z)05 R2(0),
8 (n 2) (n - 1)! (6 Hab (x))6)
+ Ro)
s 2)
where R2(0) denotes terms with elements in å of higher than the second order.
Lemma The update of the perturbation with the multi-step attack at step defined in Equa-
tion 13 can be written as Ar a +HH()))² g(x) + Re, where g(x)) def ve((((a),))
represents the gradient, and (x) def V2 (2((())), y) represents the Hessian matrix. Rt denotes terms with elements in 8 of higher than the first order.
Proof. If malt = a. g(x).
Let Vt' Ar muli = =a + aH(x)) g(x) + R then have
Az g(x ) // According to Equation
B
= =a g(x + Ar + m + ..... + mmmla)
//
According to Equation
14
=a.g x+a ++)
g(t)
Σ
1=1
where R denotes terms of elements in 5 of higher than the first order.
Using the Taylor series, we get
1-
Artmin a a + oH
Σ
R + R
(15)
where - denotes terms with elements 8t-1 of higher than the first order. T(x) in Equation (5) is given as follows.
T(x) = T+ [T oH
+ [[FH))
g(x).
(16)
Multiply (1 + afl (x)) on both sides of Equation (16). and we get
(z) a. [[T+aH(x)) [7+ a
+ JI aHH() g(x). (17)
Then, according to Equation 17 and Equation 16 we get
(x))((x) = [v [ g(x))
(18)
Substituting Equation 18) back to Equation [5 we have
Ar Ar a [ a aH g(x) Rt.
In this way, we have proved that vi 2 1, A a =a(l+a((x))]= g(x) + R7.
18

---

Published as a conference paper at ICLR 2021
Proposition 1. The adversarial perturbation generated by the step attack via gradient descent L Vgl(h(z+ 8 muda) y), where 5 denotes the perturbation after the
is given as 8 t-th step of updating, and m is referred to as the total number of steps. The adversarial perturbation generated by the single-step attack is given Ösingle am (((((x))))) The expectation of in- teractions between perturbation units in mute 18, aa mante)) is larger than Ba, at (Ssintte)) ) i.c. Lab (8000mmd))) (mdd)) Ba,b L (isinte)))))
E.1 FAIRNESS OF COMPARISONS OF INTERACTIONS INSIDE DIFFERENT PERTURBATIONS
Proposition iss valid for different loss functions of generating of adversarial perturbations. this section, we discuss the fairness of comparisons of interactions inside different perturbations.
When we compare interactions inside different perturbations, magnitudes of these perturbations should be similar, because the comparison of interactions between adversarial perturbations of dif- ferent magnitudes is not fair. For fair comparisons, in Section this paper controls the magnitude of the single-step attack by setting the step size of the single-step attack as am, where and m de- notes the step size and the total number steps of the multi-step attack, respectively. The equivalent step size makes the magnitude of perturbations generated by the single-step attack be similar to that of perturbations generated by the multi-step attack, when we use the target score before the softmax layer to generate adversarial perturbations, such as ((h((). v) = maxy øy h(x)) hy (2)) In this case, the magnitude of the gradient vz((h(z),, y) is relatively stable. In particular, this type of loss has been widely used. For example, one the most widely used attacking (Carlini Wagner 2017). uses the score before the softmax layer for targeted attacking.
E.2 PROOF OF PROPOSITION
Proof. According Lemma the update of the perturbation with the multi-step attack the step t is given as follows.
Art Amuti a [7 (x))))(-1111) 9(x) + Rd
(19)
where RS denotes terms with elements in T of higher than the first order, and represents the step size.
To simplify the notation without causing ambiguity, we write g(x) and (x) as g and H. respec- tively. this way, according Equation (14) and Equation (19), 8m muli can be written follows.
m
8m =a
=a(1+
I
+ [I aH [I+ aH2 + aH
R
9
(20)
ml am 1mm-1 1)
H 9
where m represents the total number of steps. According to Lemma between perturbation units a, in 8mm is given as follows.
the Shapley interaction
(6
5 multi,a
H 5m multi
(21)
where R2(5m)) denotes terms with elements in mi of higher than the second order.
19

---

Published as a conference paper at ICLR 2021
According to Equation (20) and Equation 21 have
Las Lal(mutti Hob amgg a a²m(m 1)
(av +...
Σ 1=1
of (fmutti..))
][
==1
terms aaate, higher than the first order, which corresponds to the term of Rt in Equutiion 20
oongb +
a²m(m - 1)
(Hrbya') +
of (f &
] + (7
2
='=1
of higher than the first order, which corresponds to term of RC in Equation 20
amm²² gagh
first-order
H
a m
1 1)m?
am 1)m?
9b
Hav gv)
9a
(Ha'bga))
Hab
2
second order terms wart. elements in H
4 (m - m)2mm²
Haa Do
(Hargaa) Hab +
V=1
e'=1
Reqll (HH)
o (0muld H multi
of Emmtti M Hab 8m
multi, R226 (626mmm)))
Re (8mm))))
a m² Has + a²(m n
1)m²
Hab Σ (Horga
2
+ am 1)m²
n
9b Hab
Σ
(w) + (8 + R (H);
2
(22)
where Rmmuli (H) represents terms with elements in of higher than the second order, and itg (mmmaa) represents terms with elements in 8m of higher than the second order.
Let us consider the single-step attack. When we compare interactions inside different perturbations, magnitudes of these perturbations should be similar, because the comparison of interactions between adversarial perturbations of different magnitudes is not fair. For fair comparisons, in Section [3.]] this paper controls the magnitude of the single-step attack, as follows. The single-step attack only uses the gradient information on the original input x, which generates adversarial perturbations as:
Single amg.
Therefore, according to Lemma I the interaction between perturbation units b of Osingle is given as follows.
Tab (Sainge)) = O Habd + Riz (52ggenn))))
(23)
m maa² Hab + ( (5singe)))
where R2 (Sinni) denotes terms with elements in &inggo of higher than the second order. In this way, according to Equation (22) and Equation (23). the expectation of the difference between I, mm) and Tas (ingle) is given follows.
20

---

Published as a conference paper at ICLR 2021
Hétograma the value of the Hessien element Has w.t1. w.t. difirent values of
Helograme of valuo of
Hooddan eisrent 25u
diferent values of
91O0A
1000
M0.00 CM.0
1.00
KOMAA
SESTEE E.
122 CAAARRRAN
(b)
Figure 4: (a) Histograms of the value of the Hessian element Has(z) wer. different values of a, b. (b) Histograms of the value of g.Hiu
>= 9- Hag W.E.f. different values of b. Because the Hessian well defined, we replace the ReLU activation with the Softplus activation [(x) log(1 + e = ). We train VGG-16, ResNet-32, and DensetNet-121 on the CIFAR-10 dataset (Krizeeskyy et al. 2009), and use the cross-entropy loss as the classification loss.
of the DNN with the ReLU activation
Ba, (amll) Las [Osingle]]
a (m
1)m²
am 1)m²
En,b
aab D
(Hubga)) + a'=1
Hab Σ (Har
2
2
Rg (8 + Ruullu (H)
Re2
a (m
ammmm - 1)m²
Eobb
Io Hab
Σ
(Ha')ga)
Hab Σ (Ha
+ Eaa (Rabl
a'=1
===1
Unb
where
Rab = (5maal + R (H) (ogiingg)))
Assumption 1: Magnitudes of elements in the Hessian matrix H(x) is small that [Hab())]< 1, where 1 S b b < n. Therefore, H Hx) x 0, if k > 2.
We verify the assumption by directly measuring the value of Has(x). As Figure 4 (a) shows, the value of (a)( is very small that [aa(()){< 1.
According Assumption 1, we have Reuuli (H) 0. Note that the magnitude of 8mm and the magnitude of &angle are small, then Rt' (5mm)))) Rv 0, and (osinte))) so 0. In this way, we have E,,n [Ran) = E.,. [2y (8 ) + R2 (Bdinggte) Reguli (H) (H)) ~ 0.
Moreover, for the expectation of Uah. we have
[Uat] n(nn 1)
Σ b=1 ag
Da Hab
Σ
(ga Hars)
d==1
21

---

Published as a conference paper at ICLR 2021
Histograms the value Eg Tas different values
Mull-sapp sangleesslpp
Muli-atap: reca
6000
4000
<606
2000
225
Eg
1ZL-19N/80
6680
5050
5646
4000
3010
3000
1059
Eg E
E2 Aubh
Figure 5: Histograms of the value of [Bs[Har] w.r. different values of
Σ Da Har
goltha B
Σ Ia Haeb
n(n 1) b=1
Let us focus on terms of and B. Note that A is the sum of n terms (n large). In comparisons, B is just single term in A. Therefore, the sign of A - B is usually dominated by the term A. In this
way, we get Prob (Aign(A B) sign(A)) ~ Therefore, Prob [(A B) 0 We verify
this assumption by measuring the value of E HaH If Prob
.6.
96 Hoe
<<
1, then
D Ba Hab
have Prob [sign(A- B) = sign(A)) == 1. As Figure 4 (b) shows, the value of
Σa Hab is very
gaH
small that
D6 Hop
& 1. To this end, we have (A B)B 2 0, and we get Do Has
Ea, 2
(24)
Due to the symmetry of a, b, we have Ea,0 Eg (Wal Therefore,
Bagb h ant (fmutt) Tabl [ningle)]
a a (m 1)m² Bo, (Va + Usal W., [Rab]
~ (m - E +0
0.
E.3 VERIFICATION OF PROPOSITION II
We verify that perturbations generated by the multi-step attack tend exhibit larger interaction than those generated by the single-step attack by measuring the value of Es [as]. As shown in Appendix[F] we prove that [a[4 = v(It) - v(2 \ {a)) - -((a)) + Because the image data high-dimensional, the cost of computing Ex( is high. As Appendix J.1 demonstrates, given the input image, we can measure the interaction at the grid level, instead of the pixel level, to reduce the computational cost. Therefore, Equation 339 to compute the interaction as Eyy 44) (1' (d)] = v(A) v(A (Apg)) v( ((AApp)) }) + (8). where (p, q) denotes the coordinate of a grid. The experiments were conducted with ImageNet validation images ResNet-32 and DenseNet-121.
divide the input image into 16x 16 (L 16) grids, and use
For fair comparisons, the magnitude perturbations generated by the single-step attack controlled to be same that generated by the multi-step attack. As Figure (left) shows, perturbations gener- ated by the multi-step attack tend to exhibit larger interaction than those generated by the single-step attack.
22

---

Published as a conference paper at ICLR 2021
E.4 PERTURBATIONS GENERATED BY THE MULTI-STEP ATTACK TEND TO EXHIBIT LARGER INTERACTION THAN GAUSSIAN NOISE
Moreover, we compare the interaction inside perturbation units generated by the multi-step attack with the Gaussian noise perturbation. Similarly, for fair comparisons, the magnitude of the Gaussian noise is controlled to be similar to that generated by the multi-step attack. As (right) shows, perturbations generated by the multi-step attack tend exhibit larger interaction than Gaussian noise.
F EXPECTATION OF THE SHAPLEY INTERACTION
In Equation 3}. the Shapley interaction between two perturbation units i, is given as follows.
(6) (S5,111) \ (1.)) (o(1)) (i)) + (1))),
where (S {i.}} S.y) is the Shapley value the singleton unit Sty = {i, j}. when pertur- bation units i form coalition. o(i] \ {i}) and d(il1 \ {i}) are Shapley values of perturbation units j. when these two perturbation units work individually. this way, we can write the Shapley interaction in closed form as follows.
(n - 1
(0)
Σ
(c(SU vOS (1)) ((S (SU v(S)), (25)
(n 1)!
SCRVIN)
where VS C Q2 v(S) max,'yy /1 (x+ ((51) 116²) (x + +())). The expectation of interaction given follows.
E., [1,
- E, (0(2) - v(22 (21((11)) (0)) - v((()) +(fi))++
(26)
which is proved as follows.
Proof. As proved in Appendix D Inf(0) (6) Therefore, the interaction between players and j is given as follows.
St (n (
I6((0)=
E (CR)
- (1)) - u
(n 1)!
= =bjw/i jjrfoo.
The expectation of the interaction can be written as follows.
Ex, [T,( (n
E2
Σ
[Kywe// - Ssw/od]
jEM
According to the efficiency property of Shapley values (please refer to Appendix A.1 for details):
Σ jEm
jjw// v(2) - =ft}}
Σ
jj/oot u(2 {i}) v(0).
(E
In this way,
Ex, [8)(
E, (u( v(2 (i)) +((i))+(00) v(0)
23

---

Published as a conference paper at ICLR 2021
G DETAILS OF OBSERVING THE NEGATIVE CORRELATION BETWEEN THE TRANSFERABILITY AND THE INTERACTION
In Section 3.2 directly measure the transfer utility and interactions of different adversarial perturbations. Here, we give more details of the experiments. We measure the transfer utility as
+ +6] [maxy'sy (g (2) (1)) (x)]- We measure Transfer Utility- mmaxy' hy (x + - hy (x the interaction as E, [6)]
(9)
[x( =(2)) {i}) - - x()]] As Appendix J.1
demonstrates, to reduce the computational cost, given the imput image, we can measure the interac- tion at the grid level, instead of the pixel level. Therefore, we divide the input image into 16x16
(L = 16) grids, and use Equation (39) compute the interaction as E(p-))((''v (')))) ('(p-))((p'g)() = 1 == (p..) [o(A) - v(A \\ (Apg]) v({Apg)) + v(0),, where (p, q) denotes the coordinate of a grid.
Using the validation set of the ImageNet dataset (Russakovsky al. 2015). we generate adversarial perturbations on four types of DNNs, including ResNet-34/152/RR --1331315)) (He et al. 2016) and DenseNet- 121/// (DN-112222))) (Huang et al.) (017). We transfer adversarial perturbations gener- ated on each ResNet to DenseNets. Similarly, we also transfer adversarial perturbations generated on each DenseNet ResNets. Given input image æ, adversarial perturbations
generated using Equation (8). i.e. min,- - -((h(x + 6), y) + c. [ s.t. x + ó E [o, 1J". where E R is a scalar constant. In this way, we gradually change the value of a different hyper-parameter' generate different adversarial perturbations, i.e. = k3 + where 3 E R is constant. Moreover, to ensure adversarial perturbations generated with different values of c change smoothly, we use the
perturbation generated with c--1 to initialize the perturbation for Ck. i.e. g(cx) 70(c--1), where 7 R is constant. In our experiments, we set 7 0.6. For fair comparisons, we need to ensure adversarial perturbations generated with different syper-parameters to be comparable with each other. Thus, we select a constant T and let | 7 as the stopping criteria of all adversarial attacks. We set the number of steps as 1000. The threshold V is set to ensure that attacks with different hyper-parameters т are almost converged when the L22 norm of the perturbation ||ö1|2 reaches т. Note that different attacking methods may successfully attack different sets of testing samples, so we select testing samples that can be successfully attacked by all attacking methods with different c values (i.e. those having reached the stopping criteria under all attacks). The interaction and the transfer utility reported in Figure I are measured on the selected samples for fair comparisons.
H PROOF OF PROPOSITION 2
To simplify the problem setting, we do not consider some tricks adversarial attacking, such as gradient normalization and the clip operation. In VR attack (Wu et al. 2018), the final perturbation generated after t steps is given as follows.
5th def vT
ve(l(x + &.).v),
a
where
Eg= ((M()) =F [((h(x + E),9))
(27)
According to Equation 27 the gradient and the Hessian matrix of ((h(x)) y) is given as follows.
((x) v=((h(x),,)
= Ee-wio,02n (1)((((++))))))
(28)
Ā V2((M(x),))
=EggNi(0.02 (2((h(z+().v))
where represents the step size.
Lemma 3. Given the Gaussian smoothed loss i(x) Ee-N10002)) (((h(z),)))
where ((h(x),)) y) is the original classification loss, Va # b. # a we have
24

---

Published as a conference paper at ICLR 2021
Eg - - & O, E. (x)
=
and Es ( ( (al = 0.
Proof. According to Equation 28 we have
(a(2)) BemN((0.021)
Iaa 5)
By (( (9a(2'))
dgal (r + +
(ga
(as(() Bg-N(0,001)
Ex' N (z')],
Omb
This indicates that the gradient and the Hessian matrix in the VR attack are both smoothed by the Gaussian noise. Because the Lipschitz constants of 9a (1r) and Has Has(z) are usually limited to certain range, we can ignore the tiny probability of large gradients and large elements the Hessian matrix, and roughly assume that (x) (á (2), oy.)) and Hat(a) ~ N (HHass))) n.y). where g, a.. E are tow constants denoting the standard deviation. Thus, ga and Ha)(x) can be written follows.
(x(x) ga(xx) Egai
Ega ~N(00,
(x)(z)= = Hat()) + CHor CHab ~ NO, H..)
(29)
To simplify the notation without causing ambiguity, we write 0(x) and (x) as 9 and H, respec- tively. Moreover, we write g(x) and (x) as g and H1, respectively. In this way, we have
Eg
E B Begu eMa
f2 (āa + Ega ( cHas)²)
E, Beg. Has
90 H2' Haa (Ōa +gg 2eg Gn Hab 22 2 Hablaa + +zg
s Ey Bega eHas
Bataa + 2eHas Hablaa + 6
EE [Ben 16 2lablab + Bean [enaan 12i.l22iabb.... + Ega ega) }²}}
Eg {0- 2g 28 Hab + EEe, (0- 10-2Harl0 cga) (pm))
O.
According to Equation (2). we have Ia = Ia + Cgar 9b = Go + Czs- Thus, we have
Ex Ha. Hag
Ey (1o Hla - (ga + eg Máp + Egg enw))
--Enee snn cHab [co Eo EH + cga Eg Has + c CHarqu
+ 6Ha 99.90 9o + + co Ho Has + cg Aab + CHanBagi
--Ex (een [B. 1E [EH]] + Een. EEn, | + Beng Ieml [EH
+ Een,. |CHn 10. + Bega 16 Hab + E,g I Ha + Beng, [CH.. 10
EE 0.0.0+0- A.D +0.0 9.. + +0-0 90 0. gr. Hos + 0 ga Aab + 0 Gngh
0.
Moreover, according to Equation 29 have
En [ga.. Alat Han
25

---

Published as a conference paper at ICLR 2021
= =En tee Ras Hithh HagcHarHles (9o + tga + (ffos + EHa.) + cHa)
- -- Ho Meb (9 Co CHao
+ Ha. Ego Has C Ega CHs Hab Ega Eaa
+ ". Hat c Hn C CH Ho
cg Cnala Heb + cg. H. du Hab + CHa
+ 9n Har Heb Gg H Heb + CHa Heb + Hns Mab)
-1 [Eezo [E
Ieee le M BeNco Be [EHaol + Be l B (eHalBene leHal
+ Ergu (ssal Benes BeN (CHalee Bego Brge Ko Ben [cHeo) Hab
Begu (Gga Begg ls BEn,, (CHanHHa.
+ Been (Egal Beso ( Hal + Eegn legal BeN, HHandcHob + Begn Ben, leH..leeHa
+ Ben le [E abb @ea.llaate Dattlo + Een lene] SHealaaHan + Eenan leH Ben, lEH
+ Bega c ge Hab Hes + Be 10 Has Heb + Beng Kaaalaade Heb + Ben IcH 1ag..
--En --En 0+0 9. +0 - + 0 Has + 1..
+ 0- Het +0 g. H.s +0- ge Hob + 0 9a Het + 0- 8a Hab +0 Gage
+ 9e Mas Hes +0 90 Has a + Goge Hab
0.
Proposition 2. The adversarial perturbation generated by multi-step attack is denoted by 6mate
- V Ahh(( 8)mata); y). The adversarial perturbation generated by VR Attack denoted by 6TH = a Ei=o V 2z((h( + + 4), where E(h(x + 6t,),1) = Eg~N((..22)) [E(h(x + 8 + n)].
Perturbation units of 8t tend to exhibit smaller interaction than 8mati i.e. Eyy Ra, bl ( (8 E,,. aT m-
Proof. To simplify the notation without causing ambiguity, we write g(x) and i1(x) as 9 and â, respectively. Moreover, we write g(x: and (xx) as g and H, respectively.
Just like the conclusion in Equation 22 we can write the interaction between 8t and 8mh as follows.
ammm 1)m²
Tab(6 = 2m2 Ingn Hab +
go Har "(Hrrlaa))
2
a'=1
(30)
a (m 1)m² 9o H
Σ
(Hav
(: Ry
where a denotes the step size, and m denotes the total number steps. enable fair comparisons, we use the same step size and number of steps as multi-step attack to make the magnitude of
d.r match the magnitude of 6multi- RH represents terms with elements in / of higher than the second order, and Íिz (6 represents terms with elements d of higher than the second order.
In this way, according to Equation (30) and Equation (22 the expectation of the difference between
Las (8%) and Lab (8m is given as follows.
E,,b [Ta((dr) (ab(dmutaa)
a² pm 1)m² Enb { Ex 12 - H2
+ 0f 9 H2 [1] + E,, [RT]]
2
26

---

Published as a conference paper at ICLR 2021
where
R a a 1)m²
Σ
[Gag Har - 9a9a Harllarr)
2
Vab
Σ
(ingn Has Haw 9oaw HarHHa))
Vha
a²m² [agg lao
R (6 It (8mi))) + RZ 12 (H)
The expectation of Peg is give as follows.
E, [Ral]
a² ammm 1)m²
Bagh Ey [Vat] + RE [Vba] a(m F5
Baßh Bagh Hab gagh Hab
2
B., [R 6Tt))
R2 (6 + R (H) Reulii (H) 2 0.
have RH O. and Reguli (H) ~ O. Note that the magnitude of 5t2 and the magnitude of Ematt are small, then Rtg (oT) as O, and R2 (&mma))) s According to
According Assumption
Lemma we have EE,r (agssAa - Ja9b Hab O, Err
(gağa Has Harb JaDa Hat Ha'b)) 0.
Therefore, get EE, [Vab] 0. In this way, EE, E,, P O.
Furthermore, according to Lemma we have E2 12 Ex, [9 < 0.
Therefore,
EyEu, b (6xx))r)) - Las (5mutu))
(m 1)m2 Eaab {
TE
[/2 122 ab
n H22
+ E,
06 H2 H2H }
E. Fa,b [N
2
a 2m(m 1)m²
Enb
Fer A2. 5 H2
+ Ey
g6 Hr.
H2 +
2
0
I PROOF OF PROPOSITION 3
To simplify the problem setting, we do not consider some tricks in adversarial attacking, such as gradient normalization and the clip operation. Note that the original MI Attack and the multi-step attack cannot be directly compared, since that magnitudes of the generated perturbations cannot be fairly controlled. The value interactions is sensitive to the magnitude of perturbations. Comparing perturbations with different magnitudes is not fair. Thus, we slightly revise the MI Attack as
def Imi p (1 vll + 8tm v).
(31)
where t denotes the step and (t 1)/t. {(h(x),)) is referred as the classification loss. To simplify the notation, we use g(x) denote v.L(h(=), v), i.e. g(x) v
In MI attack, the final perturbation generated after steps is given as follows.
[-1
5r def a
g
27

---

Published as a conference paper at ICLR 2021
where represents the step size. Furthermore, we define the update of perturbation with the MI attack at each step as follows.
A def
Imi
(32)
In this way, the perturbation can be written follows.
8th
Amti + Ar
Aot-1 m
(33)
Lemma 4. The update of the perturbation with the MI attack at step defined in Equation 32)
can written as Ar = a [T - H(x) + R4 (H))) g(x) + RG, where R4 (A(H(()) (x)) denotes terms of elements in (x) higher than the first order, and Ri denotes terms with elements in 8 of higher than the first order.
Proof. Ift 1, Ar = a g(x).
Let VL' <, Ax + R (H(x)) g(x) - P2..
According to Equation
(31) and Equation 32 we have
Ao
t - 1 4-1
+ 7o(o + 5tt
Imi
Applying the Taylor series to the term of g(x + 6t--1,, we get
1 1-1 Imi + [9(x)) H +
Azt mi
(34)
where -: denotes terms of elements in -t11 of higher than the first order.
According to Equation 33 and Equation 34 get
Azi a
1-1 1
g + [o(x) + H(x) [Ari + Ax + ... + Art + +r x-
Because Vt' < Ardh = a a Hzx) + R (H(x)) g(x) + Rt. we have Ar--1
+ (1))(()))) g(x) + Rt According to Equation 32 we get 9 [ (+o ( -H(() ((5) n
t-1
a
28

---

Published as a conference paper at ICLR 2021
In this way, we get
t
t-2 +-2H Hxx) + R (H{)) g(x)
Arii = a
T
2
(t - 2)(1 1) H(x)) + ER((H()))
+ 1 (x)
a(l 1)
g(x))
4
d=1
(1 (1-2)(1-1) 1) H²(x)
1 H(x)
Ri- (H()))
Σ (H(())
d==1
R((M(a)
t-1
+I a 2
H(x)+
g(z) +
Σ
R
+
=a I f 2
(x) + R g(0) + Rt..
where R1(H(() denotes terms of elements in H(x) higher than the first order, and Rt denotes terms with elements in 8t-1 of higher than the first order. In this way, we have proved that Vt > mi Ar mi = a [I + - (x) + R3 (x))) g(x) + R
Proposition 3. The adversarial perturbation generated by multi-step attack is denoted by 6 - V vel(h( + )mmn)) The adversarial perturbation generated by multi-step attack incor- E
porating the momentum is computed 8mm
m- Dim Perturbation units of 8m exhibit smaller I Eyj m
a
interactions than 8m i.e. Eigl [ip (m))
Proof. According to Lemma given as follows.
4 the update of the perturbation with the MI attack at the step is
Azi. = I 1-1 H(x) +(())) g(z) + R 2
(35)
where Ri (H()) denotes terms of elements in (xx) of higher than the first order. R denotes terms with elements in 5t-1 of higher than the first order.
To simplify the notation without causing ambiguity, we write g(x) and (xx as g and H. respec-
tively. In this way, according to Equation (33 and Equation (35 8m can be written as follows.
am (m
m
8mh a mI
+
H
R
(H)
g Σ
R
(36)
where m represents the total number steps. According to Lemma between perturbation units a, in 5t is given as follows.
T the Shapley interaction
Las (6m) = 8th 8T Hab + (2m)).
(37)
where R2(5m) denotes terms with elements in 7mi of higher than the second order.
29

---

Published as a conference paper at ICLR 2021
According to Equation (36) and Equation (37 get
+ a mm(m m 1)
(am = Hos amg.
(o +..
Σ 1=1
o ( 6
I[
4
of higher than the first order. which corresponds
R2 in Equation
36
amm(m - 1)
a
omgh +
2
(Habga') + 4..44 Σ
0(8 mi.b)
1+ (2(
4
=1=1
6
higher than the first order, which corresponds i in Equation 36
m2 amm29o.goH gaggHHb
firt-orddr woEt elements in
a mm
mm
am 1)m?
9b
(H
(v)
9a Σ (Ha'bDa))
Hab
4
b==1
a'=1
second arder terms w.rt. elements in
a mm - 1)2m² m
n
+
Hargoo go Σ
Harrga)) (ergga)Har Hab + +...
16
V=1
(H)
06 Mal + [o(6 Hasomi Rt2()))
3 a 1)m
n
m2 9a96 Hab
Ia Hab
E (Ha'baa))
4
ammm 1)m²
Tb Hab
(Hatt gw) + (2(m)))
RTd (H),
8=1
(38)
where RR' (H) denotes terms of elements in H higher than the second order, and R's (5)) denotes terms of elements in 8t higher than the second order
According to Equation (22 and Equation 38 the expectation of the difference between Tab (8m)
and (8 mmni)
given as Tollows.
Ba,b Hat (6 (6
3 (m 1)m²
Eas go Hab a'=1
Σ
(Harda + Hob Σ (Hargw)) + E.,b [
6/
Unb
Una
where
Rmi (8 R' (6 + Rmd (H) R.muuti (H).
According to Assumption 1, we have Rm s 0, and Ruuli (H) 0. Note that the magnitude
of 6m and the magnitude of 8m are small, then R (6 P 0. and Rg (o 0. Therefore, ab. E.,' [R's (6m) (8 (m RTT (H) muti (H) 0. Moreover, similar Equa-
E,,b
Rmi
tion
24h in the proof of Proposition have R.,b (Uat] E. [Uba] 2 0.
30

---

Published as a conference paper at ICLR 2021
Therefore,
Ba (mm()
T Tab Tab(omuli)
a²(m
1)m²
Ba, (Vab + Ubal + BB,, [Rab)
4
a am
1)m?
Eab (Uab] +050.
2
Note that Proposition just shows the revised MI Attack usually decreases the interaction between
perturbation units. The proof towards all types of MI Attacks is still challenge.
J IMPLEMENTATION OF THE INTERACTION-REDGEE ATTACK (IR ATTACK)
J.1 GRID-LEVEL INTERACTIONS FOR IMAGE DATA
Although the computation of E,, (45(() can be simplified using Equation (6) the computational cost of F,, L, is still high. Therefore, as Figure 6 shows, using the local property of images (Chen et al. 20). we can divide the entire image into L x L grids, and compute interactions at the grid level. instead of the pixel level. Let A {A1., ... ALL denote the of grids. We use (p. q) to denote the coordinate of a grid. In this way, the expectation of interactions between perturbation grids is given as follows.
(ppa))traaa ((06) T2 - (())) (o(A) - w(A (Apa3) v((Am))) v0)), (39)
Apg for the grid (p.q)
Figure 6: For the input image, we can divide the image into grids, and compute interactions the grid level.
J.2 SCALABILITY OF THE INTERACTION LOSS
In this section, we discuss about two kinds scalability of the interaction loss.
Is the computational cost of the interaction loss affordable when the number of players is large? To this end, we have proved in Equation (4) that the computational complexity of the expec- tation of the interaction is linear, which is scalable. In fact, we do not directly compute interaction using Equation (3). Instead, we compute the expectation of interactions with Equation (4) The computational cost of the IR Attack can be further reduced by calculating the grid-level interactions of images.
We further conducted experiments to measure the time cost of generating perturbations using the IR Attack. We conducted the IR attack for 100 steps on the ImageNet dataset. The time cost was measured using PyTorch 1.6 (Paszke al. 2019) on Ubuntu 18.04, with the Intel(R) Core(TM) 17-9800X CPU 3.80GHz and a Titan RTX GPU. Table 6 shows the average computational cost of generating adversarial perturbations 100 steps. It shows that the IR Attack is computationally applicable to nigh-dimessionll data and deep neural networks.
input image with size 224 224 by the IR Attack for
31

---

Published as a conference paper at ICLR 2021
Table 6: Average computational cost of generating adversarial perturbations input image by the IR Attack for 100 steps on different source DNNs.
RN-34
RN-152
DN-121
DN-201
Time (seconds)
12.882
48.774
27.519
44.812
Is the computation cost of the interaction loss affordable when we consider the continuous space of adversarial perturbations? It has been widely discussed (Ancona et al. 2019 Sundarara- Jan & Najmi 2019) that when applying the Shapley value, the feature space is regarded as binary. It because as (Sundararajan & Najmi 2019) shows that although there exist the Shapley-value attribution in continuous space, only the Shapley value in the binary space is the unique attribution that satisfies the linearity axiom, the dummy axiom, the symmetry axiom, and the efficiency axiom that only in the binary space. Thus, when we compute the interaction, the perturbation can be re- garded the binary space, i.e., whether the perturbation unit added to the input or not, which enables scalability.
K EVALUATION OF THE TRANSFERABILITY VIA LEAVE-ONE-OUT VALIDATION
As Figure 7 shows, the highest transferability of the MI Attack is achieved in an intermediate step, rather than in the last step. This phenomenon presents challenge for fair comparisons of the transferability between different attacking methods.
To this end, in order to enable fair comparisons of transferability between different methods, we estimate the adversarial perturbations with the highest transferability for each input image the leave-one-out (LOO) validation
follows. Given set of clean examples {(T;, vs)) where y: E (1.2....,CC C),. we use xf to denote the adversarial example at step w.tt. the clean ex- ample Ii. where t {1,2,.... ... TT).. and T is the number of total step. Given target DNN h(-) and an input example x, where h(.)) denotes the output before the softmax layer, we use C(x) arg max; hx(2), le E {1..... C) to denote the prediction of the example x.
def 1 s.l. arg max (eE...2.....NM.. 4
[1(()) + wr]
where [.] is the indicator function. Then the average transferability is given as follows. Transferability = E, [C())) # vil]
best transferabillty
NI
0.8
Transferability
0.4
0.1
epoch
Figure 7: The curve of transferability in different steps.
L ADDITIONAL RELATED WORK
Some studies paid attention to intermediate features to improve transferabilityy Activation At- tack (Inkawhich et al.) 2019) forced the intermediate features of the input image to be similar with the intermediate features of a target image, in order to generate highly transferable targeted example. Distribution Attack Lichw al. 2020) explicitly modeled the feature distribution of each class, and improve the targeted transferability by driving the feature of perturbed input image into the dis- tribution of specific target class. Intermediate Level Attack (Huang et al. 2019) improved the
32

---

Published as a conference paper at ICLR 2021
transferabilityy of adversarial example by maximizing the feature perturbation of a pre-specified layer. In comparison, we explain and improve the transferability based on game theory. Moreover, discover the negative correlation between the transferability and interactions.
M ADDITIONAL EXPERIMENTS ON EEERRACTION-REDUCC Loss
M.I INTERACTION REDUCTION ON OTHER ATTACKS
To further demonstrate the effectiveness of the interaction loss, we have applied the interaction loss on other attacks besides the PGD Attack, including the MI Attack, the SGM Attack, and the VR Attack. More specifically, we added the interaction loss on the MI Attack (namely the MI+IR Attack), the SGM Attack (namely the SGM+IR Attack), and the VR Attack (namely the VR+IR Attack), respectively.
For the MI Attack and the SGM Attack, we directly applied Equation 7 to these attacks, because these attacks were compatible with the interaction loss. Besides, for the VR Attack, its objective function is given as follows.
maximize BE-N((.00)) [((h(x 6 s.t. 1 &e [0,1]"
(40)
Therefore, the VR+IR Attack was implemented via sampling as follows.
maximize 1
[(h(x + 5 Ex). - Ray (b)()))), & ~ N (0,03)
K
(41)
s.t. | e, x + 1]"
where the interaction loss was computed by considering the input image as + εx. rather than Equation (26). The VR Attack reported Table followed the original paper (Wu et al.] 2018) to set K = 20. However, crucial issue for applying the interaction loss to the VR attack was its extremely high computational cost. Therefore, for the implementation of the VR+IR Attack, we set K" = 5 and reduce the number of steps from 100 50. Just like experiments in Table T also used the I.00 strategy for evaluation.
Table 7 Table and Table compare the success rates of attacks with and without the interaction loss. The results demonstrated that the performance of the MI Attack, the SGM Attack, and the VR Attack can be further enhanced by directly adding the interaction loss to reduce interactions inside perturbations.
Table 7: The success rates of Loe black-box attacks crafted by MI and MI+IR on four source models (RN-34/152, 1N-121/201) against seven target models. The interaction loss can boost the transferability of MI.
Source Method
VGG-16
RN152
DN-201
SE-154
IncV3
IncV4
IncResV2
MI
80.1
73.0+2.3
77.7+0.5
48.9=0.8
46.2+1.2
39.910.5
34,8+2.5
RN-34
MI+IR
90.0+0.5
85.7+0.3
88.5+0.6
67.0=0.1
66.9=1.8
60.2+0.7
53,9+2.3
MI 70.3+00.6
74.8=1.4
51.7=0.8
47.1=0.9
40.5+1.6
36.8+2.7
RN-152
MI+IR
7..9:11.4
82.2+2.00
68.3-0.3
63.6=1.2
59.0-0.4
56.3
MI 83.0+14.9 72.0+-0.7
91.5+002
58.4=2.6
5..6+1..6
49.2+2.4
43.9+1..5
DN-121
M1+1R
89.0+1.88
83.2+1.5
93.4+0.6
74.2=0.7
69.6:1..9
64.7.0.5
58.2+2.3
MI
77.3+:0.8
74.8+1.4
6..6=1.0
56.5+2.5
51.1=2.1
47.8=199
DN-201
MI+IR
87.3+0.3
81.6+2.00
75.4=0.6
66.6=3.3
60.0=1.0
62.1-1077
M.2 ATTACKS ON THE CIFAR- DATASET
In Table we have shown that reduction of interactions could improve the adversarial transfer- ability on the ImageNet dataset (Russakovsky et al., 2015). To further demonstrate the broad ap- plicability of such a negative correlation, we also conducted the targeted attack the CIFAR-10 10 dataset (Krizhevsky & Hinton 2009) to test the transferability of perturbations generated with the interaction loss.
33

---

Published as a conference paper at ICLR 2021
Table 8: The success rates of Loo black-box attacks crafted by SGM and SGM+IR four source
models (RN-34/152, DN-121/2011 against seven target models. The interaction loss can boost the transferabilityy of SGM.
Source
Method
VGG-16
RNI52
DN-201
SE-154
IncV3
IncV4
IncResV2
SGM
91.8:10.6 89.0.10.9
90.0.10.4
68.0.11.4
63.910.3
58.2.1.11
54.6:11.2
RN-34
SGM+IR
94.7.10.6
91.7-10.6
93,410.88
72.710.4
68.910,9
64.1-11.3
61.311.0
SGM
88.2+0..5
90.2++..
72.7+1.4
63.2=0.7
59.1=1.5
58.1+1.2
RN-152
SGM+IR
92.0+1.0
92.5+0.4
7..3+
69.6=0.8
66,2++1.0
63.6+00.9
SGM
88.7:+0.9
88. 1.0
98.0+0.4
78.0+0.9
64.7:22.5
65.44:2.
59.7+1.77
DN-121
SGM+IR
91.7-+0.22 90.4+1044
94.3+0.1
07.0
78.8
79.5+0.2
75.84-2.77
SGM
87,3+0,3
1..4=1.0
82.9
72.3=0.3
71.3:1.66
68,8+0.5
DN-201
SGM+IR
89.5=0.9
91.8+0.7
87.3+1.2
82.5=0.8
80.3+0.3
81.5+0..
Table 9: The success rates of Lxs black-box attacks crafted by VR and VR+IR on four source models (RN-34/152, DN-121/201) against seven target models. The interaction loss boost the transferability of VR.
Source
Method
VGG-16 16
RN152
DN-201
SF-154
IncV3
IncV4
IncResV2
VR
85.1
85.3
87.0
55.7
54.3
50.7
43.7
RN-34
VR+IR
90.8
92.2
93.3
75.4
75.4
67.5
66.1
VR
88.8
88.4
98.2
72.9
73.5
72.5
63.6
DN-121
VR+IR
93.0
93.5
96.2
83.7
82.8
84.0
79.8
Following Wu et al. (2018), we chose three DNNs the source DNN or the target DNN, which included: LeNet (LeCun et al. 1998), RN-20 (He et al. 2016), and DN-121 (Huang et al. 2017). We conducted the targeted attack under the Loo norm constraint, and chose he plane class as the target category. The norm constraint was set to 16/255, and the step size was set to 2/255. The transferability computed based the best adversarial perturbation during 50 steps via the leave- one-out (L.O)) validation., which has been introduced in Appendix K
As Table 10 shows, the transferability could be enhanced by reducing interactions the targeted attack on the CIFAR-I0 dataset. Particularly, when the source DNN is RN-20 and the target DNN DN-121, the transferability improvement about 30%, which was considerable gain.
Table 10: The success rates of Loo targeted black-box attacks three source models, including LeNet, RN-20, DN-121, against three target models.
Source
Method
LeNet
RN-20
DN-121
PGD
34.1110.1
19.6:10.4
LeNet
PGD+IR
4.32+-03
29.7+-111
PGD 10.810.8
41.9#11.
RN-20
PGD+IR
19.7
71.8
PGD 10.0.10.7
44.2.10.3
DN-121
PGD+IR
18.9+
58.5+1.0
Table 11: The average interaction inside adversarial perturbations generated by PGD, DI and
Method
RN-34
DN-121
Baseline (PGD Attack)
0.422
0.926
DI Attack
0.241
0.499
TI Attack
0.379
0.618
N EMPIRICAL VERIFICATION OF OTHER TRANSFERABILITY-BOSSIIIS ATTACKS
We have theoretically analyzed the MI Attack, the VR Attack, and the SGM Attack. However, for other methods of improving adversarial transferability, such Diversity Input (DI) (Xie al.
34

---

Published as a conference paper at ICLR 2021
2019), which uses random data augmentation during attacking, is difficult to mathematically prove that they essentially reduce interactions. Nevertheless, as Table 11 shows, we empirically demon-
strated that two widely-used transferabiliy--boosiin attacks, DI and TI (Dong al. 2019). also reduced interactions.
O ADDITIONAL EXPERIMENTS ON EFFECTS OF THE INTERACTION Loss
We conducted additional experiments to test the effects of the interaction loss. We conducted attacks two source DNNs (RN-34, DN-121), and transferred adversarial perturbations seven target DNNs (VGGI6, RN-152, DN-201, SE-154, IncV3, Inc,4, IncResV2).
We used the following two experimental settings compare the transferability of adversarial per- turbations generated with different À values.
First, re-drew the curves in Figure B aa) by extending the from the range of [0, 1.2) the range of [0, 2.0]. in order to show the performance of different values. We simply changed the À value in the objective function (i.e. Equation (5)) without any other revisions. This was the most direct way to test the effects of A. Experimental results are shown in Figure
Besides above experimental settings, we also compared adversarial perturbations generated with different À values, when we controlled each perturbation to have the same attacking utility. The attacking utility was defined as follows.
Attacking Utility max h (x: 0 hy + 6)
y'#y
where y denote the label of the imput image z. This setting also ensured the faimess of comparisons
from new perspective. Please see Figure for experimental results.
In sum, under both experimental settings, we found that the large À value usually yielded a high adversarial transferability in our experiments.
Source: DN-121
Transferability
VCG16
SE154
ncV3
0.0
0.4
0.8
1.2
1.6
2.0
Figure 8: The success rates of black-box attacks with the IR Attack using different values of under the first experimental setting.
Source: RN-34
Source: DN- 121
0.8
0.8
Transferabilty
Transferability
0.4
SE154
IncV3
IncVi
IncResv2
0.0
0.0
0.4
0.8
1.2
1.6
2.0
0.0
0.4
0.8
1.2
1.6
2.0
Figure 9: The success rates of black-box attacks with the IR Attack using different values of À under the second experimental setting.
35


Reviews:
[{"review_id": "vD5zjFL1zp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe authors analyze the transferability of adversarial examples from the perspective of interactions based on game theory. They have discovered and shown  the negative correlation between the transferability and interactions inside adversarial perturbations. This discovery leads to an explanation of the adversarial transferability by the interaction inside adversarial perturbations. Thus, they proposed a new loss called interaction loss to penalize the interactions between perturbation units during attacking and experiments show the improvement of the adversarial transferability.\n\nStrengths\n1 An interesting understanding of adversarial transferability is provided.\n2 Mathematical proofs are admired.\n3 Results are very nicely presented, and writing is clear throughout the paper.\n\nWeaknesses\n1 Results are reported only on one dataset (ImageNet).\n2 The appendix needs to be revised for better presentation.\n3 The variances of the results are not reported.\n\nThis paper is well motivated because the authors observe the negative correlation between the transferability and interactions inside adversarial perturbations, and they provide a possible explanation of why the related research tasks  can improve the adversarial transferability. Further mathematical proofs and experiment results verify the observation. \n\nHowever, I have some questions about this paper.\n\nQuestions\n1.\tThere are many outliers off the blue shade in the subgraph of Figure 1. Could the authors give some interpretation of why there are so many outliers?\n\n2.\tFor Equation 4, the value of the expected interactions is equivalent to the expectation of the contribution for each pixel. The authors aim to minimize the value of the expected interactions that is the same to average the contribution to all pixels. That may contradict the idea of the one-pixel attack. More interpretation should be given to understand the concept of interactions.\n\n3.\tThe authors claim to provide a unified view to understand the enhancement of transferability; however the authors only explain three baseline models. Other types of adversarial example generation methods are not considered, such as Translation-Invariant Attack that the authors already mentioned in the related work. What is more, the Translation-Invariant Attack aims to compose the gradients of neighboring pixels together, which is contradictory to the authors’ idea of reducing the interaction of perturbations.\n\t\n\n4.\tFrom the experiment results in Table 1, why the performance of the IR attack is worse than the baseline when attacking RN-152 and DN-201 in the ensemble setting?\n\n5.\tIn Section 5, Experiments, baselines: “the transferability of each baseline was computed based on the best adversarial perturbation during the 100 steps…” and you also mention that “Previous studies usually set the number of steps to 10 or 20”. I am confused about why you set the step to be 100? For a fair comparison, you should follow the setting in previous studies.\n\n6.\tIn Section 5, Experiments, baselines: “we set $\\lambda = 1$ for the IR Attack, when the source DNN was ResNet, and set $\\lambda = 2$, for other source DNNs.”, but in Figure 3, part a, the range of $\\lambda$ is from 0 to 1.2 for DN121, which does not include the \\lamda value the authors suggested. \n\n7.\tThe computational cost is not discussed. As the authors said in Section 5 “the computational cost of the interaction loss is intolerable”. No discussion about the running time is provided. Moreover, the authors choose the step to be 100, which further increases the running time.\n\n8.\tThe third line on page 5, the term $|\\delta|_{p}^{p}$ is wrong. \n\n        It should be $|\\delta |_{p}$. \n\n9.\tAppendix: The is some problem with the reference format about Figure 4 and Figure 7. I also suggest reorganizing the appendix.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IaMI146e9hO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a negative correlation between the adversarial transferability and the interaction between coordinates.  The interaction is defined by the Shapley values used in the game theory to measure the contribution of players.  The author(s) defined the interaction between coordinates (players) as the difference between the joint contribution and the sum of conditional contributions.  They show a negative correlation between the adversarial transferability and the interaction with various known adversarial attacks.\n\nThis work also shows an improvement in the transferability of adversarial perturbations by incorporating the interactive loss with the classification loss. \n\nWhat is missing in the paper is the reason and motivation for using the Shapley values for defining the interaction.  No comparison and discussion is given the difference between their method and other methods that define the interaction between variables.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6GzUTk847K", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes that the transferability of adversarial attacks has a negative correlation with the interaction within an input perturbation. By defining the interaction of perturbations with the Sharpley value, it can quantify the interactions and demonstrate the negative correlation with the transferability. Furthermore, this work shows that prior work on adversarial attacks (e.g., VR attack and MI attack) can be explained by the (expected) interaction scores. This work further demonstrates that the way of enhancing transferability by minimizing the interaction within input perturbations, with the experiments on the image classification task. \n\nOverall, I think this work provides a new perspective of understanding transferability and presents solid analysis/experiments to verify the hypothesis. \n\nOnly one comment about the definition of interaction scores. In some literature [Lundberg et al., 2019], it is called the Shapley interaction index, which uses the definition in equation (13) of Appendix D. Shapley interaction index has mainly been used in the machine learning literature recently for explaining feature interactions within models. E.g., \n\n1. Lundberg et al. Consistent Individualized Feature Attribution for Tree Ensembles. 2019\n2. Chen and Ji. Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers. 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4Kk5Lqynsa", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper mainly deals with the negative correlation between adversarial transferability and the interaction inside adversarial perturbations. The authors claimed that utilizing the correlation can be regarded as a unified perspective to understand previously proposed methods. To this end, they presented an adversarial attacking loss, which can directly reduce the interaction, defined as the individual interaction between two perturbation units to the total reward function.\n\nThis paper provides an intriguing perspective that can explain adversarial attacking mechanisms. Although I am not sure about the exact settings that draw the numbers, the empirical results seem good; thus, reducing the interaction inside perturbations could be useful in practice. However, I cast doubt on the theoretical significance of the central hypothesis. This concern derives from lacking analyses on 1) a valid (theoretical) reason for the negative correlation, 2) qualitative benefits of directly reducing the interaction based on the game theory. \n\n[Quality]\n\nThe paper follows one solid logical structure aforementioned in Abstract and Introduction. However, I think Section 3 & 4 are confusing, which indeed are the core parts. For example, the propositions are overly verbose and hard to follow. Also, the theoretical statements and interpretations are not well aligned. I defer the detailed comments below.\n\n[Originality]\n\nThe originality of the paper is not outstanding, but sufficient for acceptance. \n\n[Significance]\n\nThe paper reveals one aspect of adversarial transferability for sure. However, bringing the game theory-based approach in this domain is quite unconventional. Hence, I strongly feel there should've been more theoretical arguments that strengthen the significance of this approach. I guess the authors' intention was to prove their claims empirically, and the paper provides some good results. Unfortunately, I feel this work is reluctant to compare the IR method with the State-of-the-Art methods (such as \"MI vs. vanilla IR,\" \"VR vs. vanilla IR,\" \"SGM vs. vanilla IR\").\n\n\n[Comments & Questions (sorted by priority)]\n1. As far as I understand from Proposition 1, it says that \"multi-step attacks generate more interaction than single-step attacks.\" Please elaborate that this statement can be generalized to the following: \"the adversarial transferability and the interactions inside adversarial perturbations are negatively correlated.\" \n2. In Table 3, why is the result of (vanilla) IR Attack not reported? The table shows that HyridIR, a combination of all techniques, achieves the best performance. Hence, it implies that reducing the interaction in multiple ways can be stacked for achieving good performance. However, the table does not show the success rate of the pure IR attacking method. My concern is whether solely applying the IR attacking method has a clear contribution to be accepted in this conference. Please describe how IR attack (using only Eqn. 5) will perform compared to the SGM attacking method, which seems to be one of the State-of-the-Art. \n3. According to the definition (Eqn. 3), the actual computation of the interaction is not very scalable when the set of players (\\Omega) is large. I think Eqn. 4 also lacks scalability because it is natural to think that the set of the adversarial perturbations is a continuous space.  \n4. Is high $\\lambda$ always effective? In Fig. 3, how the success rate changes when $\\lambda = 10^2, 10^3, \\dots$? In other words, I want to know when the tendency of Fig. 3 (a) converges to Fig. 3 (b) as $\\lambda$ goes higher.\n5. (Similar to #2) Why are the results of MI, VR, SGM methods not presented in Table 1?\n6. Some of the arguments, especially propositions, are unnecessarily lengthy and hard to follow. To improve readability, I suggest clarifying the main points in the theoretical arguments. For example, please remove the redundant part \"Given an input image $x \\in\\mathbb{R}^n$ ... $\\delta^m_{multi}), y)$.\" in Props. 1, 2 & 3 and make this statement as a proper Definition.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work provides interesting insights on the transferability of adversarial perturbations and proposes ways of making it more effective. While several reviewers have found parts of the paper unsatisfactory, there are interesting results to merit acceptance.
Decision: Accept (Poster)