Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets
Authors: Dongxian Wu
Yisen Wang
Shu-Tao Xia
James Bailey
Xingjun Ma
Venue:  (2020)
Keywords: Adversarial Example
Transferability
Skip Connection
Neural Network

Abstract: Skip connections are an essential component of current state-of-the-art deep neural networks (DNNs) such as ResNet, WideResNet, DenseNet, and ResNeXt. Despite their huge success in building deeper and more powerful DNNs, we identify a surprising \emph{security weakness} of skip connections in this paper. Use of skip connections \textit{allows easier generation of highly transferable adversarial examples}. Specifically, in ResNet-like (with skip connections) neural networks, gradients can backpropagate through either skip connections or residual modules. We find that using more gradients from the skip connections rather than the residual modules according to a decay factor, allows one to craft adversarial examples with high transferability. Our method is termed \emph{Skip Gradient Method} (SGM). We conduct comprehensive transfer attacks against state-of-the-art DNNs including ResNets, DenseNets, Inceptions, Inception-ResNet, Squeeze-and-Excitation Network (SENet) and robustly trained DNNs. We show that employing SGM on the gradient flow can greatly improve the transferability of crafted attacks in almost all cases. Furthermore, SGM can be easily combined with existing black-box attack techniques, and obtain high improvements over state-of-the-art transferability methods. Our findings not only motivate new research into the architectural vulnerability of DNNs, but also open up further challenges for the design of secure DNN architectures.

Published as a conference paper at ICLR 2020
SKIP CONNECTIONS MATTER: ON THE TRANSFER- ABILITY OF ADVERSARIAL EXAMPLES GENERATED WITH RESNETS
Dongxian Wu Yisen Wang Shu-Tao Xia James Bailey* Xingjun Ma¹
Tsinghu University
Shanghaii Jiao Tong University
3PCL Research Center of Networks and Communications, Peng Cheng Laboratory
"The University of Melbourne
ABSTRACT
Skip connections are an essential component of current state-of-the-ard deep neu- ral networks (DNNs) such as ResNet, WideResNet, DenseNet, and ResNeXt. De- spite their huge success in building deeper and more powerful DNNs, we identify a surprising security weakness of skip connections in this paper. Use of skip con- nections allows easier generation of highly transferable adversarial examples. Specifically, in ResNet-like (with skip connections) neural networks, gradients can backpropagate through either skip connections or residual modules. We find that using more gradients from the skip connections rather than the residual mod- ules according to a decay factor, allows one to craft adversarial examples with high transferability. Our method is termed Skip Gradient Method (SGM). We conduct comprehensive transfer attacks against sfate-ofthee-ee art DNNs including ResNets, DenseNets, Inceptions, Inception Reste,, Squeeze-ang and -xcitxxitttiin Net- work (SENet) and robustly trained DNNs. We show that employing SGM on the gradient flow can greatly improve the transferability of crafted attacks in almost all cases. Furthermore, SGM can be easily combined with existing black-box at- tack techniques, and obtain high improvements over state-of-the-art transferability methods. Our findings not only motivate new research into the architectural vul- nerability of DNNs, but also open up further challenges for the design of secure DNN architectures.
INTRODUCTION
In deep neural networks (DNNs), skip connection builds short-cut from shallow layer to a deep layer by connecting the input of a convolutional block (also known as the residual module) directly to its output. While different layers of neural network learn different "levels" of features, skip con- nections can help preserve low-level features and avoid performance degradation when adding more layers. This has been shown to be crucial for building very deep and powerful DNNs such as ResNet (He et al. [0166 WideResNet (Zagoruyko Komodakis (016). DenseNet (Huang et al.) 2017) and ResNeXt (Xie et al.) 2017). In the meantime, despite their superior performance, DNNs have been found extremely vulnerable
adversarial examples (or attacks), which are input examples slightly perturbed with an intention to fool the network to make a wrong prediction (Szegeyy et al.) 2013 Goodfellow et al.] 2014; Ma et al. 2018 Bai al.) 2019) Wang al. 2019: 2020). Adver- sarial examples often appear imperceptible to human observers. and are transferable across different models (Liu et al. 2017). This has raised security concerns the deployment of DNNs in security critical scenarios, such as face recognition (Sharif et al.) 2016), autonomous driving (Evtimov et al.
2018), video analysis (Jiang al., 2019) and medical diagnosis (Ma et al. 2019).
Adversarial examples can be crafted following either a white-box setting (the adversary has full access to the target model) or black-box setting (the adversary has no information of the target model). White-box methods such as Fast Gradient Sign Method (FGSM) (Goodfellow al. (014)). Basic Iterative Method (BIM) (Kurakin et al. Projected Gradient Decent (PGD) (Madry et al. 2018) and Carlini and Wagner (CW) (Carlini & Wagner 2017) often suffer from low transferability
Cooresponnence Yisen Wang (eewangyisen @ @gmail.comm

---

Published as a conference paper at ICLR 2020
99
Skip connection
94.46/23.04
100/62.10
76.4/111.34
44.38/7.22
96.78/13.96
100/55.24
f2-2
fi-1
fi
96. 36.20/18.66
Residual module
50.06/7.74
99.58/34.76
46,02/7.00
30.06/5.48
80.86/9.44
100/51.20
103/52.2
ft-2
fe-1
f
Left: Illustration of the last skip connections (green lines) and residual modules (black boxes) of a ImageNet-trained ResNet-18. Right: The success rate (in the form of "white-box/black box") of adversarial attacks crafted using gradients flowing through either a skip connection (going upwards) or residual module (going leftwards) at each junction point (circle). Three example backpropagation paths are highlighted in different colors, with the green path skipping over the last two residual modules having the best attack success rate while the red path through all 3 residual modules having the worst attack success rate. The attacks are crafted by BIM on 5000 ImageNet validation images under maximum Loe perturbation = 16 (pixel values are in [0, 255)). The black-box success rate is tested against VGG19 target model.
Figure I:
black-box setting, thus posing only limited threats DNN models which are usually kept secret in practice (Dong et al. 20188) Xie et al. 2019). Several techniques have been proposed to improve the transferability of black-box attacks crafted on surrogate model, such momentum boosting (Dong et al.) 20 diverse input (Xie et al., 2019) and translation invariance (Dong et al., 2019). Although these techniques effective, they (as well white-box methods) all treat the entire network (either the target model or the surrogate model) as a single component while ignore its inner architectural characteristics. The question of whether or not the DNN architecture itself can expose more transferability of adversarial attacks is an unexplored problem.
In this paper, we identify one such weakness about the skip connections used by many state-of-the art DNNs. We first conduct toy experiment with the BIM attack and ResNet-18 on the ImageNet validation dataset (Deng et al., (009) to investigate how skip connections affect the adversarial strength of attacks crafted on the network. At each of the last 3 skip connections and residual modules of ResNet-18, 18, we illustrate the success rate of attacks crafted using gradients backpropagate the residual module in Figure
through either the skip connection success rate drops more drastically whenever using gradients from a residual module instead of the skip connection. This implies that gradients from the skip connections are more vulnerable (high success rate). addition, we surprisingly find that skip connections expose more transferable information. For example, the black- success rate was even improved from 52.52% to 62.10% when the attack skips the last two residual modules (following the path in green color).
As can be observed, the
Motivated by the above observations, this paper, we propose the Skip Gradient Method (SGM) to generate adversarial examples using gradients more from the skip connections rather than the residual modules. In particular, SGM utilizes a decay factor to reduce gradients from the residual modules. We find that this simple adjustment adversarial examples, and the more skip connections in a network, the more transferable are the crafted attacks. This is in sharp contrast to the design principles (e.g., "going deeper" with skip connections) underpinning many modern DNNs. particular, our main contributions are:
the gradient flow can generate highly transferable
We identify one surprising property of skip connections in ResNet-like neural networks, i.e. they allow an easy generation of highly transferable adversarial examples.
We propose the Skip Gradient Method (SGM) to craft adversarial examples using gradients more from the skip connections. Using a single decay factor on gradients, SGM is an appealingly simple and generic technique that can be used by any existing gradient-based attack methods.
We provide comprehensive transfer attack experiments, from different source models against 10 -aute-of-the-art DNNs, showing that SGM can greatly improve the transferability of crafted adver- sarial examples. When combined with existing transfer techniques, SGM improves the of the-art transferability benchmarks by large margin.
2

---

Published as a conference paper at ICLR 2020
2 RELATED WORK
Existing adversarial attacks can be categorized into two groups: 1) white-box attacks and 2) black- box attacks. In the white-box setting, the adversary has full access to the parameters of the target model, while in the black-box setting, the target model is kept secret from the adversary.
2.1 WHITE- BOX ATTACKS
Given a clean example z with class label y and target DNN model f. the goal of an adversary is find an adversarial example Tade that fools the network into making an incorrect prediction (eg. f (Zudd) # y), while still remaining in the e-ball centered at æ (eg. [Rado - x xo S e).
Fast Gradient Sign Method (FGSM) (Goodfellow et al. 2014). FGSM perturbs clean example for one step by the amount of c along the gradient direction:
Bade
sign V (f((f())) v)).
The Basic Iterative Method (BIM) (urakin et al. 2016) is an iterative version FGSM that per- turbs for T steps with step size e/T.
Projected Gradient Descent (PGD) (Madry et
al. 2018). PGD perturbs normal example for T
steps with smaller step size. After each step of perturbation, PGD projects the adversarial example back onto the e-ball of if goes beyond the e-ball:
x: adv
II. [I + a sign(V VV 9))),
where (-) is the projection operation. Different to BIM, PGD allows step size > e/T.
There are also other types of white-box attacks including sparsity- based methods such as Jacobian- based Saliency Map Attack (JSMA) (Papernot et al., 6), sparse attack (Modas et al. [201)). one- pixel attack (Su al. 2019), and optimization-bann methods such Carlini and Wagner (CW) (Carlini & Wagner 2017) and elastic-net (EAD) (Chen et al. 2018).
2.2 BLACK-BOX ATTACKS
Black-box attacks can be generated by either attacking a surrogate model or using gradient estima- tion methods in combination with queries the target model. Gradient estimation methods estimate the gradients the target model using black-box optimization methods such as Finite Differences (FD) Chen et al. 2017 Bhagoji et al. 2018) or Natural Evolution Strategies (NES) (Iyas et al., 2018 Jiang et al. 2019). These methods all require a large number of queries to the target model, which not only reduces efficiency but also potentially exposes the attack. Alternatively, black-box adversarial examples can be crafted on a surrogate model then applied to attack the target model. Although the white-box methods can be directly applied on the surrogate model, they are far less effective in the black-box setting (Dong al. 2018 Xie al.) (219)). Several transfer techniques have been proposed to improve the transterability of black-box attacks.
Momentum Iterative boosting (MT) (Dong al. 2018). MI incorporates momentum term into the gradient to boost the transferability:
Valls (w) xc 1+1 IT, (2 + a- sign(g ggggggg. gh+1=111 =--1=p0 --t- + V (f (a v1
where g is the adversarial gradient at the 1-th step, a = e/T is the step size for a total of steps, J is a decay factor, and
is the L1 norm.
Diverse Input (DI) (Xie et al. 2019). DI proposes craft adversarial exampels using gradient with respect to the randomly -rrnsfoorrdd input example:
x
1++ adv
= II, (2 + a sign ( (p)
(4)
where H ( i P) is stochastic transformation function on æ d for a given probability p.
Translation Invariant (TI) (Dong et al., 2019). TI targets to evade robustly trained DNNs by generating adversarial examples that are less sensitive to the discriminative regions of the surrogate model. More specifically, TI computes the gradients with respect to a set of translated versions of the original input:
1+1 (T 'du + sign(W (al(s (o v)))
(5)
3

---

Published as a conference paper at ICLR 2020
where W is predefined kernel (e.g., uniform, linear, and Gaussian) matrix of size (2k + +) + 1) (k being the maximal number of pixels to shift). This kernel convolution is equivalent to the weighted sum of gradients over (2k: + 1)² number of shifted input examples.
Furthermore, there are other studies focusing on intermediate feature representations. For example, Activation Attack (Inkawhich et al. 2019) drives the activation of a specified layer on a given image towards the layer of larget image, yield a highly transferable targeted example. Intermediate Level Attack (Huang et al., 2019) attempts to fine-tune an existing adversarial example for greater black-box transferability by increasing its perturbation on a pre-specified layer of the source model.
Although the above transfer techniques are effective, they (including white-box attacks) either 1) treat the network (either the surrogate model or the target model) as single component or 2) only use the intermediate layer output of the network. other words, they do not directly consider the effects of different DNN architectural characteristics. [i et al. (2018) investigated the use of skip connections and dropout layers for sampling networks, which generates a huge set of ghost networks to perform an ensemble attack. Here, we focus on the architectural property of skip connections from the gradient view without modifying generating any extra networks.
PROPOSED SKIP GRADIENT ATTACK
In this section, we first introduce the gradient decomposition of skip connection and residual module. Following that, propose our Skip Gradient Method (SGM), then demonstrate the adversarial transferability property of skip connection via a case study.
3.1 GRADIENT DECOMPOSITION WITH SKIP CONNECTIONS
In ResNet-like neural networks, skip connection uses identity mapping to bypass residual layers, allowing data flow from shallow layer directly to subsequent deep layers. Thus, we can decompose the network into a collection of paths of different lengths (Veit et al. (016). We denote a skip connection together with its associated residual module as building block (residual block) of network. Considering three successive building blocks (eg. 2i+1 = 2; + fi+1(2;)) in residual network from input 2 to output 23, the output 23 can be expanded as:
Z3 = Z2 + fz(z2) = [2] + f2(21)] + fz(21 + f2(21))
(6)
| + fz0) + f2 + fi (=)))] + fa ((zo + fi(=)) + fa(zo + fi()))
According to the chain rule in calculus, the gradient of a loss function with respect to input can then be decomposed as,
OL de 0zg Oz2 dzi
de
0fz
0f2 0zi
0f
( Oz2
(1+
(1
)[
Ozo
8zg Oz2 0z1 Ozo
0zg
Ozo
Extending this toy example to network with L residual blocks, the gradient can be decomposed from L-th the (1 + 1)-th (0 1 < L) residual block as,
08
00
L
0f+t Oz,
+ +1) Ozi Ox:
(8)
0x
OZL.
==1
The example illustrated in Figure I is a the above decomposition of a ResNet- 18 at the last 3 building blocks (1 = L 3).
3.2 SKIP GRADIENT METHOD (SGM)
In order use more gradient from the skip connections, here, we introduce a decay parameter into the decomposed gradient to reduce the gradient from the residual modules. Following the decomposition in Equation (8. the "skipped" gradient is,
V=l=
0e
II i=0
L-1
(1
0f4
+ +1) Ozo
dzy
Oz.
dx
where 2u x is the input of the network, and y E (0, 1] is the decay parameter. Accordingly, given a clean example æ and a DNN model f, an adversarial example can be crafted iteratively by.
x
1+1 adv
= II, (Zadd + a sign( gzl 1=0
II (~: 0z:
I-
0f++
+ 0x ))
0zo
(10)
4

---

Published as a conference paper at ICLR 2020
Table 1: The success rates (%dst over 5 random runs) of black-box attacks (untargeted) crafted by PGD and its "skip gradient" (SGM) version, on different source models against a Inception V3 target model. The best results are in bold.
RN18
RN34
RNS0
RN101
RN152
DN121
DN169
DN201
PGD
23.23=00.99
24.38=0.41
22.80:10.55
22.98:1.883
26.56:1..55
30.71=0.60
30.9010.31
36.01=0..59
SGM 28.92:00.45 0.32
36.71+0.55 0..3 44.84=0.14 0.13 6,,45+0042 65.48++0.23
SGM is generic technique that can be easily implemented on any neural network that has skip connections. During the backpropagation process, SGM simply multiplies the decay parameter to the gradient whenever it passes a residual module. Therefore, SGM does not require any computa- tion overhead, and works efficiently even on densely connected networks such as DenseNets. The reduction of residual gradients is accumulated along the backpropagation path, that is, the residual gradients at lower layers will be reduced more times than those at higher layers. This is because, compared to high-level features, low-level features have already been well preserved by skip con- nections (see feature decompositions in Equation (
3.3 ADVERSARIAL TRANSFERABILITY WITH SKIP CONNECTIONS: A CASE STUDY
To demonstrate the adversarial transferability of skip connections, conduct a case study 10- step PGD, and their corresponding SGM versions, to investigate the success rates of black-box attacks crafted with or without manipulating the skip connections. The black-box attacks are gen- erated on 8 different source (surrogate) models ResNet(RN)- (00/BN/))88/////1100000 18/34/50/101/152 and DenseNet(DN)- 121/169/201, then applied to attack a Inception V3 target model. All models were trained on Ima- geNet training set. We randomly select 5000 ImageNet validation images that are correctly classified by all source models, and craft untargeted attacks under maximum Loo perturbation = 16, which a typical black-box setting (Dong et al. 2018 Xie et al. 2019: Dong et al. 2019). The step size of PGD was set to = 2, and the decay parameter of SGM was set to 7 = 0.5.
We run the attack for 5 times with different random seeds, and report the success rates (transferabil- ity) of different methods in Table T As can be seen, when the skip connections are manipulated with our SGM, the transferability of PGD is greatly improved across all source models. On all source models except RNI8, the improvements are more than 13%. Without SGM, the best transferability against the Inception- V3 target model is 35, 35.48% which is achieved by PGD on DN201, however, this is improved further by our proposed SGM to 65.38% ( (> 29% gain). This not only highlights the surprising property of skip connections in terms of the generation of highly transferable attacks, but also indicates the significance of this property, as such a huge boost in transferability only takes a single decay factor.
The 8 source models can be interpreted from 3 ResNet families: 1) RN18/34 are ResNets with normal residual blocks, 2) RN50/101/152 are ResNets with "bottleneck" residual blocks, and 3) DN121/169/20 are densely connected ResNets. Another important observation is that when there are more skip connections in a network within the same ResNet family (e.g., RN34 > RN18, RN152 > RN101 > RN50, and DN201 > DN169 > DN121), or from ResNets to DenseNets (e.g. DN121/169/201 > RN 1834 and DN121/169/201 > RN50/101/152), the crafted adversarial exam- ples become more transferable, especially when the skip connections are manipulated by our SGM. This raises questions about the design principle behind many state-of oftheafft DNNs: "going deeper" with techniques like skip connection and x convolution.
4 COMPARISON TO EXISTING TRANSFER ATTACKS
In this section, we compare the transferability of adversarial examples crafted by our proposed SGM and existing methods on ImageNet against both unsecured and secured target models.
Baselines. We compare SGM with FGSM, PGD, and 3 state-of-the-art transfer attacks: (1) Momen- tum Iterative (MI) (Dong et al. 2018), (2) Diverse Input (DI) (Xie et al., 2019), and (3) Transition Invariant (TI) (Dong et al. 2019). 9). Note that the TI attack was originally proposed to attack secured models, although here we melude TI attack both unsecured models and secured models. For TI and our SGM, we test both the me-step and the iterative version. however, the other methods DI and MI only have an iterative version. The iteration step set to 10 and 20 for unsecured and secured target models respectively. For all iterative methods PGD, TI and our SGM, the step size is set to 2. For our proposed SGM, the decay parameter is set y 0.2 (0.5) and y 0.5 (0.7)
5

---

Published as a conference paper at ICLR 2020
on ResNet and DenseNet source models in PGD (FGSM) respectively. For simplicity, utilize SGM to indicate FGSM+SGM in one-step attacks, and PGD+SGM in multi-step attacks. Other parameters of existing methods are configured as in their original papers.
Threat Model. We adopt a black-box threat model in which adversarial examples are generated by attacking a source model and then applied to attack the target model. The target model is of a different architecture (indicated by the model name) to the source model, expect when the source and target models are of the same architecture, where we directly use the source model as the tar- get model (equivalent to white-box setting). The attacks are crafted 5000 randomly selected ImageNet validation images that are classified correctly by all source models, and are repeated for 5 times with different random seeds. For all attack methods, we follow the standard setting (Dong
jet al., 2018: Xie et al. 2019) to craft untargeted attacks under maximum Loo perturbation e 16 with respect to pixel values in [0, 255].
Target Models. We consider two types of target models: 1) unsecured models that are trained on ImageNet training set using traditional training; and 2) secured models trained using adversarial training. For unsecured target model, choose 7 tfte-oofthheart DNNs: VGG19 (with batch normalization) (Simonyan & Zisserman 2015). ResNet- 152 (RN1 (52) (He et al. (0166a), DenseNet- 201 (DN1552, 154 layer Squeeze-and-Fxcitaae
network (SE154) (Hu et al. 2018), Inception V3
(3n) (Szegedy et al. 2016). Inception V4 (IncV3) (Szegedy et al. (017)) and Inception- ResNeet V2 (incReV2) (Szegedy et al. 2017). For secured target models,
consider 3 robustly trained DNNs using ensemble adversarial training (Tramèr et al.) 20188; ; V3 (ensemble of Inc networks), V3e (ensemble of 4 V3 networks) and IncRes V2ena3 (ensemble of 3 IncResV2 networks).
Source Models. We choose 8 different source models from the ResNet family: ResNet(RN)- 18/34/50/101/152 and (DnneN(((N)122111699200
Whenever the input size of the source model does not match the target model, we resize the crafted adversarial images to the input size of the tar- get model. For VGG19, ResNet and DenseNet models, images are cropped and resized to 224 x 224, while for Inception/Inception ResNNet models, images are cropped and resized to 299 x 299.
4.1 TRANSFERABILITY AGAINST UNSECURED MODELS
We first investigate the transferability of all attack methods against 7 unsecured models, which is to find the best method that can generate the most transferable attacks on one source model against all target models.
One-step Transferability. The one-step transferability is measured by the success rate of one-step attacks, as reported in Table [ Here, we only show the results on two source models: 1) RN152 which is the best ResNet source model with the highest success rate on average against all target models, and 2) DN201 which is the best DenseNet source model. Also note that, when the source and target models are the same, the result represents the white-box success rate. Overall, adversarial examples crafted on DN201 have significantly better transferability than those crafted on RN152, especially for our SGM method. This is because there are ~ 30x more skip connections that can be manipulated by our SGM in DN201 compared to RN152. In comparison and TI, transferabilityy is improved considerably by SGM in almost all test scenarios, except when transferring from RN152 to VGG V/GG1///////I//VV V4 where SGM is outperformed by TI. This implies that, when transfereing across different architectures (eg. ResNet + VGG/Inception) translation adaptation may help increase the transferability of one-step perturbations. However, this advantage of TI disappears when there are more skip connections, as is the case for the DN201 source model.
to both FGSM
Table 2: One-step transferability: the success rates (sdst over 5 random runs) of black-box attacks crafted by different methods on 2 source models against unsecured target models. The best results are in bold.
Source
Attack VGG19
RN152
DN201
SE154
IncV3
Inc
IncResV2
FGSM
41.96.10.52
71.53.0.34
37.4910.10
30.0011.556
25.6611.007
21.55-10.16
19.90.10.49
RN152
TI
49.61+0.11
49.33+0.35
36.87+0.42
29.95+0.32
33,5
29.05-00.4
20.62+0.09
SGM
47.54+0.14 7..90-0.60
43.734..21
31.16+0..45
29.41
25.11+
22.63++.115
FGSM
49.8710.17
38.8910.29
81.5110.33
34.9410..33
31.2110.47
27.0810.23
23.8710.45
DN201
TT
54.37+-0.58
33.49+0.18
57.7140.05
34.46+0.47
34.45+0.25
30.17+0.23
20.36+0.33
SGM
56.97+-0.25
4..5
0..73
42.31
07.91
32.8
29.64+0.25

---

Published as a conference paper at ICLR 2020
Teiget VGC19
Target model: SE 154
Target model Inck3
50
PCD
SCM (o
yours
(ours
HOTTEE Sourcw Mocul
HATTS BOarcw Model
Sourcs Mooo
Figure 2: The attack success rates of black-box attacks crafted by different attack methods on 8 source models against unsecured target models: VGG19 (left), SE154 (middle) and V3 (right).
Multi-step transferability: the success rates (td over 5 random runs) of black-box attacks crafted by different methods on 2 source models against 7 unsecured target models. The best results are in bold.
Table 3:
Source
Attack
VGG19
RN152
DNN20
SE154
IncV3
Inc
IncRes
PGD
45.03::0.21
99.9110.04
51.49:0.51
29.35=#0.49
26.56::0.75
21.03:2.19
19.10=0..7
TI
5..59+40.33 99.73+0.11
63.77=-0.93
41.89=
37.97+00.39
1.125+1113
28.90
RN152
MI
65.42:10.60
99.77:10.02
75.79:10.99
53.07=10.44
50.22:10.07
43.32:10.27
41.71
DI
74.01 99,90.10.22
77.81
57.49
53.95110.68
47.1611.522
43.47110.0
SGM
79.90=:0.69 99,87#0.03
81.5610.35
61.83+0.17
5..22++0.51
48.57:0.09
05.44
PGD
57.61 59.8410.776
99.8910.02
39.78.0..69
36.0110.559
31.7610.27
25.9210.13
54.90.10.75 50.6311.02
99.6410.07
40,4010.31
39.1310.52
37.0311.00
28.8310.449
DN201
MI
75,0910.80 76.3910.61
99.8400.05
64.38.0.69
59.6211036
54.8510.566
50.05
DI
78.11 18.184091
99.81
61.75+0888
60.0440.81
56.15+0.36
49.00++0.3
SGM
82.66+00229
86.65 99.67+0.08
72.03+0.53
65.48+0.23
58.77+ 54.97++0.25
Multi-step Transferability. First we provide a detailed study about the transferability of all attack methods from the 8 source models to the 3 representative unsecured target models. We then com- pare different attack methods best ResNet source model and the best DenseNet source model. The multi-step (e.g.. 10 step) trans- ferability from all source models to three representative target models (VGG19, SE154 and V3)) is illustrated in Figure 2 In all transfer scenarios, our proposed SGM outperforms existing methods consistently on almost all source models except RN18. Adversarial attacks crafted by SGM become more transfcrable when there are more skip connections in the source model (e.g., from RNI8 to DN201), An interesting observation is that, when the target model is shallow such as VGGI9 (left figure in Figure P shallow source models transfer better, however, when the target model is deep such as SE154 and V3 (middle and right figures in Figure 2 deeper source models tend to have better transferability. We suspect this is due to the architectural similarities shared by the target and source models. Note that against the VGG19 target model, the success rate of baseline methods all drop significantly when the ResNet source models become more complex (from RN18 to RN152). The small variations at RN50 and DN121 source models may be caused by the architectural differ- ence between RN18/34 1/33 which consist of normal residual blocks, RN50/101/152 which consist of "bottleneck" " residual blocks and DN121/169/201 which has dense skip connections.
best source models against all unsecured target models: the
Results for the best source models RN152 and DN201 against unsecured target models are reported in Table 3, The proposed SGM attack outperforms existing methods by a large margin consistently against different target models. Particularly, for transfer DN201 - SE154 (a recent state-of-the-ane DNN with only 2.251% 1% top-5 error on ImageNet), SGM achieves success rate of 72.03%, which is > 7% and > 10% higher than MI and DI respectively.
Combining with Existing Methods. We further demonstrate that the adversarial transferability of skip connections can be exploited in combination with existing techniques. The experiments are conducted on DN201 (the best source model in the above multi-step experiments), and TI attack is excluded as it was originally proposed against secured models and demonstrates limited improve- ment over PGD against unsecured models. The results are reported in Table E The transferability of MI and DI is improved remarkably of 11.98% 21. 198%% by SGM. When combined with both MI
Foor simplicity, we omit the standard deviations here they are very small, which hardly effect the results.
7

---

Published as a conference paper at ICLR 2020
Table Combined with existing methods: the success rates (%) of attacks crafted on source model DN201 against 7 unsecured target models. The best results are in bold and + indicates improvement.
Attack Target
VGG19
RN152
DN201
SE154
Inc
IncV4
IncRes
MI
75.09
76.39
99.84
64.38
59.62
54.85
50.05
MI+SGM
+12.01
+13.24
99.52
+17.16
+21.88
+15.57
+18.35
DI
78.11
78.18
99.81
61.75
60.04
56.15
49.00
DI+SGM
+12.28
+13.76
99.52
+20.92
+17.66
+15.78
+20.20
MI+DI
87.16
87.28
99.76
79.80
76.68
75.20
71.05
MI+DI+SGM
93.00
93.92
99,42
89.86
85.72
81.23
80.50
Table 5: Transferability against secured models: the success rates (%istd over 5 random runs) of multi-step attacks crafted on RN152 and DN201 source models against 3 secured models. The best results are in bold.
Source
Attack
IncV3
Inc
IncRes
PGD
12.77
10.72::1.37
6.97
TI
45.36=:0.97
45.81=00.33
08.19:00.8
MI
14.20=
22.04=00.98
16.50
RN152
DI
18,48=11221
24.19=t1.22
1..7
SGM
31.57
27.77-10.47
20.02-10.66
TI+SGM
52.62.10400
52.80
43.96.10.22
PGD
18.16:00.56
15.30=10.62
10.40:10.49
42.76=:0.11
42.01=10.79
34.28
MI
31.79=#0.83
28.21
20,60
DN201
DI
34.84
29.23-10.83
20.64
SGM
41.45
37.25
29.01
TI+SGM
47.38
39.32-10.80
and DI, SGM improves the state-of-the-arr (MI+DI) transferability by huge margin consistently against all target models. In particular, SGM pushes the new state-of-the-art to at least 80.52% which previously was only 71% This illustrates that skip connections can be easily manipulated to craft highly transferable attacks against many state-of-the-art DNN models.
4.2 TRANSFERABILITY AGAINST ROBUSTLY TRAINED MODELS
The success rates our SGM and other bascline methods against the secured target models are reported in Table 5 Overall, with translation adaptation specifically designed for evading adversar- ially trained models, TI achieves the best standalone transfcrability, while SGM is the second best with higher success rates than either PGD, MI or DI. When combined with TI, SGM also improves the TI attack by a considerable margin across all transfer scenarios. This indicates that, although manipulating the skip connections alone may not sufficient to attack secured models, it still can make existing attacks more powerful. One interesting observation is that attacks crafted here on RN152 are more transferable than those crafted on DN201, which is quite the opposite to attacking unsecured models.
4.3 A CLOSER LOOK AT SGM
In this part, we conduct more experiments investigate the gradient decay factor of our proposed SGM, and explore the potential use of SGM for ensemble-base attacks and white-box attacks.
Effect of Residual Gradient Decay 7/. We test the transferability of our proposed SGM with vary- ing decay parameter 7 E [0.1, 1.0]. where 7 1.0 means no decay on the residual gradients. The attacks are crafted by 10-step SGM on 5000 random ImageNet validation images. The results against 3 target models (9GG19, SE154 and I3c)3) are illustrated in Figure As can be observed, the trends are very consistent against different target models. On DenseNet source models, decreas- ing decay parameter (increasing decay strength) tends to improve transferability until it exceeds certain threshold, e.g., 7 0.5. This is because the decay encourages the attack to focus on more transferable low-level information, however, it becomes less sufficient if all high-level class-nelevant information is ignored. On ResNet source models, decreasing decay parameter can constantly im- prove transferability for 7 2 0.2. Compared to DenseNet source models, ResNets require more
8

---

Published as a conference paper at ICLR 2020
Taiget
Targel model: SE154
Targer model IncV/S
Atboc 0.00095 136
DMI60
RM15
RH 01
DN-00
RM15
RM 01
EH 50 DMN0
mag-
15-12
DNE01
RMTIO
RAGH
RNTIO
RM00
16131
1940
1090
Paraeeer for SGM
Parameter for SIGM
Parameter for SIM
Figure 3: Parameter tuning: the success rates of black-box attacks crafted by 10-step SGM with varying decay parameter 7 E [0.1, 1.0]. The solid and dash curves represent results on ResNet and DenseNet source models respectively.
Table 6: Multi-step transferability of ensemble-based attack: the success rates (td over ran- dom runs) of multi-step attacks crafted by different methods on an ensemble of 3 source models (e.g. RN34, RN152 and DN201) against 7 unsecured target models. The best results are in bold.
Source
Attack
VGG19
RN152
DN201
SE154
IncV3
Inc
IncRes
PGD
86.69-00.20
99.99-0.00 0.01
99.99-10.02
69.65-00.7
65.95-+0.35
59.30-H0.32
53.91+0.400
RN34
TI
84.35
99.59=0.11
99.77+10.06
01.67++0.19
67.22:+0.82
66.02:+066
0.88
MI
92.86:10.19 99.91=0.04
99.91100066
86.11+0.338
83.25:10.35
1..2
76.53:10666
RN152
DI
96.34=:..23 99.84=0.20
99.84=0.20
89.72:1..2
87.53=0.29
8..04
81.11:044
+
SGM
97.36:0.17
99.87=0.07
99.86=t0.09
90.40:0.26
87.86.0.51
82.97:0.71
80.93+0.55
DN201
DI+SGM
98.65100.88
99.84=0.04 99.86+0.04 94.36+0.19 93.08=0.41
89.56+0.07 88.27=0043
Table 7: Transferability of ensemble-based attack against secured models: the success rates s%d over 5 random runs) of black-box attacks crafted on an ensemble of 3 source models (e.g. RN34, RN152 and DN20I). The best results are in bold.
Source
Attack
V3c33ensi
IncVensd
IncResensi
PGD
37.6
32.69-10.62
23.49110.55
RN34
TI
7..04=0.50
75.94=10.6
66.24+10.45
+
MI
54.68=0.27
50.24=0048
39.37=00.3
RN152
DI
65.29=#031
57.48++0.45
46.41+0.42
SGM
66.08=0042
62.22-+0.73
51.16+0.12
DN201
T14SGM
87.65
85.11
77.75.1..1
decay on the residual gradients. Recalling that skip connections reveal more transferable informa- tion of the source model, ResNets require more penalty on the residual gradients to increase the importance of skip gradients that reveal more transferable information of the source model.
As for the selection of y under a scenario without knowing the target model, from Figure and Appendix C we can see that the influence of y is more related to the source model rather than the target model, that is, given a source model, the best 7 against different target models are generally the same. This makes the selection of quite straightforward: choosing the best ry on the source model(s). For instance, in Figure 3 suppose the unknown target model is SE154 (middle figure), the adversary could tune y on source model DN201 to attack VGG19 (left figure) and find the best y 0.5. The attacks crafted on DN201 with 7 0.5 indeed achieved the best success rate against the SE154 target model (and other target models).
Ensemble-based Attacks. It has been shown that attacking multiple source models simultaneously can improve the transferability of the crafted adversarial examples, and is commonly adopted in practice. We follow the ensemble-based strategy (Liu et al., (017)) and craft attacks on an ensemble of RN34, RN152 and DN201. According to the discussion above, we select the best y individually for each source model: choose for source RN34 and RN152 against target DN201, and for source DN201 against target RN152. The success rates (transferability) against unsecured models and 3 secured models are reported in Table and Table respectively Similar to the results of single source model attacks, against unsecured target models, SGM has a similar standalone performance with DI, better than the others (except the two "white-box" scenarios against RN152 and DN201). When combined with other methods, egg. DI, it improves the success rates again by large margin.

---

Published as a conference paper at ICLR 2020
107
92
Adoo H0110
19:00
15V137
D7/193
2011
DW1221
D4110
Meximum pertuibation
Maximum parturbation
(a) White-box (c
8)
(b) Varying e on ResNets
(c) Varying c on DenseNets
Figure 4: White-box success rate for FGSM versus SGM. In (b) and (c), each color corresponds to
one model, with FGSM is represented by solid curve and SGM represented by dashed curve.
Against secured models, SGM achieves the second best standalone transferabilityy with TI is still the best. When combined with TI, SGM improves the success rate by ~ 10% consistently against all secured target models. In particular, against Inc TI+SGM achieves higher success rate (87.65%) than reported in (Dong et al. 2019) (84.8%), even if only 3 source models are used here and the source models (e.g. RN34, RN152 and DN201) are all of different architecture to Inc target model ((Dong et al., 2019) uses 6 source models including even the Inc model), From all aspects analyzed above, the existence of skip connections makes transferable attacks much easier to craft in practice.
Improving Weak White-box Attacks. In addition to the black-box transferability, we next show that SGM can also improve the weak (one-step) white-box attack FGSM. Note that the one-step version of SGM is equivalent to FGSM plus residual gradient decay. Our experiments conducted on the source models, and the white-box success rates under maximum L perturbation c = 8 (a typical white-box setting) are shown in Figure 4a As can be observed, using SGM can help improve the adversarial strength (i.e., higher success rate). We then vary the maximum perturbation E [1, 64], and show the results on ResNet and DenseNet models separately in Figure 4b and Figure 4c| Compared to FGSM, SGM can always give better adversarial strength, except when c is extremely small (c S 2). When the perturbation space becomes infinitely small, the loss landscape within the space becomes flat and the gradient points the optimal perturbation direction. However, when the perturbation space expands, one-step gradient becomes less accurate due to changes in the loss landscape (success rate decreases as e increases from 4 to 16), and in this case, the skip gradient which contains more low-level information is more reliable than the residual gradient (the improvement is more significant for e [4, (6) Another interesting observation is that adversarial strength decreases when the model becomes more complex from RN18 to RN152, or DN121 1 to DN201. This is likely because the loss landscape of complex models is steeper than shallow models, making one-step gradient less reliable.
5 CONCLUSION
In this paper, we have identified surprising property of the skip connections used by many state-of- the-art ResNet-like neural networks, that is, they can be easily used to generate highly transferable adversarial examples. To demonstrate this architectural "weakness", we proposed the Skip Gradient Method (SGM) craft adversarial examples using more gradients from the skip connections rather than the residual ones, via a decay factor on gradients. We conducted a series of transfer attack experiments with 8 source models and 10 target models including 7 unsecured and secured models, and showed that attacks crafted by SGM have significantly better transferability than those crafted by existing methods. When combined with existing techniques, SGM can also boost state-of- tht-hharr transferability by a huge margin. We believe the high adversarial transfcrability of skip connections is due to the fact that they expose extra low-level information which is more transferable across different DNNs. Our findings in this paper not only remind researchers in adversarial research to pay attention to the architectural vulnerability of DNNs, but also raise new challenges for secure DNN architecture design.
ACKNOWLEDGEMEN
Shu-Tao Xia is supported in part by National Key Research and Development Program of China un- der Grant 2018 YOB 1800204. National Natural Science Foundation of China under Grant 61771273, R&D Program of Shenzhen under Grant JCYJ201 0050815220440 and research fund of PCL Fu- ture Regional Network Facilities for Large-scale Experiments and Applications (PCI.201KKP00),
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Yang Bai, Yan Feng. Yisen Wang, Tao Dai, Shu-Tao Xia, and Yong Jiang. Hilbert-basee generative defense for adversarial examples. In ICCV, 2019.
Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. Practical black-box attacks on deep neural networks using efficient query mechanisms. In ECCV, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In S&P, 2017.
Pun-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order opti- mization based black-box attacks to deep neural networks without training substitute models. In AlSec, 2017.
Pin-Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich. Ead: elastic-net attacks to deep neural networks via adversarial examples. In AAAI, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Gavin Weiguang Ding, Luyu Wang, and Xiaomeng Jin. AdverTorch v0.1: adversarial robustness toolbox based on pytorch. arXiv preprint arXiv:1902.07623, 2019.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. CVPR, 2018
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariint attacks. In CVPR, 2019.
Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Song. Robust physical-world attacks on deep learning models. In CVPR, 2018.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 201
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016b.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatie networks. In CVPR, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. In ICCV, 2019,
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. ICML, 2018
Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In CVPR, 2019.
Linxi Jiang, Xingjun Ma, Shaoxiang Chen, James Bailey, and Yu-Gang Jiang. Black-box adversarial attacks on video recognition models. In ACM MM, 2019.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In ICLR, 2016.
Yingwei Li, Song Bai, Yuyin Zhou, Cihang Xie, Zhishuai Zhang, and Alan Yuille. Learning trans- ferable adversarial examples via ghost networks. arXiv preprint arXiv: 18/2,034/3 2018.
11

---

Published as a conference paper at ICLR 2020
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial exam- ples and black-box attacks. In ICLR, 2017.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey, Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018.
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu. Understand- ing adversarial attacks on deep learning based medical image analysis systems. arXiv preprint arXiv: /90iv 1wt.:/9977.45. 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Apostolos Modas, Seyed-Mohsen make a big difference. In CVPR, 2019.
and Pascal Frossard. Sparsefool: few pixels
Nicolas Papemot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In EuroS&P, 2016.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter, Accessorize to crime: Real and stealthy attacks on state-of-the-art face recognition. In CCS. 2016
Carl-Johann Simon-Gabriel, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, and David Lopez- Paz. First-order adversarial vulnerability of neural networks and input dimension. In ICML, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. In IEEE Transactions Evolutionary Computation. IEEE, 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, 2017.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In ICLR, 2018
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. In NeurIPS, 2016.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML, 2019.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. ICLR, 2020.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang. Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In CVPR, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint
12

---

Published as a conference paper at ICLR 2020
A VISUALIZATION OF ADVERSARIAL EXAMPLES CRAFTED BY SGM
In this section, visualize 6 clean images and their corresponding adversarial examples crafted using our SGM on either a ResNet- 152 or a DenseNet201 Figure [5] These visualization results show that the generated adversarial perturbations are human imperceptible.
Figure 5: Visualization of clean images and their corresponding adversarial examples. The clean images are shown in the top row, adversarial images crafted on ResNet-152 are shown in the middle row, while those crafted on DenseNet-201 are shown in the bottom row. All adversarial images crafted using our proposed SGM (10-step) under maximum perturbation c = 16.
B COMPARISON WITH PREVIOUSLY PUBLISHED RESULTS
In this section, compare the experimental settings in previous and our works, and discuss some small discrepancies of the baseline performance reported in ours and previous works.
Table 8 and 9 summarizes these differences for single-source and ensemble-based attack respec- tively. Out of all these works (Dong et al., 2018 2019 Xie et al. 2019), results reported (Xie) er al. 2019) are more complete. Our reported success rate of baseline attacks (e.g. MI and DJ) matches that reported in (Xie et al. 2019)), sometimes even higher. The slight discrepancy caused by the difference in experimental settings. Table I0 summarizes the different source models used by baseline attacks, and Table [ summarizes the difference dataset, number of test images, input image size, maximum Logo perturbation e, number of attack steps N and attack step size a. Com- pared to 299 x 299 image size, here The use of smaller input size may reduce the effectiveness of existing attacks (Simon-Gabriie et al, 2019)
standard image size 224 224 ImageNet.
In another work by Liu et al. (011)) ). 81 % success rate was reported for optimization attack crafted on ResNet-155 132 against target VGG16, which is higher than our 65.52% from ResNet-152 to VGG19. This because they did restrict the maximum perturbation c. The root mean square deviation (RMSD) of their attacks is which indicates that many pixels are perturbed more than 16 pixel values. our experiments, the RMSD is 6.29 for PGD, 7.71 for SGM, and 12.55 for MI. This appears to be another reason for the performance discrepancy. Note that the advantage of bounded small perturbation is increasing imperceptibility human observers (see Figure 5).
13

---

Published as a conference paper at ICLR 2020
Table 8: Previously reported attack success rates (%) of baseline single-source attacks against target models. "_" means no results were reported.
Reference
Attack
IncV3
Inc
IncRes
Inc IncV3ens3
V3eaa
IncResena3
FGSM
35.0
28.2
27.5
14.6
13.2
7.5
Dong et al. 2018
BIM
26.7
22.7
21.2
9.3
8.9
6.2
MI
53.6
48.9
44.7
22.1
21.7
12.9
FGSM
20.2
17.7
9.9
Dong et al.
2019
MI
25.1
23.7
13.3
DI
40.5
36.0
FGSM
34.4
28.50
27.1
12.4
11.0
6.0
PGD
20.8
17.2
14.9
5.4
4.6
2.8
Xie al. 2019
MI
50.1
44.1
42.2
18.2
15.2
9.0
DI
53.8
49.0
44.8
13.0
11.1
FGSM
26:56
21.03
19.10
PGD
26.56
21.03
19.10
12.47
10.72
6.97
Ours
MI
50.22
43.32
41.71
24.20
22.04
16.10
DI
53.95
47.16
43.47
34.84
29.23
21.64
Table 9: Previously reported attack success rates (%) of ensemble-based baseline attacks against target models. "." means no results were reported.
Reference
Attack
IncV3
Inc
IncRes
Inc V3
I3enaa
IncResena3
FGSM
45.7
39.9
38.8
15.4
15.0
6.4
Dong
BIM
72.1
61.0
18.7
MI
87.9
81.2
76.5
37.6
40.3
23.3
FGSM
27.5
23.7
13.4
MI
50.5
48.3
32.8
Dong et al. 2019
DI
45.9
T1+D1
84.8
82.7
78.0
PGD
43.7
36.4
33.3
12.9
15.1
8.8
Xie et al. 2019
MI
69.9
67.9
64.1
36.3
35.0
30.4
DI
71.4
65.9
22.8
15.8
DI+MI
80.7
80.6
80.7
44.6
44.5
39.4
PGD
65.95
59.30
53.91
37.63
32.69
23.49
MI
83.25
79.25
76.53
54.68
50.24
39.27
Ours
DI
87.53
85.04
81.11
65.29
57.48
46.41
DI+SGM
93.08
89.56
88,27
80.14
76.52
66.40
TI+SGM
87.65
85.11
77.75
Table 10: Source models used by existing single-source and saeemble-based black-box attacks. "Hold-out" refers the hold-out target model from the group, with all remaining models are used as source models. Group consists of ResNet-v2-152. IncV3, IncV4 and IncRes, while group consists of ResNet-v2-152. Inc IncV4, IncRes, Inc Inc een44 and IncRes
Reference
Single-source attack
Ensemble-base attack hold-out from group or group 2
Dong et al. 2018)
ResNNet-v21522
Dong 2019)
ResNet-vv21522
hold-out from group
Xie et al. (0019)
ResNct-v22152
hold-out from group
Ours
RN152
RN34 RN152 DN201
For proper implementation, we use open-source codes and pretrained models for our experiments, e.g., AdverTordh (Ding et al.) 2019) for FGSM, PGD and MI, and source/target models from two GitHub repositories for all models. We reproduced DI and TI in PyTorch.
https:: htteete///taaoeeeeeee///ttaaaatt
imagenet_models
14

---

Published as a conference paper at ICLR 2020
Table 11: Difference in experimental settings of our work compared to previous works. "NeurIPS 2017" indicates the dataset used for NeurlPS 2017 adversarial competition. e: maximum per-pixel perturbation; N: number of attack steps; attack step size.
Reference
Dataset
Number
Input size
N
Dong 2018
ImageNet
1000
299
299
16
1.6
Dong 2019)
NeurlPS 2017
1000
299
299
16
10
1.6
Xic et al. 2019)
ImageNet
5000
299 x 299
15
19
1.0
Ours
ImageNet
5000
224
224
16
10/20
2.0
C TRANSFERABILITY OF DECAY PARAMETER 7
In this section, we study the "transferability" of decay parameter across different target models. RN152 and DN201 are used as the source model, and the target model are varying to observe trends. As indicated in Figure 6 all black-box target models share the same best selection of 7 which makes 7 selection quite simple. Even if the true target model is unknown, the adversary can tune the gamma for a resnet-like neural network through another model and obtain the best selection as well.
Source model: RN152
Source model: DN201
100
100
Atrack success rate
60
50
50
40
IncV4
RN152
IncV3
IncRes
RN152
-
IncV3
IncRee
10
0.8
0.1
Parameter y
y
Figure 6: "Transferability" of decay parameter: the success rates of black-box attacks crafted by 10-step SGM with varying decay parameter E (0.1, The curves represent results against different target models respectively.
15


Reviews:
[{"review_id": "SkgLpWdiYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is about adversarial attacks and highlights a security weakness of skip connections in ResNet-like CNNs, namely: skip connections make it easier to obtain adversarial examples. This observation leads to new approach to adversarial attacks, named Skip Gradient Method (SGM), which weights the residual gradient w.r.t  the skip connection gradient. The approach is validated on a variety of image classification attack scenarios (e. g. white-box and transfer attacks) using two families of source models (ResNet and DenseNet). The results show the superiority of SGM when comparing to other adversarial attack scenarios.\n\nStrengths:\n- Simple approach that seems to be giving good results\n- Large number of adversarial attack scenarios tested\n- Good related work review\n\nWeaknesses:\n- Results are reported without variance information\n- There are some details missing on how the decay factor is selected \n- Results are reported only on one dataset (ImageNet)\n\nThe paper is well written, the authors have identified a \"problem\" of ResNet-like models and proposed an approach that can exploit the problem in adversarial attacks scenarios (SGM). To the best of my knowledge, this is the first time someone has identified the skip connections security problem in ResNets. The SGM is compared against many existing adversarial attacks methods achieving making the evaluation section quite detailed. Thus, I'd lean towards paper acceptance.\n\nHowever, I have some questions with respect to the evaluation section:\n1. Although, the paper includes ablation analysis for different values of the decay factor, I could not find details on how the decay factor hyperparamenter is selected. Given that this factor is a hyperparamenter, it feels like it should be selected on a validation set and tested on a test set. The paper comments about 5000 random validation set images that are used to compere methods (test set), however, I could not find any mention about validation set used to select decay factor. Could the authors specify how the decay factor is set?\n2. Since all the results are reported with 5000 random imagent images, it would be interesting to see the variance of results if the sampling of images is repeated.\n3. Section 3.3, 2nd paragraph: \"Another important observation is that when there are more skip connections in a network.... the crafted attacks become more transferable....\". Although, this statement seems to be correct when comparing DenseNet to ResNet, it does not necessarily hold when comparing models within the same family (e. g. RN18 to RN 152 for FGSM or RN34 to RN 152 for PGD) suggesting that the best is not only connected to the number of skip connections. Maybe the superiority of DN is rather related to the nature of the DN models and not to the number of skip connections. Could the authors clarify?\n4. Section 4, threat models: \"... the same architectures but trained separately.\" Could the authors clarify what does it mean? Are these models re-trained from scratch changing the random seed of model initialization, order of the dataset, or something else? In general are the authors using pertained models or train all the models \"from scratch\".\n5. Fig 2: Since x-axis values are not continuous - it might be better to use bar plot.\n6. In some tables, the results are reported just for SGM while in other tables SGM is reported in combination with other method. I guess that when the results are reported for SGM they represent SGM from Eq. 10 that represents SGM+PGD. Is this correct? The authors might consider clarifying this in the paper.\n\nSome typos:\nIn the following, we exploits an architectural security weakness about....\n... the the crafted attacks...\n... not only reminds researcher to pay....\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BklvMFCmqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper discovers a very interesting phenomenon that adversarial examples are more transferable when the perturbations are obtained by propagating a higher ratio of gradients via shortcuts in ResNets and DenseNets. \nThe method is simple and comprehensive experiments are conducted to prove its correctness. \nIt is a good empirical paper that can inspire future research on investigating the role of shortcuts when defending adversarial examples.\n\nTo improve the paper:\n1. The \"transferability\" (or robustness) of gamma γ should be studied. In real applications, γ should be selected before transferring to new black-box models. The experiments in this paper are done by reporting the best results under the best γ. This is a hindsight. How can you pick up γ without accessing the transferability results?\n2. More theoretical explanations. It is unclear to me that why it works so well, since the back propagation is no longer coherent with the forward function.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJxG0SRl9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper proposes a modification to standard Projected Gradient Descent to improve transferability of adversarial examples, when the source model is a ResNet-like model containing skip connections. The method, Skip Gradient Method (SGM) modifies the backwards pass to scale down the gradient computed in each residual branch of the model, before these gradients are combined with the gradient from the skip connection. This thus upweights the gradients from the skip connections as opposed to residual modules. The paper demonstrates significant improvements in the single-model black-box transfer setting, against a variety of undefended and defended models.\n\nStrengths:\n- Lots of interesting empirical results here! One result which stands out to me is Table 4, where across a wide range of target models, adding SGM to existing techniques cuts defender accuracy by ~1/2 (e.g. 79.9% to 89.66% for SE154). The results in Table 3, showing that even without additional techniques, SGM results in large improvements and outperforms previous approaches, are also quite nice. \n- It's notable that such a simple approach leads to significant improvements.\n- Results are very clearly presented, and writing is clear throughout. \n\nSuggestions for improvement:\n\nI have 3 major concerns: (1) discrepancies between baselines and previously published results (2) unrealistic threat model (3) framing / conclusions drawn by paper.\n\nI suspect (1) is easily addressed, but was not clear to me from the current paper.\n\n(1) Baselines: \n- For multi-step transfer against undefended models (Table 3), the attack success rates seem low compared to numbers reported in e.g. Liu et al. For example, using ResNet-152 as the source, and VGG-19 as the target, Liu et al reports 19% defender accuracy = 81% attacker success. This is significantly higher than the 65.52% reported for MI (both are non-regularized optimization attacks), also stronger than the 80.68% reported for SGM. In general, considering these are *untargeted* attacks, with eps=16, the transfer rates seem pretty low.\n- For multi-step transfer against defended models (Table 5), the numbers are slightly lower than previously reported. E.g. attacking IncV3_ens3, Dong et al 19 reports 46.9% for MI, but here, 44.28% is reported. (I realize these differences are slight.)\n- In general, it would be nice if the paper were structured so as to make these comparisons easier. For example, the appendix could include tables comparing the baselines reported here, to values previously reported, and explain any discrepancies. Particularly with black-box transfer, where baseline performance is so sensitive to small choices, it's important to ensure baselines are properly implemented, and the current writing of the paper makes it impossible for the reader to assess this unless they are very familiar with the black-box transfer literature.\n\n(2) Unrealistic threat model:\n- For black-box transfer, all the strongest attacks use multiple source models (see e.g. NeurIPS 2017 Adversarial Examples contest, the baselines cited in the paper). While this paper shows significant improvements in the single-source setting, these results are significantly weaker than any multiple-source attack. For instance, Liu et al 17 achieve near 100% untargeted success (and near 100% targeted success) against all undefended models they study (many which overlap), and Dong et al 19 report ~85% accuracy against IncV3_ens3 (compared to <60% here).\n- There's no reason in practice that an adversary would not employ an ensemble-based attack if they wanted to fool an unknown model.\n\n(3) Framing / conclusions:\n- The paper frames the results as a \"security vulnerability of ResNets,\" but the results don't show this. In particular, they show that ResNets make effective *source* models to be used by *attackers*, but they don't imply which models defenders should use in order to be robust to black-box attacks. In this way, the main message of the paper seems misleading to practitioners.\n- The main message of the paper thus seems to be \"on the transferability of adversarial examples generated  with resnets\" as opposed to the \"security of skip connections.\" I would encourage rewriting of the title/intro/conclusion as such.\n\nOverall, there are several very interesting empirical findings in this paper. I view the two main impacts these results could later have would be leading to (1) improved understanding of what causes transfer of adversarial examples and (2) improved understanding of ResNet-like architectures (for example, the results provide some support for the view of ResNets as ensembles of shallow models, cited in the paper). If the paper were written with this view, then concern (2) above becomes unimportant.\n\n\nSuggestions I believe could strengthen the paper, but I do not view as weaknesses, or necessities:\n- Can you decouple the effect of SGM on optimization and transfer? The paper does a bit of this, but restricts the analysis to the single-step case. For instance, if the main effect of SGM is on optimization, this has interesting implications for optimization of ResNets. If the main effect is on transfer, this has interesting implications for what causes transfer.\n\nMinor:\n- I'd suggest moving Table 1 (one-step attacks) to the Appendix. The results are less impressive than multi-step, and the community as a whole favors stronger attacks.\n\nOverall, there are several interesting empirical findings in this paper (modulo concerns about baselines indicated above). I suggest that the paper either consider more realistic threat models to be useful to the adversarial examples field, or focus on the insights revealed by these findings. I hope that the authors can address the concerns outlined here, and I would be happy to adjust my score if so.\n\nNote that my current indicated confidence rating is for the current state of the paper. I would be happy to adjust my evaluation if revisions to the paper can one/several of the concerns indicated above.\n\n\n__________________\n\nEDIT: Rating adjusted from 3: Weak Reject to 8: Accept, after accounting for revisions and author rebuttal. See reply below.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes the observation that, by adjusting the ratio of gradients from skip connections and residual connections in ResNet-family networks in a projected gradient descent attack (that is, upweighting the contribution of the skip connection gradient), one can obtain more transferable adversarial examples. This is evaluated empirically in the single-model black box transfer setting, against a wide range of models, both with and without countermeasures.

Reviewers praised the novelty and simplicity of the method, the breadth of empirical results, and the review of related work. Concerns were raised regarding a lack of variance reporting, strength of the baselines vs. numbers reported in the literature,  and the lack of consideration paid to the threat model under which an adversary employs an ensemble of source models, as well as the framing given by the original title and abstract. All of these appear to have been satisfactorily addressed, in a fine example of what ICLR's review & revision process can yield. It is therefore my pleasure to recommend acceptance.
Decision: Accept (Spotlight)