Learning Transferable Adversarial Perturbations
Authors: Krishna kanth Nakka
Mathieu Salzmann
Venue: NeurIPS 2021 Poster (2021)
Keywords: Adversarial attacks

Abstract: 
While effective, deep neural networks (DNNs) are vulnerable to adversarial attacks. In particular, recent work has shown that such attacks could be generated by another deep network, leading to significant speedups over optimization-based perturbations. However, the ability of such generative methods to generalize to different test-time situations has not been systematically studied. In this paper, we, therefore, investigate the transferability of generated perturbations when the conditions at inference time differ from the training ones in terms of the target architecture, target data, and target task. Specifically, we identify the mid-level features extracted by the intermediate layers of DNNs as common ground across different architectures, datasets, and tasks. This lets us introduce a loss function based on such mid-level features to learn an effective, transferable perturbation generator. Our experiments demonstrate that our approach outperforms the state-of-the-art universal and transferable attack strategies.

Learning Transferable Adversarial Perturbations
Krishna Kanth Nakka', Mathieu Salzmann 1.2
CCVLab, EPFL. Switzerland
"CacarSpace. Switzerland
(krishna mrrish mathieu. salzmann)@epfl. .ch
Abstract
While effective, deep neural networks (DNNs) are vulnerable to adversarial at- tacks. In particular, recent work has shown that such attacks could be generated by another deep network, leading to significant speedups over optimization-bart perturbations. However, the ability of such generative methods to generalize to different test-time situations has not been systematically studied. In this paper, we therefore investigate the transferability of generated perturbations when the conditions inference time differ from the training ones in terms of target archi- tecture, target data, and target task. Specifically, we identify the mid-level features extracted by the intermediate layers of DNNs common ground across different architectures, datasets, and tasks. This lets us introduce a loss function based on such mid-level features to learn effective, transferable perturbation generator. Our experiments demonstrate that our approach outperforms the state-of-the-att universal and transferable attack strategies.
Introduction
In recent years, deep neural networks (DNNs) have achieved great success in wide range of applications [1, 2, 3, 4]. However, DNNs have been demonstrated to be vulnerable to adversarial examples [5] crafted by adding imperceptible perturbations to clean images. In particular, two broad categories of attacks have been studied. The first one consists of iterative algorithms [5, 6, 7J. which optimize the perturbation for each instance, and thus tend be computationally expensive. The second one encompasses generative methods [8, 9, 10], which train a deep network to produce perturbations. As such, attacking the target network only involves a forward pass through the generator, typically resulting in much faster attacks than iterative methods. However, speed is not the only important factor to assess the strength of an attacker; its ability to generalize to different situations is also key to its success.
In this paper, we therefore study the transferability of perturbations obtained with generative methods. Specifically, we investigate the transfer of such perturbations when the conditions at inference time differ from the training ones in terms of target architecture, e-g., the generator was trained to attack VGG-16 16 but the target network is a Res (ii) target data, e.g., the generator was trained using the Paintings dataset but the test data comes from ImageNet; (iii) target task, e.g.. the generator was trained attack image recognition model but faces object detector at test time. To the best of our knowledge, for generative methods, our work constitutes the first attempt at transferability across tasks, and only [11] has studied generalization across architectures and data, by introducing loss function acting on the relative class probabilities of the attacked and unattacked examples.
Here, by contrast, we improve the transferability of a perturbation generator across architectures, data, and tasks by exploiting the mid-level features of DNNss The key motivation behind our approach is our observation that the mid-level features extracted by DNNs with different architectures, different data, for different tasks bear strong similarities. This is illustrated Figure 1, where we visualize the features extracted by different backbones and for different tasks, and thus different datasets, using
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Rosgalks Clasii
Iund
image
Advensrial prediction
Projection
H
1o0
Fasture
Paintings Domaie
prediction
Training Phase
AnugeNer Dumsiia
Testing Phase
VGGIe Lager Filter
DemscNet121; Layer Pilter
Classifier B
Genceator
|+++
pelicau
Cross-architecture
Itel
PASCAL POC
Adverstrial
Dosmaie
Cress-dnmain
Cros-domate Cross archilterture
Generator
Figure 1: Learning transferable perturbations. We observe that mid-level features are common across architectures and tasks, and thus propose to exploit them to train perturbation generator by maximizing the relative distance between normal and perturbed features. We show that such generator is effective even in the presence of different model, dataset, or task at test time.
the method of [12]. Our analysis suggests that adversarial perturbations that significantly affect the mid-level features of one sample in architecture also affect them in different architecture, for a different task and with different data.
We therefore propose to train perturbation generator by maximizing the distance between the normal features sample and their adversarial counterparts extracted in the intermediate layers of pretrained classifier. The resulting perturbations are then transferable across architectures, datasets and tasks because they similarly affect the mid-level features the corresponding filters the target setup. The perturbed mid-leve features are then propagated to the top layers of the network, and, as result, lead to incorrect predictions.
Contributions.
Our contributions can be summarized as follows: 1. We identify the intermediate features CNNs common ground across different architectures, different data distributions and different tasks. 2. We introduce approach that exploits such features to learn an effective, transferable perturbation generator. 3. We systematically investigate the effect of target architecture and target data distribution on the transferability of adversarial attacks, Our experiments demonstrate that our approach yields higher fooling rates than the kate-of-the--e universal [8] and transferable [11] attacks. Our code is available at https: : //gittu...... www.siinaaattaakka...k Transfffarrraeeeeeeeeeaae erable_. Perturbations.
Related Work
Below, review the recent works adversarial attacks, with focus on generator-based approaches.
Adversarial Attacks. Adversarial attacks were first investigated in [5] to identify the vulnerability of modern deep networks to imperceptible perturbations in the context of image classification. Since then, several attack strategies have been studied, including single-step fast gradient descent [6, 13], and computationally more expensive optimization-bast attacks, such CW [14], JSMA [15], and others [16, 17, 7].
While the above methods are smage-dependent, the existence of Universal Adversarial Perturbation (UAP) was first shown [18], considering the task of learning a single perturbation that can fool classifier independently of the input image. Such UAP was iteratively updated based its effectiveness to move the individual data samples across the decision boundary. Other UAPs based on the computation of the singular vectors of the Jacobian matrices of feature maps have been studied

---

in [19]. In parallel to UAPs, several works have shown that iterative adversarial attack strategies could be transferred across architectures [20, 21, 22]. In particular, the recent work of [23] proposed to improve targeted attack transferability across architectures by training auxiliary classifiers at each intermediate layer to attack a CNN's feature maps. The above-mentioned methods, however, either are computationally expensive at inference time [16, 7, 15, 14, 21, 23), or suffer from low transferabilitty rates [18, 20, 22j. Furthermore, most of the attacks aiming for transferability [18, 19, 23] strongly depend on the availability of data from the target domain. By contrast, we introduce an efficient generative method to produce adversarial perturbations that generalize not only across architectures, but also across training data and across tasks.
renerator-based Attacks. Generative adversarial perturbations (GAP) were first introduced in [10]. In particular, [10] showed that a generator network can be used to craft a UAP that transform an input image to an image-dependent perturbation. Similarly, [24] introduced the advGAN generative framework to learn to produce adversarial perturbations, and further proposed to make use of distillation to perform black-box attacks. Based on the observation that the effectiveness of these two methods strongly depends on the availability of data from the target domain during training, [11] introduced the relativistic cross-entropy loss, which was shown to better generalize across datasets. Furthermore. [8] proposed to generate UAPs by transforming the generator's output using regional norm layer that enforces perturbation homogeneity. While effective, the senoeemet works explicitly rely on the classification boundary of the attacked model, which we will show to tend to make them overfit to the source data. By contrast, we identify the mid-level features as a more robust signal shared across not only different architectures and datasets, but even across different tasks. Our experiments will showcase the superiority of exploiting this information over the losses used in previous approaches to train transferrable perturbation generator.
3 Methodolgy
Let x; E #xxxx33 be a color image of size H W, and y; be the associated ground-truth label. Furthermore, let f denote the task-related convolutional neural network that takes x as input and extracts, at layer / {4 a feature map fx,)) E R Nex which we assume to be reshaped in matrix form, such that N₂- = H₂ x We, with He and We the spatial dimensions of the feature map, and D3 its number of channels.
Our goal is to train a generator 9 that produces a perturbation 6, E R Hx3x3 which, when added to the clean image x,, results in predicting different label from y.. To this end, we feed the input image x, to the generator to synthesize an unbounded adversarial image G (xx)), which is then clipped be within an e bound of x; under the lo norm. Let X; be the adversarial image obtained after such a clipping of (X;)). In contrast to [10, 111, which rely on the final classification boundary of the relkere network f to train the generator with a cross-entropy-baat based loss, we exploit the mid-level features of f. Specifically, we maximize the L2 distance between the normal feature map fe(x,) and adversarial feature map fix, at layer using a feature separation loss term defined as
[xe x,) = (1)e(x))) (e(
(1)
where || || denotes the Frobenius norm.
The overall training scheme of our perturbation generator is provided in Algorithm 1. In practice, we observe the feature separation loss of Eq. 1 to encourage the generator to learn low-level, quasi- imperceptible structured patterns that affect that activations a few filters while either suppressing the other ones or leaving them unaffected. More importantly, these perturbations are highly transferable.
Reason for transferability. Given the simplicity of our formulation, a natural question that arises is what affects the transfcrability of perturbations among CNNs. To understand this, we analyze the internal workings of CNN, whose filters in each layer encode different levels of information. In particular, in Figure 2, we visualize the filters of various CNNs, sampled at different layers, using the technique of [12], which maximizes the mean activation of the each filter. This allows us to discover common characteristics across the different architectures. On one hand, the bottom filters, close to the input image, extract color and edge information, as shown in the left block of Figure 2. On the other, the top-level filters, close to the output layer, shown in the right block, are more focused on the object representation, and thus more task specific. By contrast, the mid-level filters learn more nuanced features, such textures, and therefore tend to display similar patterns across architectures,
3

---

SqueeseNet
VGG16
ResNert
Des
RoNNett2
BoNNet DraerNed121
VGG16
VIGG66
Res
Luw Litte
Levil tevr
Initial layers
Intermediate layers
Final layer
Figure 2: Filter visualization. The filters the mid-level layers of different CNNs follow similar activation patterns, also common across tasks, whereas those near the classification layer focus on object-level features and are thus more task-specific.
Layer
Layer
Layer 13
(a) Layer 0
(b) Layer
Layer
(d) Layer 13
Figure 3: Feature visualization. We visualize the features extracted 4 different layers of DenseNet121 for different datasets using I-SNE.
datasets tasks. As consequence, and as indicated by our experiments, attacking the initial layers requires large perturbation strengths. Furthermore, while attacking the top-level features works well in the white box scenario, it does not transfer well to different data or architectures, because the features are already too architecture- and task-specific. Attacking mid-level features thus comes as natural choice, which we will show to yield highly transferable perturbations.
Let us now focus understanding the reason for the transferability when the target data is unavail- able to the attacker. In Figure 3, we plot t-SNE visualizations of different datasets at 4 layers of DenseNet121. The proximity of the features extracted from the Comics or Paintings datasets, con- taining images from few classes particularly focused on humans, to those extracted from ImageNet suggests these datasets as good candidates train generator that will fool ImageNet-trainedd network (as shown in Tables and 2). By contrast, the ChestX dataset presents larger domain gap. Nevertheless, we will show that, albeit for a drop fooling rate, our use of mid-level features still make ChestX-traiee Figure 4, we plot the top activations ImageNet, Paintings and ChestX layer 8 of 1enseNet121. This shows that the peaks of the ImageNet dataset (orange) are more strongly correlated to those of the Paintings dataset (left) than to those of ChestX (right). This is confirmed by our experiments, where a Paintings-trained generator is more effective than ChestX-trained one. In short, while mid-level features lead to generator transferability, the resulting effectiveness remains affected by the similarity of the data.
generator reasonably effective on ImageNet-based classifiers. Furthermore, in
Algorithm Training transferable adversarial perturbation generator
Input: x, clean images; f: pretrained classifer; e: perturbation &oo bound
Initialize generator G; Load classifier f and freeze its parameters;
repeat
Get a clean image x; and feed x; to ff to obtain fe(x;);
Generate unbounded adversarial image G(x:) and clip it within an bound of x; to get X;; Pass the adversarial image to f to obtain fe(X;);
Compute the feature separation loss Leeati Compute the gradient of L w.r.t. the weights of G and update these weights using Adam. until convergence
return trained Generator g"

---

3.0
30
300
30
345
(a) Paintings
TmtgeNet
(b) Chestx ImageeNet
Figure 4: Top activated channels. Top-80 activated channels for ImageNet (orange) in comparison to Paintings in (a), and ChestX in (b), for layer 8 of DenseNet121 using 500 images from each dataset. The blue bars are more correlated to the orange ones in the Paintings case, thereby achieving higher transferability rates than using ChestX. Best viewed in color and zoomed in.
Traininx (dan)
Discriminater
VGG16
RoNNe1122
Inception-13
DenseNet121
SqueeaeNetL.1
GAP CDA Olt) Ours
VOG--6
9R.- /988" 3IR3"
S15/536/ 6K4
4646
58928655 K4.7
6787 70.6 M6.5
55.5 555/5885/20 71.6
ImsgeNet
(1.2M)
ReaNert
Iroeption
952/96.K/ 99L1
88.//9.2/
97.6" 19960" 290.7
83.4/52.7/
60.5 06 74.9
90.6"
89./933./
$7.5/942/ 98.8
79.7781.35 /9566
$2.0/115
88.//114/441
3/939
98.5
SquasseMM
50.4/57.1270
62.67 3/.3/
Aserage
92.8% 96.1 98.6
74.9/ 96878677
60.61 75.6
799/546/995
98.7
86.4/853/922
73.5/ 76.5/8688
Table 1: White- and standard black-box settings. We set 10 and report the fooling rate (in %) on 5K samples from the ImageNet val-set. The best result in each section is shown in bold. denotes the white-box setting.
4 Experiments
We evaluate the effectiveness of our attack strategy in diverse settings. Below, we first discuss our experimental setup and then compare our attacks with the state-of-the-art generator-based ones: CDA [11], RHP [8], and GAP [10]. To further demonstrate the generalizability of our approach, we also report its performance on adversarialll-ttrinedd models and on the SSD [25] object detector with 4 different backbones. We discuss the experimental details in the supplementary material.
Datasets. To train the generator, similarly to [1I], we use data from either ImageNet [26], Comics [27], Paintings [28] or ChestX [29] as source domain, containing 1 1.2M, 40K, 80K, and 8K images, respectively. We then randomly select 5000 images from the ImageNet [26] validation set as target domain to evaluate the transfcrability of our attacks. As in [11, 18, 10), we report the fooling rate and the absolute difference in top-1 error between before and after the attack. The fooling rate is the percentage of images for which the label is changed after the attack.
Models. For the target models, we make use of the publicly available PyTorch [30] versions of VGG- 16 [31], VGG- 19 [31], ResNet152 [32], DenseNet121 [33], Inception-V3 [34] and Squeezenet [35] pretrained ImageNet. We chose this family of networks to understand the deeper impact of transferability across diverse architectures. As in [11], to evaluate transfer across datasets, we use ChestXNet [29] pretrained on ChestX. For the perturbation generators,
the same architectures as in [11. 10]. To train them, we use the Adam optimizer with a learning rate of 2e and a batch size of 16.
Attack settings. We perform attacks in four settings: 1. White-box attacks, where the attacker has access to the exact target model and target data distribution; 2. Standard black-box attacks, where the attacker has access to substitute model from a different family of architectures trained on the target data and to the target data itself; 3. Striet back-box attacks, where the attacker also uses substitute model trained on the target data, but without having access to the target data itself; 4. Extreme black-box scenario, we perform attacks without any knowledge of the target model or the target data.
4.1 Transferability to Unknown Target Model
Let us first study the standard white box and black-box settings. In this experiment, we train the generator with target data containing 1.2M ImageNet training samples, using one of the 5 ImageNet pretrained models as target architecture. We then evaluate the transferability of the attack to the remaining 4 models.
5

---

Evaluated Family
Evaluated ResNet Family
100%
Figure 5: Transferability within the same family, We report the fooling rates when transferring attacks to other net- works within the same family. The gen- erators were trained with 1e and ResNet1 with e 10.
50%
65%
Ours
Ours
60%
Dh:121
DM263
DN201
1928
IN54
Ab.2
RM501
RN152
(a) Trained on DenseNet121
(b) Trained ResNet152
Favarrtten vid11
Fax uated Derombe121
800
(a) VGG-16
(b) ReaNett52
(c) Inceptions3
(d) DenseNeri21
Figure 6: Limited data setup. We report the fooling rates obtained using varying amounts of ImageNet training data. The generators trained with SqueezeNet [35] and transferred to other networks. The r-axis represents the amount of training data.
10, and
Table 1 compares the effectiveness of our attacks to the CDA and GAP ones. We outperform these baselines on average by 10.5 and 13.5 percentage points (pp) (i.e., absolute difference of fooling rates), respectively. The differences are particularly pronounced when the generator is trained with SqueezeNet. Furthermore, in the DenseNet121 case, the difference between CDA and Ours 13.6pp.
similar trend can be seen when transferring attacks within the same family of networks. For instance, Figure 5 (a) shows the transferability when the generator is trained with DenseNet121 and evaluated on other networks of the same family, namely, DenseNett61. DenseNet169, and DenseNet201. Figure 5 (b) shows the results for similar experiments method performs on average 1668pp. 6.4pp better than CDA and GAP in the case of ResNets, which demonstrates that our feature separation loss generalizes well for attacks within the same family. Importantly, the performance gap between Ours and the baselines increases with the difference in depth from training- from 0.02pp DenseNet121 to 6.3pp DenseNer201 w.r.t. GAP.
the ResNet family. Our
In Figure 6, we further study the transferability attacks as the number of samples from the target data to train the generator varies. Our method (blue bar) consistently maintains higher fooling rates, even for small number of training samples, than GAP (red bar) and CDA (yellow bar). Note that all methods tend to saturate when around 5K samples are available, with our method consistently outperforming the basclines.
4.2 Transferability to Unknown Target Data
Let us now turn to the strict black-box setting, where the target data is unavailable to the attacker, who then relies on a different dataset to train the generator. Note that, here, the attacker still has access to substitute model trained on the target data, therefore we also study the impact of transferability to an unknown target model. Following [11], we consider two synthetic datasets, Comics and Paintings, depicting objects similar to the ImageNet ones, and one dataset, ChestX, containing completely different type of data, thus suffering from a large domain gap.
Table 2 compares the results of our approach with those of the baselines. Our method yields clearly superior results for all 3 datasets. For instance, with Comies, Ours outperforms GAP and CDA on average by 14.9pp and 10.3pp, respectively, and with Paintings, by 14.9pp and 1.0pp, respectively. Note that, while we still outperform the baselines when using the ChestX data, by around 22pp on average, the fooling rates of all methods drop significantly in this challenging scenario. Nevertheless, altogether, these results evidence that one can learn to generate adversarial examples with high fooling rates despite not having access the target data, even by using data from completely

---

Gen. Training
Dicrrminater (ImageNet)
VGG16
ResNet152
Inception-r3
DenseNet121
SqueezeNet
Average
(dara)
[AP[1O]/CO
/1tr/Ours
VGGG-1
99.8 9999799999999
54.3 54/0/40777 77.4
45.8/45./
46,9/
93.6
70.7/68.4/90
67.4766.3/
Comics
(40K)
DenseNet121
Inception V3
RetNer152
96,9/92.0, /6655
75.37
84.3/85.6/ 99.0
99.3
97.6/98- /981//91111 /996
91.2/97.3/
28.0/86.39933
90.4
83.1/65.7/ 82.5
51.77
99.//998/ 99.6
66.5 7731
S5.5/88.0/
/94//8.4/98.
45.1787:77
/ 9886
96.7
82.4/82.3 2/.3//./9/3 93.2
67.31 86.0 99.7
78.3/75.7/99
91.2/83.6/
90.5
90.5/90.6
(3.4/56.8//
/90.// 95.8
SqueezeNet
87.7/
51.0 /0.0//5./799
51.2/51.//54
66.7/76.3/
99.7/99.8/99)
72.3/75.1
Average
88.8/92.6/
80.2/72.8/872
6 78.5
73.67
79.7/82.5/9..8
76.9/50.5 /9.8
VGG-16
99.4/99.99 99.0
51.1/5766 66.6
36.57 66.6/ 50.0
50.8 73.8 84.6
63.7 / 16.4
58.3 70.1 77.3
Painiings
(80K)
ResNet152
DenseNer121
Inception V3
80.3
80.17 98.7
87.6/%6.5/9M. 96.2
9K.6
95.8 96.4 88.2
/5.5/97.51 9 99.4
/81.2
99.6 99.6 193.5
50.7 62.1 /
50.4/ 76.0
ST.7 87.2 /95.2
70.4 62,3 /97.9
/989 97.4
73.67755 73.7/
17.5 72.8 19088
70.4 /8.1 /89.2
78.3 78.1
88.2/87.33 3 144.5
83.5/ 91.6
SquezzeNet
92.8/80.7/95. 95.2
46.07 46.07 73.4
44.5/ 47.4
59.3/ 56.5 /772
99.4/199 99.3 99.6
66.41 66.0
Averge
71.7/758//83
56.5/612//799
73.4/99.//22
90./9915
72.9/76.
VGGG-6
74./8556//3..
23.2/233/41.8
25.5/22.//31.
27.5/28.2// 28.2 53.4
46.1 /48.0/ 64.3
40.2 (122:/4..//5 56.8
Chesx (10K)
Inception V3
RecNer152
39.9744487 1..44..8.550 $6.4
56,0/50.3/914
21.0 25.3 62.8
35.9/32.07 69.5
28.21 25.7 27.7
46.4/3..1/ S4,9
25.91
45.91
35.4 77.4
26.6 38.1
65.1 (57.7 10/557777555. 7756
44.9 44.9/47.1/
6605
33.2/ /3.2/33.99 /49,2
49.5 442.1 79.8
DenseNec121
42.8/42.3/640
41.91
76.0
54.2 /48.8/
38.7
SqueezeNet
51.7/51.1/8.1
27,9/316//22
52.5
30.2/33.1/ /47.1
31.6/33.1/64.2
/ 99.4
41.5 46.0// 68.3
Average
53.8/588/712
54.2222/54.2
31.3/30.1/ /4.00
11./3467/619
58.3/ /6.1 11.27440.5 62.0
Table 2: Transferability in the strict black-box setting with no access to target data. We set 10 and report the fooling rate for 5K samples from the ImageNet validation set.
different domain. Furthermore, the resulting generator generalizes particularly well to the unseen target domain when trained with our feature separation loss.
Discriminator (Model)
Gen. Training (data)
VGG16
ResNet152
Inception-43
DenseNet121
SqueezeNetL.1
Average
GAP [DA
CDA (11)/Ous
ComicstOK
65,4758.2/77 77.3
50,5
57,4
55.4 46.5 60.2
61.4748.17699
78.666.0/ 79.7
60.9/519/70.
[ trained on Chest))
ChestXNet
Comics10K
Paintings 10K
8a0
66.8/58.1
76.6/63.2/78 78.2
76.7/61.3 78.4
52./
53.81
46.2 56.8
55.2
59.9
56.8/529/
55.0
49.5/49.544 61.7
60.2
73.7/58.3/17
74.6/505/
79.1
77.7
78.6/67.4/88
79.3/70.2/
68.1/688//0
80.5
68.0/54.1 /7.9 60.9/52111
68.0/57.5 70.7
ChestX 1Ok
43.3/48.. 66.3
29.2/23.7148 49.5
26.9/26.5 48.5
28.8/ 65.0
/2.8/15.5/72
/7.1/34.5 33/33 60.4
Average
65.7/57.8/754
47.3/378/5588
49.9/11//7.8
61.7716.4/4.7
70.5/610/785
59.0/50.0;66
Table 3: Extreme cross-domain transferability analysis using generator trained with ChestXNet and different datasets. We set val-set samples.
10 and report the fooling rates on 5K ImageNet
4.3 Extreme Cross-Domain Transferability
We further study the transferability of adversarial attacks in the extreme scenario where neither the target architecture nor the target data are available, and where even the classifier used to learn the generator was trained on data that differs from the target domains. To this end, we use ChestXNet [29] pretrained on ChestX dataset as classifier. We train the generator with different datasets on dense- block12 and evaluate its transferability in Table 3. Despite the challenging nature of this setting, our method still yields satisfying fooling rate; 68.5pp on average vs 87.0pp in the standard black-box scenario. Furthermore, our approach outperforms the state-of-the-art CDA by 18.5% on average.
4.4 Transferability to Robust Models
In this section, we study the transferability of our generators five state-of-the-art defenses, i.c.. the high-level representation guided denoiser (HGD) [36], the input preprocessing defense through random resizing and padding (R&P) [37], Feature Denoising (FD) on ResNeXt-101 [38], Projected Gradient Descent (PGD) [39] on ResNet50, and the average of three ensembles of adversarially trained Inception models [40] (EnsembleAdv).
addition to GAP and CDA, we compare our method to RHP, which constitute the tfate-of-the-an universal attack in terms of effectiveness against defenses. As shown in Table 4, Ours achieves the overall best performance on HGD, R&P and EnsembleAdy, However, we observe that none of the methods are successful in attacking Feature denoising and PGD defenses. Note, however, that with PGD and FD, the error rate on the clean samples significantly higher than with other defenses, making these strategies ill-suited for practical applications.
7

---

Nework
Mithod
EmuembrrAd [40)
HGDD(6)
RP [P7)
Fi (38)
PGD
Ereer pormal samples
24.6
19.5
21.0
51.1
53.7
Table 4: Transferability to adversarially trained models. We report the absolute per- centage increase top-1 error with e 16
Inev3
RIP
29.5
26.8
23.3
2.38
2.40
Ine-v4
RHP
25.1
23.4
20.2
1.90
2.20
IncRes x2
RHP
28.8
26.9
25.1
2,20
2.20
SquceneNet
GAP
27.2
33.1
26.7
4.24
5.84
on the 5K samples from ImageNet val-set following [8].
SqureoeNet
CDA
25.8
11.4
26.5
4.06
5.42
SquezoeNet
Ours
X3.7
43.9
32.7
7,86
3.30
Gen. Training
Discriminator (Trineed on ImageNot)
VGG16
ResNet50
EfficennNNe
MobileNetts3
Average
(lata)
[10]/ CDA [1II]
No Attack
68.12
66.08
61.07
55.44
62.68
VGGG-1
14.9/22.//340 /360
14.27 20.7 7.55
11.3/126.8.00
07.5/11
11 4.30
16.9/16.9/
5.86
ReaNet152
32.1/24.0:900
25.5/
16.9/ 8.13
30.61
75.7
21.4/
12.0/ 5.44
17.4/ 77.58
Comics
(40K)
Inception-33
DemieNet21
25.1/23.2/880 88.8
33.7/32.77 18.0
29.0/ 28.5 20.2
21.8/1988
32.3/3133 14.5
20.4:20.7
22.7/22.1
16.8/1533/.66
14.0
11.5/19.8
29.4/28.6
16.6
SquaezeNer
78.5/268/13.7
25.1/244/11.a
20.8/208::12
14.2/
20.9/22.6/1L. 11.1
Average
2688/25.9/ 10.5
22.7/22.1/111
22.9/20.2211
16.57
16.07/.77
22.2/21.1/100
VGG-16
1687160/880
1K2/17.//2.1
11:67
11,
7.86 77.9 1
/3.6/12.9/
Paintings (8DK)
ResNe1
Inception-v3
31.07 99.66
33.9/33.0/ 16.0
20.5
28.4/ 28.7/ 18,9
/..1
27.4: 14.2
17.8
/ 7.72
22.6/214// 12.1
5.45
27.8/ 15.3
/1.1/ 8.04
De1teNe1211
28.5 /8.14
20.2/ 20.6 8,99
19,3/20.2:7.00
15.47 16.0
20.8: 21.5 7.75
SqueezeNer
23.8/23.8 11.4
12.9
7.69 15.9
11.4
Average
22.2/21.5/22 27.7/25.4/1.1
19.0/ 10.9
15.4/14.5
19.2
VGG-16
12.//2211/1.
9.19: 11.4
7,76/
ResNet152
7.8//66.///.56
13:37 13,7
11.0:932/6.15 6.15
12.27 /.22/666//3.66 3.662
13:57 11.6 6.48
ImageNet (1.2M)
Inception-1 v3
12.8/20.11 1.5.8
15.2/ 19.1 18.4
12.1 12.0
LL88 13.6/9.96
13.0 16.2 14.5
DenteNet121
20.3 19.9 4.73
14.07 14.6 6.37
11:71
11676.1
11.4 11.2/286
14.4/ 14.3 5,02
Squeezeeet
24.1/24.5 13.1
21.8/ 20.4 10.8
/7.1/20.1/115
12.7 15.4/6.19
19.27 20.1 10.4
Average
18.3/19.0/00
1644 16.0/1.0
123/9997
11.2/ 10.3/6.5
14.5/ 14.4914,49.311
Table 5: Transferability across tasks with access to neither the target data nor the target model. We set = 16 and report the mAP for 4952 samples from the PASCAL VOC test-set.
4.5 Cross-Tast Transferability Analysis
To demonstrate the transferability of our approach across tasks, we attack object detectors using perturbation generators trained with image classifiers. Specifically, we choose the SSD framework with 4 different backbones, namely, VGGI6, ResNet50, EfficientNet, and MobileNet-v3 pretrained on PASCAL VOC [41]. In Table 5, we report the mAP on the PASCAL VOC test-set, containing 4952 images, for perturbation generators trained with different image classifiers and with e = 16. The performance of the SSD detector on clean images is 68.1, 6,. 61.1 and 55.4 using VGG16. ResNet50, EfficientNet and MobileNetv3 as backhone, respectively. Our attacks significantly decrease these score and yield lower mAPs than the baselines in most cases. These results highlight, for example, that a generator trained on the synthetic Comics dataset with an image classifier can fool an SSD detector trained on different domain, with a different architecture, and for a different task. As evidenced by the comparisons with CDA and GAP, learning to generate perturbations that affect the mid-level features more effective than focusing on the classification boundary.
4.6 Additional Analysis
To further analyze our method, we visualize the resulting perturbed images using t-SNE plots. Specifically, in Figure 7()), we provide t-SNE plots obtained using the final latent representation of 1000 normal (blue) and attacked (yellow) ImageNet images for the strict black-box and extreme black-box scenarios. In both cases, our method yields features that clearly separate the normal and adversarial images with large margins, whereas CDA fails to do so. In Figure 8, we visualize the unbounded adversarial images obtained with different methods and also show the bounded adversarial images computed with e 10 with our approach in the last column. As can also be seen in Figure the perturbations follow repeating patterns corresponding to the patterns of the most disrupted mid- level filters. Note that RHP [8] explicitly enforces perturbation homogeneity using a predefined pattern, such as vertical or horizontal patterns. Our filter visualizations for multiple networks reveals that the bottom layers learn such horizontal and vertical stripes. While we tried training the generator to perturb such layers, we observed that larger strength are required in this case. Overall, our results evidence that attacking mid-level filters that have strong correlations across architectures consistently improves the transfer rates.
8

---

CDA
Train: Paintings on SqueezeNet; Eval: DenseNeti21
(b) Train: ComiesIOK on ChestXNet; ResNet152
Figure 7: t-SNE visualizations. We show t-SNE plots for 1000 normal (blue) images and their adversarial (yellow) counterparts in (a) the strict black-box setting and (b) the extreme black-box setting. More visualizations can be found in the supplementary material.
Original
GAP
CDA
Ours
Ours (Bounded)
VGG16
ResNer 12
DenseNet1211 22
SqueszeNet
Figure 8: Qualitative Results. Visualization of the unbounded outputs obtained with generators trained on ImageNet with different methods. Our approach produces images that are structured and better retain the content of the original images than the baselines. Best viewed in color and zoomed.
Limitations. our experiments, we choose a single layer time. To this end, we sweep over each block of layers, typically around 5 on average for the studied architectures. We empirically found that the best-performing layer is independent of the choice of the target model to attack. For example, a generator trained with feature separation loss on the conv4-1 layer in VGG16 transfers to all attack settings at inference time, avoiding repeated sweeps and computational costs. For all our experiments, we set the layer I to attack to relu after conv4-1, layer3, mixed6b, denseblock8, and fire 10 for VGG 16, 55, Inception-v3, DenseNet121, and SqueezeNet, respectively. We report additional results with generators trained at different layers in the supplementary material. Nevertheless, we have no guarantees that such layers truly are optimal; in particular, better results might be achievable by attacking ensembles of layers, but this would require tuning the layer combination.

---

Boxplot of top disrupted filters layer 18 VGG16
Boxplot top disrupted layer SqueezeNet pos
Synthesized images of top disrupted filters in VGGI6 Synthesized images of top disrupted filters in SqueezeNet
Unbounded adversarial images with VGG16
Unbounded adversarial images with SqueezeNet
Final projected adversarial images with VGG16
Final projected adversarial images with Squeeze
(a) Generator trained against VGG16
(b) Generator trained against SqueezeNet
Figure 9: Correlation between adversarial patterns and most disrupted filters. The first row shows box plots for the shift in magnituee of the top 30 disrupted filters, i.e., with the largest shift. In the second row, we show the corresponding visualizations obtained using [12] for the top 4 filters. The third row evidences the strong correlation between the top-disrupted filters and the unbounded adversarial images. In the last row, we visualize the bounded adversaries for 10. Comparing the left and right blocks shows that the disrupted filters in VGG16 and SqueezeNet contain visually-similar patterns, thereby explaining our high transfer rates. Best viewed in color and zoomed.
5 Conclusion
In this paper, have explored the of mid-level features in conjunction with generative networks to learn transferable perturbations. We have shown that a generator trained with feature separation loss successfully fool models across architectures and tasks, and that proxy dataset from different domain can be leveraged to learn effective perturbations. Our experiments demonstrate that method outperforms the oate--ofttheart attacks across wide range of architectures and tasks. We have limited our experiments to undefended models due to the unavailability of publicly available robust models across architectures but will study this in future work. Furthermore, we believe that a deeper analysis of learned filter banks with respect to changes in architectures can shed light on building better black-box models.
Broader Impact
Understanding the strength of adversarial attacks is critical to building robust defenses in the future.
The practical implications adversarial examples in safety-critich applications, such as autonomous driving and biometrics, are huge. Our extensive set of experiments highlights the vulnerability of modern networks to adversarial attacks even in the extreme black-box case, which has high potential for societal impact. We hope that our work will motivate the community to further study the internal workings of neural networks in the context of adversarial attacks, and in particular exploit such analysis to design effective defense mechanisms.
10

---

Acknowledgement
This work was supported by the Swiss National Science Foundation.
References
[1] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3441-3440, 2015.
[2] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 1105, 2012.
[3] Shaoqing Ren, Kaiming He, Ross B Girshick, and Jian Sun. Faster r-can: Towards real-time object detection with region proposal networks. In NIPS, 2015.
[4] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, (5(1):221-231, 2012.
[5] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus, Intriguing properties of neural networks. arXiv preprint arviv: 1316.6199, 2013.
[6] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 141.65572 2014.
[7] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arX6v: 06083, 2017.
[8] Yingwei Li, Song Bai, Cihang Xic, Zhenyu Liao, Xiaohui Shen, and Alan L Yuille. Regional homogene- ity: Towards learning transfcrable universal adversarial perturbations against defenses. arXiv preprint arXiv: UrXiv:I904.0000 1904.00994.0 2019.
[9] Shumeet Baluja and lan Fischer. Adversarial transformation networks: Learning to generate adversarial examples. arXiv preprint arXiv:1703.0988 2017.
[10] Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4422-4431, 2018.
[11] Muzammal Naseer, Salman Khan, Muhammad Haris Khan, Fahad Khan, and Fatih Porikli. Cross-domain transferability of adversarial perturbations. In 33rd Conference on Neural Information Processing Systems (NeurIPS), volume 32. NEURAL INFORMATION PROCESSING SYSTEMS (NIPS), 2019.
[12] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009.
[13] Alexey Kurakin, Ian Goodfellow, Samy Bengio, et al. Adversarial examples in the physical world, 2016.
[14] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pages 39-57 57. IEEE, 2017.
[15] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (EaroS&P), pages 372-387. IEEE, 2016.
[16] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9185-9193, 2018.
[17] Seyed- Mreed-Mohsennaa Moosavi- Dezioiii,,,oi,,, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574-2554 2582, 2016.
[18] Seyed- Mohsen Moosavi-Dezfoole Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adver- sarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1765 1173, 2017.
11

---

[19] Valentin Khrulkov and Ivan Oseledets. Art of singular vectors and universal adversarial perturbations. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 8562-8570,
2018.
[20] Konda Reddy Mopuri, Utsav Garg, and R Venkatesh Babu. Fast feature fool: A data independent approach to universal adversarial perturbations. British Media Vision Conference, 2017.
[21] Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7066- 7074, 2019.
[22] Konda Reddy Mopuri, Phani Krishna Uppala, and R Venkatesh Babu, Ask, acquire, and attack: Data-free uap generation using class impressions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 19-34, 2018.
[23] Nathan Inkawiich, Kevin J Liang, Lawrence Carin, and Yiran Chen, Transferable perturhations of deep feature distributions. arXiv preprint arXiv:2004. 12519, 2020.
[24] Jamie Hayes and George Danezis. Learning universal adversarial perturbations with generative models. In 2018 IEEE Security and Privacy Workshops (SPW), pages 43-49. IEEE, 2018.
[25] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 2 21-37. Springer, 2016.
[26] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105,
[27] Cenk Bircanoglu. Painter by number. http::/wwwwaagge...c.cccocmomm/s
2017.
[28] Painter by number. http::/wwww..eeee..o////////////
2017.
[29] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang. Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. Chexnet: Radiologist-leva pneumonia detection on chest x-rays with deep learning. arXiv preprint 17XIiv:711..52200 2017.
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, tigh-peeformance deep learning library. arXiv preprint arXiv: 1912.01703, 2019.
[31] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1506. 2014.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[33] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700 4708, 2017.
[34] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Rethinking the inception architccture for computer vision. In Proceedings of the IEFE conference on computer vision and puttern recognition, pages 2818-2826.. 2016.
[35] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size, arXiv preprint arXiv.1602.07366 2016.
[36] Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang. Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1778 1787, 2018
[37] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In International Conference on Learning Representations, 2018
[38] Cihang Xie, Yuxin Wu. Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 501 509, 2019.
12

---

[39] Aleksander Madry. Aleksandar Makelov. Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference Learning Representations, 2018.
[40] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, lan Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv:1705.07204, 2017.
[41] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303 338, 2010.
13


Reviews:
[{"review_id": "fzpILLwbnlj", "reviewer": "Reviewer_WvGo", "summary": "This paper shows that mid-level features show a common pattern across different domains/tasks/architectures. Motivated by this, this paper introduced to train a generative model which leveraging the mid-level features to produce adversarial examples. Experimental results demonstrated that the proposed method is effective under different settings. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "---\nStrength:\n- The introduced methods show improvements over existing methods (CDA) in several different settings, including a challenging black-box attack setting. The attacker does not know any information about the target architecture/dataset/task. \n- The experiments results are well designed, comprehensive and outperform existing methods. Analysis using t-SNE shows the introduced method can produce well-separated features even in cross-domain/architecture attacks.\n- The writing and presentations of this paper are easy to follow.\n---\nConcerns and Questions:\n- Although the visualization of mid-level features in Figure 2 shows certain similarities, a qualitative metric would be more convincing. Such as CKA/CCA [1].\n- Why not include RHP in Table 1,2,3? It is also a baseline method.\n- In Table 3, why is the fooling rate lower for ChestX10k than others? Using ChestX data to train generators against ChestXNet should have a higher fooling rate, and this result is a bit puzzling. \n- Figures 3 and 4 in supplementary is the missing label for the x and y-axis. Presumably, x for layers and y for fooling rate, there is no need for >100 for the y axis. Is Figure 3/4 only considered the mid-level layers for different architectures? For ResNet models, layers are chosen from 1-4, while others choose from a wide range of layers. Is there any method/reason for how to choose these layers? The introduced methods rely on the choice of intermediate layers for the attack. There should be a more comprehensive ablation study on this. Such as a simple comparison for introduced method applies to low/high-level features with mid-level features. \n\n[1] Similarity of Neural Network Representations Revisited, ICML 2019.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have adequately addressed the limitations and potential negative societal impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "GnquoY2ANK", "reviewer": "Reviewer_gZcr", "summary": "This paper studies the transferability of adversarial examples crafted by attacking one model to another model. The main idea is to employ a generator to produce adversarial examples and maximize feature divergence on a latent space.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The core idea of this work is to run an attack on mid-features at the intermediate layer of the source network. The main intuition is that mid-layers tend to learn common patterns (e.g., textures) across various deep nets trained on the same dataset or similar datasets.\n\nHowever, the arguments in the paper regarding why maximizing feature divergence on mid-layers can boost transferability have not been addressed comprehensively and thoroughly. The current arguments are too general to me. It would be better if the authors delve into what really happens with mid-layer features and what changes in the pattern of adversarial examples when maximizing feature divergence on mid-layers. This would help insightful demonstrate the usefulness of the proposed method.\n\n--Post rebuttal---\nThe responses from the authors have addressed my concerns so I have increased my score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although the experiments are quite comprehensive, the proposed method is restrictively novel. Moreover, the arguments of the proposed method are too general, hence needing a more comprehensive study.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "DSNV8GSRzKw", "reviewer": "Reviewer_9Cvd", "summary": "The paper proposes an adversarial attack generated by another deep neural network (DNN), that shown to have potential to transfer the attack to different network architectures, different datasets and different tasks. The authors inspire the attack from an observation that mid-level features of a DNN share similarities with different architectures, datasets and tasks. Learning the adversarial perturbation generator network to attack the mid-level features of a DNN shown to transfer to different attack settings. The effectiveness of the attack is justified on different attack settings (white box and variants of black box attacks) that emphasize the transferrable nature of the attack.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nLearning the generator to attack the mid-level features and demonstrating such generator to transfer to different attack settings is interesting and novel. Previous works are cited and differences are discussed sufficiently.\n\nQuality: \nThe paper provides an interesting finding and technically sound. The proposed method is appropriate and the claims are well supported by the extensive experimental results. This is a complete work and the strengths of the method are reveled by crafting the experiments carefully. The weakness of the method is not sufficiently addressed (see below). \n\nClarity: \nThe paper is clearly written, well organized and also provided details to reproduce the results.\n\nSignificance: \nThe paper addresses an important topic of transferring the attack to different test settings.\nThe results of the proposed attack outperforms the previous attacks in all the different attack settings. Future works would benefit from the unique findings made in this work. \n\nI appreciate that authors study the effectiveness of their method on few of the adversarially trained models. However, I wonder that the authors missed the PGD based adversarially trained model as a baseline to test their method. CK Mummadi et al. has shown that PGD based adversarially trained models are also robust to universal perturbations and thus raises a question how their method performs on such defense method. \n\nBesides the interesting finding that mid-level features are shared across different architectures, my major concern about this work is the assessment of the proposed attack against potential defenses. How strong is the proposed attack when a defense is crafted to address it specifically? It seems the attack objective could be easily get integrated into PGD based adversarial training to achieve a robust model. If so, I wonder whether the proposed attack would circumvent such defense mechanism. I believe this is an important discussion to be addressed in order to evaluate the significance of the proposed attack. Finally, I rate this paper as marginally above acceptance threshold. However, I am willing to reconsider my rating based on the author responses and other reviews.\n\nCK Mummadi et al.  Defending Against Universal Perturbations With Shared Adversarial Training, ICCV 2019.\n\n-----------------------------------\nPost rebuttal remarks:\nI read the other reviews and authors responses. I thank the authors for conducting additional experiments on defended models. The results show that the proposed attack is effective against HGD and R&P defended models, but not effective against PGD and Feature Denoising based defended models (same as other attacks). However, my major concern of crafting a defense that specifically target to circumvent this attack still remains open and hence I keep my rating same as before.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I appreciate the authors for covering the limitations and broader impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "D2pagg6T3pS", "reviewer": "Reviewer_cUgt", "summary": "The work studied the generalization of adversarial attacks that are induced by generative methods. Specifically, the transferability of the perturbations for different testing conditions are evaluated. The mid-level features of DNNs are used to learn a transferable perturbation generator. Evaluations of transferable attacks are performed on tasks with a different network architecture, and/or data distributions. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "originality: the work presents an interesting finding that by maximizing the difference of mid level features of neural networks between the input image and the perturbed image, the generated perturbation can be transferred to another task setting such as a different data distribution or a different network structure. This finding itself is a useful contribution. However, the method of using generative method to generate perturbation is not new. The results indicate that there is a large gap between the white-box setting and the black-box setting. This means that when the target data distribution is hugely different and/or target network architecture is different, the transferability of the attack degrades. \n\nquality: the paper is technically sound and the claims are solid. However, the results indicate that there is still a large gap between the non-transfer case and the transfer case. In order to train an attacker that can transfer or generalize well to different settings, there could be other methods such as using meta learning to train the attacker on multiple data sources or multiple network architectures. It would be great if the author could provide results on meta learning based attack to see how good is the proposed attack comparing with meta learning based attack. \n\nclarity: the paper is presented in a clear way except for some tiny fonts in some of the figures. \n\nsignificance: the results are important and the insight from the paper is useful for other applications. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mentioned their limitations pretty well, in terms of only testing on a single layer. However, it would be great if the authors could add additional experiments comparing with meta learning or domain randomization based attack for improving generalization of attacks. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studied the generalization of adversarial attacks induced by generative methods. The authors found that, by maximizing the difference of mid-level features of neural networks between the clean and perturbed images, the generated perturbation can be better transferred to another task setting such as a different data distribution or a different network structure. The finding is interesting and useful to the community. However, there is still a gap between the attack performance under the white-box and black-box settings. Furthermore, the proposed attack is effective against HGD and R&P defenses, but not effective against PGD and feature-denoising-based defenses. We suggest the authors discuss these in more detail in future revisions.
Decision: Accept (Poster)