A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attacks
Authors: Jacob M. Springer
Melanie Mitchell
Garrett T. Kenyon
Venue: NeurIPS 2021 Poster (2021)
Keywords: adversarial examples
robustness
transferability

Abstract: Adversarial examples for neural network image classifiers are known to be transferable: examples optimized to be misclassified by a source classifier are often misclassified as well by classifiers with different architectures. However, targeted adversarial examples—optimized to be classified as a chosen target class—tend to be less transferable between architectures. While prior research on constructing transferable targeted attacks has focused on improving the optimization procedure, in this work we examine the role of the source classifier. Here, we show that training the source classifier to be "slightly robust"—that is, robust to small-magnitude adversarial examples—substantially improves the transferability of class-targeted and representation-targeted adversarial attacks, even between architectures as different as convolutional neural networks and transformers. The results we present provide insight into the nature of adversarial examples as well as the mechanisms underlying so-called "robust" classifiers.

A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attacks
Jacob M. Springer Los Alamos National Laboratory Los Alamos, NM, 87545 bacmspringer@gmail. com
Melanie Mitchell Santa Fe Institute Santa Fe, NM, 87501 mm@santaffe. edu
Garrett T. Kenyon
Los Alamos National Laboratory Los Alamos, NM, 87545 gkenyon@lanl
Abstract
Adversarial examples for neural network image classifiers are known to be trans- ferable: examples optimized to be misclassified by source classifier are often misclassified as well by classifiers with different architectures. However, targeted adversarial oottimize to be classified as chosen target class-tend to less transferable between architectures. While prior research on constructing transferable targeted attacks has focused on improving the optimization procedure, in this work we examine the role of the source classifier. Here, we show that train- ing the source classifier to be "slightly robust"-than is, robust to small-magnitude adversarial
improves the transferability of class-targeted and representation-taree tarreettd adversarial attacks, between architectures dif- ferent as convolutional neural networks and transformers. The results we present provide insight into the nature of adversarial examples well as the mechanisms underlying so-called "robust" classifiers.
Introduction
Neural- -etwork image classifiers are well-known to be susceptible to adversarial examples examples-imagee images that perturbed in way that is largely imperceptable humans but that the neural network to make misclassifications, Much research has gone into methods for constructing adverarial examples in order understand the nature of neural-network vulnerabilities and develop methods make neural-network classifiers more robust to attacks
45 50 55 73
Untargeted adversarial examples are designed elicit unspecified incorrect class, while targeted adversarial examples are designed to elicit a specific (incorrect) target class. A given adversarial perturbation is designed via optimization with respect to given trained network; here call this the source network. While untargeted adversarial examples are often transferable teanserralle-examples designed to attack source network also successfully attack other trained networks with different parameters and targeted adversarial examples tend to be less transferable [44].
architectures 21 [3
Prior research on constructing transferable adversarial examples for image classifiers has focused primarily on improving optimization methods for generating successful image perturbations. this paper, we take different tack- tack-focusing examples. Specifically, we find that "slightly-robust" convolutional neural networks (CNNs)-one that have been trained to be robust small adversarial perturbations can be leveraged to substantially improve the transferability of targeted adversarial examples to different architectures. Surprisingly, show that targeted adversarial examples constructed with respect to slightly robust CNN transfer successfully not only to different CNN architectures but also to transformer architectures such as ViT [16), LeViT (22), CCT (24) and even CLIP [57]. which is trained different objective than the source CNN. Such transfers are all "black box" attacks-whie generating an adversarial example
the source neural network used in constructing adversarial
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

requires knowledge of the source network's 's architecture. no such knowledge is required for the black-box architectures that can also be attacked by the same example.
The vulnerability of networks to both targeted and untargeted attacks has huge significance for security purposes, and thus understanding how construct highly effective attacks can help motivate defenses. However, we believe that targeted attacks are especially important for understanding neural-network classifiers, as they provide tool to compare the features of two models. When a targeted attack transfers from one network to another, it suggests that the two networks rely on similar information for classification, and that they use the information in the same way. In our work, we show that each individual slightly-robust neural network transfers features effectively to all tested non-robus. networks, suggesting the surprising result that slightly-robust networks rely on features that overlap with every non-robust network, even though it is not the case that any particular non-robust network has features that substantially overlap with all other non-robust networks.
In addition, we leverage the techniques of adversarial transferability to examine which features are learned by neural networks. In accordance with prior work [7]], we find that, on the spectrum from non-robust (standard) to highly robust classifiers, those that are only slightly robust exhibit the most transferable representation-taageledd adversarial examples, suggesting that the features of slightly-robust networks overlap substantially with every tested desination network. This can explain why slightly robust networks give rise to more transferable adversarial attacks and have better weight initializations for downstream transfer-learingg tasks [43] 76, 80).
The main contributions of this paper are the following:
1. We demonstrate that adversarial examples generated with respect to slightly robust CNNs are more transferable than those generated with respect standard (non-robust) networks, This transferability extends not only to other CNNs, but also to transformer architectures.
2. We find that, as the robustness of the source network increases, there is also substantial increase in transferability of targeted adversarial examples to adversarially-defendde networks.
3. We examine the role of the adversarial loss function generating transferable adversarial exam- ples.
4. We show, surprisingly, that non-robust neural networks do not exhibit substantial feature (rep- resentation) transferability, while slightly-robust neural networks do. This helps explain why slightly-robuse neural networks enable superior transferability of targeted adversarial examples.
2 Background
Adversarial Examples. In this paper we are primarily concerned with properties of source net- works that facilitate transferability of adversarial examples. Let F X' -> y denote a '"whieeboxx" net- work (i.e., one whose architecture and weights are known to the adversary) and let G X + y denote a "black-box" network (weights and architecture are unknown to the adversary). Let (2r, y) X x y be an (unperturbed) input-label pair, where X' is the input-space and ) is the label-space. Given maximum perturbation size s, we construct an adversarial example x + where [|6]| I E, such that F(x + 5) + y for the untargeted case, and F(x + 6) t for some target class E y for the targeted case. We then say that + ô is transferable to black-box network G G(x + 6) + for the untargeted case and G(æ + 8) = t for the targeted case.
Optimizers. Prior research has identified number of methods for optimizing adversarial examples given a white-box classifier F, many of which are based on the Iterative Fast Gradient Sign Method (I-FGSM) [2 (4), in which a perturbation 8; is iteratively updated to maximize the loss of the network while obeying an lo norm constraint.
We adopt the th-t method recently proposed by 91,, which combines three variants of I-FGSM and optimizes over many steps:
1. Diverse Input Iterative Fast Gradient Sign Method (DI2-FGSM), which applies a random affine transformation to the input at each step prior to computing the gradient [87]. Translation-IIvariannt Iterative Fast Gradient Sign Method (TI-FGSM), which convolves the gradient with a Gaussian filter [1].
Momentum Iterative Fast Gradient Sign Method (MI-FGSM), in which momentum term added to the gradient [13]

---

We follow the convention of Zhao et al. [91] and call the combination of these processes TMDI-FGSM We describe the method in detail in the appendix.
For targeted adversarial examples, the loss function should be maximized when the target label is predicted with high confidence. For untargeted adversarial examples, this occurs when the predicted label differs from the true label, and when the true label is given low confidence. A number of adversarial loss functions have been proposed, including standard cross-entropy loss [73], CW loss 77,, and feature-disruption loss [34]. We use the highly-effective logit loss, proposed by [9]], which maximized for targeted examples when the logit score for target class (i.e., the value of the output neuron associated with the target class prior to the softmax operation) is maximized. Similarly, the untargeted version aims to minimize the logit score associated with the true class.
Constructing Robust Source Networks. We construct robust source networks by performing
adversarial training with projected gradient descent [21 45 Each source network is trained to be robust to adversarial examples with l norm less than a specified E parameter, which we call the robusiness parameter. For this paper, we rely on pre-trained robust ImageNet models [62], which have been released under the MIT License. These source networks, along with many of our experiments, are implemented in Py PyTorch [56].
Features. In this paper, we will refer to neural network features 32
78 A feature f X IR
maps input to real number to describe how "strongly" the feature appears in the image. Every neuron in a neural network computes feature. However, we are primarily concerned with the representation-layee features, i.e., the features computed by the neurons the penultimate layer (32]. When we are referring to the features of specific neural network, we are referring to the features described by the neurons in the representation layer. When we say that the features of two different neural networks overlap, we mean that the patterns of the input that affect the features of one neural network also affect the features of the other neural network. When features are easily manipulated by small perturbations to the input, they manipulated in this way, they are said to be robust [32. Robust networks, i.c., networks that are less vulnerable to adversarial perturbations, should rely primarily robust features, though non-robust. networks can rely on a mixture of non-robust and robust features 32 71
said to be non-robust; likewise when they not easily
3 Transferability of Adversarial Examples
In this section, we describe the methodology and results of our experiments on the transferability of adversarial examples as a function of the robustness of the source network. We evaluate the targeted
and untargeted effectiveness of each constructed adversarial example collection of convolutional classifiers, Xception 19), VGG [68) ResNet [25] 266., Inception 74,, MobileNet (29). DenseNet (30), NASNetLarge (93), and EfficientNet 775) as well transformer-be -basedee----adddddeee classifiers, ViT 16) LeViT
(2]. CCT 224, and CLIP [57]. Here we use the term destination network to denote the networks on which we will evaluate transferability of adversarial examples that were generated with respect to a
source network. For our ImageNet experiments, we rely on pre-trained models [10 62).
Generating Adversarial Examples. We choose 1000 images randomly from the ImageNet vali- dation dataset such that every image has different class. We generate target classes randomly for each image such that each class is a target for exactly one image, and no image has a target that is the same as its true class. For each classifier (of different robustness), we generate targeted adversarial examples for each of the 1000 selected images, targeting the associated target class. To optimize each adversarial example, we run the TMDI-FGSM algorithm for 300 iterations. When the exact image input dimensions differs between the source and destination network, we rescale the adversarial example to fit the dimensions required by the destination network using bilinear interpolation. To generate adversarial examples, we use the Robustness library 17
Transferability to Convolutional Network Classifiers. We find that across every destination convolutional network, using adversarial examples optimized with respect to a source network with small robustness parameter improves transfer success rate in both the targeted and untargeted setting compared to the success rate of the non-robust (s = 0) network (Figure IT.. Our results on untargeted adversarial examples can be found in the Appendix. The success peak is approximately the same
3

---

Destination networks
Xception
VGG16
ResNet50V2
InceptionV3
MobileNetV2
DenseNet121
1.0
Targeted transfer success rate
0.5
0.0
0.5
0.5
0.0
1.0
0.0
1.0
0.0
0.5
1.0
0.5
NasNetLarge
EfficientNetB4
VIT
LeViT
CCT
CLIP
1.0
0.5
0.0
0.5
0.5
0.0
1.0
0.0
1.0
1.0
0.5
Source model e-robustness
Maximum perturbation size:
12
24
Baseline transfer rate
Figure 1: Targeted transfer attack success rate against ImageNet classifiers using adversarial examples optimized with respect to e-robust ResNet50 source models. Success rate is the fraction of ad adversarial examples classified as their adversarial target by the destination network. Higher is more successful attack. Baseline refers the rate at which unperturbed images are classified as the target class. (Best viewed color.)
across every convolutional destination network = 0.1), suggesting that there is optimal source- network robustness parameter in order to maximize transferability to convolutional architectures trained ImageNet. Note that attacks do not exclude images that already misclassified (untargeted case), and misclassified as the target class (targeted case) prior to adversarial perturbation. In Figure [ plot the baseline performance of the attacks, i.e., the performance of the attacks for unperturbed images.
Interestingly, we observe substantial drop in attack success robustness increases past the optimal value. While not shown in Figure [ this trend continues as E increases above 1 (see Appendix). We hypothesize that the robustness of source network increases beyond a certain point. the network begins to entirely ignore many of the non-robust features relied upon by the (non-robust) destination networks, and thus attacks do not modify these features, reducing attack success. This is consistent with prior literature [78). We dedicate the second half of this paper to describing a possible mechanism to explain these results.
Transformer- -ease Classifiers. Few studies have addressed the robustness of transformer-based classifiers to transfer attacks [66]. our knowledge, our paper is the first address targeted transfer attacks against transformer architectures. Shao et al. [66] finds that transformer- based image classifiers are more robust transfer attacks than convolutional classifiers, especially when the source model is convolutional. We find an even more striking result: with only minimal attack performance against transformer networks, previously published methods that use non-robust source networks are almost entirely ineffective at constructing targeted transferable adversarial examples using convolutional source models (Table []. This suggests that the features learned by transformer-bast models and non-robust convolutional models are largely different.
However, find that using slightly-robust ResNet50 classifier as a source network dramatically improves the transferability of targeted adversarial examples to transformer- optimal robustness parameter for the source network is different for destination transformer net- works and destination convolutional networks, though we find that any amount of robustmess in the source model (below critical value) substantially improves transferability. Thus, in a real-world black-box attack setting-in which the destination network's architecture is entirely unknown-aa an adversary could find balance between source network robustness parameter that optimizes transfer performance for CNN classifiers and for transformer-bact based classifiers.
classifiers. The

---

Table 1:
Direct comparison of targeted transfer attack success rate between our technique (slightly- robust source models, i.e,, E > 0) and previously proposed strong baseline attacks (non-robus source models, i.e., & = 0). We compare three different loss functions: cross-entropy, Poincaré distance combined with triplet loss, and logit loss. In addition, we report the success rate of FDA from the original paper (see text for discussion). We limit the foc norm of the adversarial examples to the standard value of 16/255.
Xcept VGGI6 RN50v2 Inc MNv2 DN121
NNL ENB4
VIT CLIP
Xent 8
10.4
9.6
10.6
6.4
40.5
13.1
6.8
0.8
Po+Trip E =
20.8
15.2
10.0
23.0
11.6
59.3
31.2
14.2
1.3
0.3
Logit == 0
45.9
40.0
15.3
43.6
22.9
77.9
60.3
39.6
3.9
0.4
FDA*
43.5
22.9
57.9
Xent E 0.1
54.0
59.4
45.8
50.8
32.1
78.8
66.0
41.1
8.6
2.4
Po+Trip e 0.1
59.1
S7.9
53.0
56.5
39.2
78.4
72.6
45.1
11.4
3.3
Logit ε 0.1
77.2
78.8
64.5
73.1
52.5
84.0
81.6
68.9
33.4
11.2
Xent ==1 60.4
69.3
66.6
58.2
46.7
69.9
61.3
56.9
29.9
19.9
Po+Trip E == 48.5
54.4
60.2
49.5
39.9
62.6
53.3
45.0
22.0
12.4
Logit
56.1
62.4
59.5
55.0
47.2
67.0
58.3
53.6
36.0
26.7
CLIP. Radford et al. [57 describes CLIP, a transformer-base classifier based on the VIT architec- ture. CLIP is trained to simultancously encode images and short textual descriptions of the images. CLIP can be used for highly effective zero-shot classification by determining which class label, encoded as text, has an encoding most similar that of the input image. Despite the fact that CLIP has not been explicitly trained with ImageNet labels or to optimize for the classification task, we find that the transfer performance of targeted adversarial examples is improved substantially when the source network (ResNer50) is slightly robust (Table rightmost column), again suggesting that slightly-robust neural networks rely on features which overlap with non-robust networks, even when the non-robust networks differ in architecture and training algorithm.
For our experiments, we use ViT-B/32, the CLIP architecture based on ViT-B/32, CCT-14//7x,, and LeVTT-256.
Improvements Upon Existing Attacks. We directly compare the targeted transfer attack success rate previous state-of-the-art black-box attacks and find that method substantially outperforms the previous methods under similar constraints (Table I In particular, we evaluate our method's attack performace with three different loss functions: standard cross-entropy loss (Xent), Poincaré distance with triplet loss term (Po+ Prip) [41) and logit loss. We include a comparison with the feature distribution attack (FDA) (33], however, FDA requires that we train multiple supplemental models for each individual target class, which would require thousands of supplemental models to attack all thousand classes of ImageNet. Thus, we do not perform direct comparison and instead report the targeted transfer attack success rate that is reported in the original FDA paper [33
Attacking Adversarially-Traie Trained Models. Adversarial training has been shown to improve robust- ness to transfer attacks [45). We evaluate the transferability of adversarial examples to adversarially trained destination networks. Even though the adversarial perturbations which we use to attack each adversarially-traine network larger than the magnitude for which the destination networks trained to be robust, the adversarial examples generated using non-robust networks do not transfer to the adversarially trained networks. However, shown in Figure as the robustness of the source network increases, the attack success rate increases substantially. Similar to Figure [ Figure 2 includes the baseline rate which the destination network classifies unperturbed images as the (incorrect) target class.
Extending Our Methods to CIFAR-10. We repeat many our experiments for the CIFAR-10 dataset (39]. The results are consistent with our ImageNet results. We present and discuss these results in detail in the Appendix.
Computational Limitations. Due to the computational requirement of training multiple robust ImageNet classifiers, restrict our experiments to those compute using pre-trained robust
5

---

DenseNet161, e
VGG16, MobileNetV2, ==3
ResNeXT50, e=3
ShuffleNetV2, =
Targeted transfer success rate
1.0
Source model e-robustness
Porturbation size:
12
24
Baseline
Figure 2: Targeted transfer attack success rate against adversarially trained (c 3) destination ImageNet classifiers, where examples generated using e-robust ResNet50 source networks. Higher is more successful attack. Baseline refers the rate at which unperturbed images are classified the target class. (Best viewed in color.)
networks. Thus, test adversarial attacks only using the ResNet50 architecture source network. and we evaluate attacks only on destination networks which are easily available to us. Similarly, we do not compute ensemble attacks using slightly-robust source networks, although expect this technique to improve the success of our attacks.
4 Adversarial Transferability of Features
We have shown that we can construct class-tave hass-targeted adversarial examples that transfer a specific (incorrectly) predicted class destination networks. In this section, aim address important question to help us answer why class-targeted adversarial examples transfer: to what extent do class- targeted adversarial examples transfer the representation-layer features imagine that class-targeted transferability arises from the overlap of a small number of especially vulnerable features where manipulation of these features can change the model classification, or, alternatively, the overlap of many features. To answer this question, we will construct and evaluate representation-taree -tareeeed adversarial examples. We refer the degree which adversarial examples of source model can analogously affect the features that are computed by the representation layer of the destination model the representation transferability from source destination. By contrast, class transferability (what is commonly referred to as just transferabiliry) refers to the degree to which adversarial examples can analogously affect the output of the destination network.
networks? One could
Representation Transferability. Our goal is to study the representation transferabilitty of source classifiers (with varying degrees of robustness) to non-robust models. We will show that slightly- robust networks exhibit substantially higher degree of representation transferability than non- robust networks and more-robust networks. This can directly explain why class-targeted adversarial examples constructed using slightly-robust source networks examples generated with slightly-robust networks will broadly transfer features, and will thus rely less on a small number of hughelevulneraale features that may be present in every model. this section, we show that networks are highly transferable, even across a substantial difference network architecture, such as the difference between CNNs and transformer networks.
more transferable, adversarial
tettssentttiittttrettte adversarial examples generated with slightly-robust
Measuring Representation Transferability. To measure representation transferabililty, we rely on simple but powerful test assess the similarity between the representations produced by two different networks. Let æ and be two inputs that produce identical (or very similar) patterns of activation in the source network's representation (penultimate) layer. If the source network has high degree of representation transferability to the destination network, then the responses to and will be very similar in the destination network well. By contrast, if and y do not share similar representations the destination network, then the source network has a low degree representation transferability.
This method allows us test the representation transferability source network to destination network by constructing images and y with similar representations in source network and

---

Xception
VGG16
RN50
IncV3
MNV2
DN121
NNL
ENBA
VIT
LeVIT
CCT
CLIP
Targets
Figure 3: 1-SNE plots of destination-network representations representation-ttageeed adversarial examples generated by using whitebox ResNer50 models of specified e-robustness. The 10 images at the bottom are the ones from ImageNet that we use as representation targets, as described the text. (Best viewed in color and magnified.)

---

Table 2: Cosine similarity between representations (in the destination network) of representation- targeted adversarial examples y and the corresponding target image 2, as function of robustmess parameter ε of the source network. Each value an average over the 9900 (x, y) pairs of representa- tion vectors.
Source network robustness parameter (E)
Destination
0.01
0.03
0.05
0.1
0.25
0.5
1
3
5
Xception
0.462
0.505
0.531
0.563
0.594
0.585
0.572
0.543
0.449
0.404
VGGI6
0.333
0.401
0,417
0.494
0.528
0.520
0,520
0,486
0,383
0.333
ResNet50V2
0.284
0.348
0.379
0,432
0.497
0.496
0.510
0,484
0.380
0.321
InceptionV3
0.577
0.612
0.627
0.644
0.673
0.662
0.655
0.636
0.572
0.539
MobileNetV2
0.431
0.459
0.460
0.493
0.517
0.513
0.513
0.504
0.455
0.425
DenseNet121
0.672
0.689
0.685
0.713
0.726
0.714
0.706
0.679
0.616
0.584
NasNetLarge
0.356
0.422
0,452
0.488
0.541
0.513
0.482
0.437
0.315
0.271
EflicientNe:B4
0.085
0.111
0.137
0.144
0.237
0.220
0.226
0.202
0.112
0.074
VIT
0.066
0.087
0.109
0.129
0.195
0.206
0.206
0.203
0.120
0.086
LeViT
0.051
0.077
0.096
0.111
0.165
0.163
0.176
0.176
0.107
0.07
CCT
0.048
0.081
0.109
0.137
0.202
0.227
0.248
0.241
0.144
0.093
CLIP
0.529
0.541
0.550
0.563
0.585
0.599
0.606
0.613
0.581
0.566
measuring the similarity of the representations in the destination network. To construct these images,
we select two images, x and yo from the ImageNet testset. We construct a representation--trgeee adversarial example y yo + 8 targeting the representation of x. More precisely, we run the TMDI-FGSM algorithm to minimize the the &g distance between the representations of x and yy:
arg min (x) Se
(y + +5
where FreP represents the representation layer of the source network. Since we want to observe how well the representations transfer under the conditions of typical adversarial examples, the perturbation to have an fxo norm of e, which, for this paper, we set to be the standard 16/ 16/255, To limit the computational requirements of this experiment, representations we use as targets x and 990 images to use as initial images yo- Of these 1000 total images, no two share the same ImageNet class. For each source network, and for each initial image yo. we construct ten representation-tuue targette adversarial examples- one for each target r-for a total of 9900 representation-taagenn -argeted adversarial examples per source classifier. For each target æ, the 990 tetpseentationttageteee adversarial examples will have similar representations to x in the source network. By measuring the similarity of the (x. y) representations in the destination network, we can measure the representation transferability of the source network.
limit
randomly select 10 images whose
Here we present different similarity metrics for comparing representations of and y. First, we plot the t-distributed stochastic neighbor embedding (I-SNE) of the representation vectors of each representation-targeee
adversarial example in the destination network (Figure (3.. Fach color corresponds to one of the 10 target images (x)). The 10 stars in each plot correspond to the t-SNE embedding the destination-network representation of each target æ. If the cessinattionnetworr representation of each y associated with a particular color is near to its corresponding star, the representation transferability high. When representation transferability is low, the destination- network representations of pereesenttttio-taageeeee adversarial examples that target the same image will dissimilar, and thus not appear grouped Figure 3 By contrast, when representation targettd adversarial examples transfer successfully, we observe clusters grouped by target image (in Figure 3 by color). The visual tightness of each cluster represents the similarity between the representations associated with each point and its neighbors. As second similarity metric, we report the mean cosine similarity of the representations (in destination networks) between (Ir, y) pairs, averaged over all 9900 such pairs for each source network (Table The higher the mean cosine similarity, the higher degree of representation transfcrability from the source network to the corresponding destination network.
Representation Transferability Is Poor Non-Robust Networks. Our first surprising result is that representatiion-tareee adversarial examples generated with standard (i.e., non-robust) networks do not have substantial representation transferability to most tested destination networks (s columns of Figure and Table 2). This suggests that even when the adversarial classification output
8

---

is successfully transferred, the individual features of the destination networks are not substantially perturbed, suggesting that transferability from non-robust source networks arises from only a slight overlap in features, or the overlap of only a few highly vulnerable features. The result implies that the features of non-robust networks may not overlap substantially with each other. Interestingly, we observed some degree of representation transferability of adversarial examples generated with non-robust networks to 12nseNet121, which may explain the high degree tlasstrransfrrabiiity to DenseNet121 observed in Figure
Niightl-Robust Networks Exhibit Good Representation Transferability. We find that pnresentatiiontargetee adversarial examples generated with slighty-robuss networks (approximately 0.03 S E S 1) have high degree of representation transferability (i.e., clusters are tight and cosine similarity is high for these values in Figure P and Table 2. This representation transferability appears to peak, for convolutional networks, approximately when 0.1 < E S 0.5, which is coincident with the optimal source robustness for class-targeted transferability (Figure []. For the transformer networks, including VIT and CLIP, representation transferability peaks when 0.5 S = S 1, which, similarly, occurs close to the optimal robustness parameter for class-targeted transferabilityy Surprisingly, representation transferability from slightly-robust ResNer50 classifiers CLIP was high, despite the fact that CLIP is not trained on the traditional classification problem and is instead trained to encode images and corresponding textual description similarly. The high degree of representation transferability of slightly-robust networks likely explains the effectiveness of slightly-robust networks for generating class-targeted adversarial examples. More broadly, the high degree of representation transferability suggests that the features of slightly-robust networks overlap substantially with the features of every tested (non-robust) destination network, which the claim of Springer et al. [7]]
Representation Transferability in Very Robust Networks is Poor. Interestingly, when networks are ayeessarilll-ttrannee with large parameter, the degree to which the features they learn overlap with the features of non-robust networks diminishes as E increases. We speculate that certain non- robust features are ignored by robust neural networks with a sufficiently large parameter, and thus as robustness increases past point, many of the features of non-robust networks are ignored and representation transferability decreases.
Related Work
The vulnerability and defense of neural networks adversarial examples have been studied exten- sively [ [1
19
21 27 37, 45
49
50
55
55
63,
64
[72 73 79 83
First proposed by Goodfellow al. (11]. transferable adversarial examples generally constructed
by optimizing a perturbation to fool white-box (source) network with hopes that it will transfer to black-box (destination) networks. Recent transferability research has focused improving the
optimizer to add generalization priors [40] 54 88 [2]: researchers have added momentum to the
gradient descent process [13), introduced transformations to the input [14 67 87|. modified the adversarial loss functions
9I] attacked intermediate feature representations 31 35 60, 92]
linearizing the source network 233,, and additional manipulations the gradient computation [85] Additionally, some research has examined the role of source classifier(s): many of the aforementioned papers test the difference between source classifier architectures and researchers have proposed using generative networks 770 90). and ensemble attacks [4 (77).
Robust networks have been shown have number of valuable properties, including serving as a good starting point for transfer learning 43
80 and gradient interpretability [8]] The vulnerable features learned by neural networks have been studied both empirically [4] 15
62
76
20 32 36 46 |2 52, 66) 7 82 84] [89 and theoretically [1/2 12 28 65 8 [6). Similarly, there has been
some research related to the so-called "universality" hypothesis [3 42 53 58). which speculates
that under the right conditions, all neural networks may learn analogous features.
In prior work, the similarity of the feature spaces of different neural networks has been compared via linear transformations; however, these techniques are either anecdotal or do not account for possible non-linear relationship between neural networks (38 42 58)
Our research draws inspiration from recent work that proposes that slightly-robust features exhibit the universality principle, and presents a limited experiment that slightly robust networks

---

can be used to increase transferabilitty. By contrast, our work provides comprehensive study demonstrating that the technique can be used to achieve oftte-o-thheaan transferability across wide variety of architectures.
Conclusion
We have demonstrated that slightly-robust networks learn features that can be exploited to construct highly transferable targeted adversarial examples. These adversarial examples, constructed with convolutional networks (ResNets), can attack other convolutional networks with state-of-the-art performance, well networks with substantially different architectures, such as the transformer- based networks including ViT. Le and CCT, and with different learning objective, such as CLIP. In fact, this work is the first we are aware of that constructs targeted transferable attacks against transformmerbasel networks. We propose that the class-targeted transferability of adversarial examples generated with slightly-robust networks can be explained by the analogous representation- transferability of the networks. We find this to be true by showing that representation-ttageeee adversarial attacks generated with slight networks are highly transferable. Furthermore, since most previous transferable adversarial generation techniques rely on optimizing adversarial examples over non-robust source network, our technique can be combined with virtually any previously existing optimization technique by replacing the non-robust source network with a slightly- robust network.
As discussed, our paper is important to the field of adversarial machine learning, as we improve transfer attacks and study the mechanism of adversarial examples. More generally, our paper reveals phenomenon that is significant for the broader field of deep learning: we find that different non-robust networks, even when trained with similar convolutional architectures, do not necessarily have many features that substantially overlap. This can have important implications for the reliability of neural networks; when different networks rely on different features, they susceptible different types of errors. In addition, we present an argument that, for a given task, there are features that are useful every tested neural network, and that these features can be learned with small-s adversarial training, even when the source network architecture and learning objective are dissimilar to those of the destination network. Thus, by studying the features of single slightly-rouust network, we can empirically discover properties that will be applicable across all non-robust networks. We speculate that this phenomenon can explain why slightly-robust networks are successful transfer-learning tasks [43 62. 76. [0). With applications across the field of machine learning, we expect that the contributions in this paper will provide an important stepping stone toward discovering a general understanding of the features learned by neural networks.
Of course, our research has potential negative implications. Firstly, propose method to improve the generation of targeted transferable adversarial examples. While we hope that our research leads to the development of more robust and interpretable machine learning systems, in principle, an adversary could use our technique to attack existing systems. Secondly, we advocate for the use of adversarial training, which can be computationally intensive and could lead to excessive energy consumption.
Acknowledgments and Disclosure of Funding
The authors would like to thank Rory Soiffer, Juston Moore, and Hadyn Jones for their helpful discussions and comments.
Research presented in this article was supported by the Laboratory Directed Research and Develop- ment program of Los Alamos National Laboratory under project number 20210043DR.
Melanie Mitchell's contributions were supported by the National Science Foundation under Grant No. 2020103. Any opinions, findings, and conclusions recommendations expressed in this material those of the author and do not necessarily reflect the views of the National Science Foundation.
References
[1] 7. Allen-Zhu and Li. Feature purification: How adversarial training performs robust deep learning, arXiv preprint arXiv:2005.10190 2020.
10

---

[2] D. Arpit, S. Jastrzębski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj. A. Fischer, A. Courville, Y. Bengio, al. A closer look at memorization deep networks. arXiv preprint arXiv: 1706.05394 2017.
[3] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. In International conference on machine learning, pages 284-293. PMLR, 2018.
[4] M. Aubry and B. C. Russell. Understanding deep features with computer- generated imagery. In Proceedings of the IEEE International Conference on Computer Vision, pages 2875-2883, 2015.
[5] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šmdić, P. Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. In Joint European conference machine learning and knowledge discovery in databases, pages 387-402. Springer, 2013.
[6] N. Carlini and D. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 3-14. 2017.
[7] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privucy (sp), pages 39-57. IEFE, 2017.
[8] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, and A. Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705, 2019.
[9] F. Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 1251-1258, 2017.
[10] F. Chollet et al. Keras.
io. 2015.
[11] Cohen, Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smooth- ing. In International Conference on Machine Learning, pages 1310-1320. PMLR, 2019.
[12] G. De Palma, B. Kiani, and S. Lloyd. Random deep neural networks are biased towards simple functions. In Advances in Neural Information Processing Systems, pages 1964-1976, 2019.
[13] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li. Boosting adversarial attacks with momentum, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9185-9193, 2018.
[14] Y. Dong, T. Pang. H. Su, and J. Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4312-4322 4321, 2019.
[15] A. Dosovitskiy and T. Brox. Inverting visual representations with convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4829-4837, 2016.
[16] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 19999, 2020.
[17] L. Engstrom, A. Ilyas, H. Salman, S. Santurkar, and D. Tsipras. Robustness (python library), 2019. URL httpstts::t. //github. com///aarryyaal/rrruuuessee
[18] I. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, B. Tran, and A. Madry. Adversarial robustness as prior for learned representations. arXiv preprint arXiv: 1906.00945 2019.
[19] R. Feinman, R. R. Curtin. S. Shintre, and A. B. Gardner. Detecting adversarial samples from artifacts. arXiv preprint arXiv: /003.007 2017.
[20] R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. arXiv preprint 07X7v:2004.07780, 2020.
11

---

[21] J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1PXiv:1412.6572 2014.
[22] B. Graham, A. El-Nouby, H. Touvron, Stock, A. Joulin, H. Jégou, and M. Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2104.01136 2021.
[23] Y. Guo, Q. Li, and H. Chen. Backpropagating linearly improves transferability of adversarial examples. arXiv preprint arXiv:2012.03358, 2020.
[24] A. Hassani, S. Walton, N. Shah, A. Abuduweili, J. Li, and H. Shi. Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.05704 2021.
(25] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[26] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016.
[27] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. arXiv preprint arXiv: 1507.0770,7774 2019.
[28] K. Hermann, T. Chen, and S. Kornblith. The origins and prevalence of texture bias in convolu- tional neural networks. Advances in Neural Information Processing Systems, 33, 2020.
[29] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 7704.04841, 2017.
[30] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 4708, 2017.
[3]] Q. Huang. I. Katsman, H. He. Z. Gu, S. Belongie, and S.-N. Lim. Enhancing adversarial example transferability with an intermediate level attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019.
[32] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pages 125-136, 2019.
[33] N. Inkawhich, W. Wen, H. H. Li, and Y. Chen. Feature space perturbations yield more transferable adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7066-7074, 2019.
[34] N. Inkawhich, J. J. Liang, L. Carin, and Y. Chen. Transferable perturbations of deep feature distributions. arXiv preprint arXiv:2004.12519, 2020.
[35] N. Inkawhich, K. J. Liang, B. Wang, M. Inkawhich, L. Carin, and Y. Chen. Perturbing across the feature hierarchy to improve standard and strict blackbox attack transferability. arXiv preprint arXiv:2004. 14861, 2020.
[36] Jo and Y. Bengio. Measuring the tendency of enns to learn surface statistical regularities. arXiv preprini 1711
[37] S. Kaur, J. Cohen, and Z. C. Lipton. Are serceptually-aligned gradients a general property of robust classifiers? arXiv preprint arXiv: 1et.::910000..0 08640, 2019.
[38] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529, PMLR, 2019.
[39] A. Krizhevsky, G. Hinton, et Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
12

---

[40] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv preprint arXiv: 02300, 2016.
[41] M. Li, C. Deng, T. J. J. Yan, X. Gao, and H. Huang. Towards transferable targeted attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 641-649, 2020.
[42] Y. Li, Yosinski, J. Clune, H. Lipson, and E. Hoperoft. Convergent learning: Do different neural networks learn the same representations? In FE@ NIPS, pages 196-212, 2015.
[43] K. Liang, J. Y. Zhang, O. Koyejo, and B. Li. Does adversarial transferability indicate knowledge transferability? arXiv preprini arXiv:2006. /4512, 2020.
[44] Y. Liu, X. Chen, C. Liu, and Song. Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv: 1.02770, 2016.
[45] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 2017.
[46] A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1188-5196, 2015.
[47] R. T. McCoy, E. Pavlick, and T. Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXIv:1902201000,2001 2019.
[48] L. Melas Myriazi
Vit 2020.
pytorch.
https://githhb..com/luueee
PyTorch- PyTorch-Pretrained ViT
[49] J. H. Metzen, T. Genewein, V. Fischer, and B. Bischoff. On detecting adversarial perturbations. arXiv preprint arXiv:1702.04006 2017.
[50] S.-M. Deesfoll,,,,,,,,,, A. Fawzi, and P. Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574-2582, 2016.
[51] Nakkiran, D. Kalimeris, G. Kaplun, B. Edelman, T. Yang, B. Barak, and H. Zhang. Sgd on neural networks learns functions of increasing complexity. In Advances in Neural Information Processing Systems, pages 3496-3506, 2019.
[52] C. Olab, A. Mordvintsev, and L.. Schubert. Feature visualization. Distill, 2(1 1 1):77, 2017.
Carter. [53] C. Olah, N. Cammarata, I.. Schubert, G. Goh, M. Petrov, S.Crter Zoom circuits. Distill, 2020. doi:
in: An introduction http://dissilluubb2222//cccuuui
in.
[54] N. Papernot, P. McDaniel, and I. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint 17X1v:1605.07277, 2016.
[55] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami. The limitations of deep learning in adversarial settings. 2016 IEEE European symposium on security and privacy (EuroS&P), pages 372-387. IEEE, 2016.
[56] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. Veto,, M. Raison, A. Tejani, Chhlammkurthy, B. Steiner, L. Fang, Bai, and S. Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http: www.:eeeur...pcppeeeee -iiaaninnn--barryy library
an
pdf
13

---

[57] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. arXiv preprini arXiv: 103.00000, 2021.
[58] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl- Dickstein. Sveca: Singular vector canonical corre- lation analysis for deep learning dynamics and interpretability. arXiv preprint 170.v:1006.0555 2017.
[59] A. Raghunathan, J. Steinhardt, and P. Liang. Certified defenses against adversarial examples. arxiv preprint arXiv: 1801. .09444, 2018
[60] A. Rozsa, M. Günther, and T. Boult. Lots about attacking deep features. In 20/7 IEEE International Joint Conference on Biometrics (IJCB), pages 168-176. IEEE, 2017.
[61] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L.. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), (-5(3):211-252 2015. doi: 100077/11263-01--000
[62] H. Salman, A. Ilyas, Engstrom, A. Kapoor, and A. Madry. Do adversarially robust imagenet models transfer better? arXiv preprini 2007.0000 2020.
[63] S. Santurkar, A. llyas, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Image synthesis with a single (robust) classifier. In Advances Neural Information Processing Systems, pages 1262-1273, 2019.
[64] A. Shafahi, W. R. Huang, C. Studer, S. Feizi, and T. Goldstein. Are adversarial examples inevitable? arXiv preprint arXiv:/809.02 104, 2018.
[65] H. Shah, K. Tamuly, A. Raghunathan, P. Jain, and P. Netrapalli. The pitfalls of simplicity bias in neural networks. arXiv preprint arXi::200.007110, 2020.
[66] R. Shao, Z. Shi, J. Yi, P.-Y. Chen, and C.-J. Hsieh. On the adversarial robustness of visual transformers. arXiv preprint arXiv: 15670, 2021.
[67] Y. Sharma, G. W. Ding, and M. Brubaker. On the effectiveness of low frequency perturbations. arXiv preprint arXiv: 1903300073 2019.
[68] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[69] K. Simonyan, Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: 2013.
[70] Y. Song, R. Shu, N. Kushman, and S. Ermon. Constructing unrestricted adversarial examples with generative models. arXiv preprint 180Xiv/805.000 00884, 2018
[71] M. Springer, M. Mitchell, and G. T. Kenyon. Adversarial perturbations are not so weird: Entanglement of robust and non-robust features in neural network classifiers. arXiv preprint arXiv:2102.05/100 10, 2021.
[72] D. Stutz, M. Hein, and B. Schiele. Disentangling adversarial robustness and generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6976-6987, 2019.
[73] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intrigu- ing properties of neural networks. In 2nd International Conference Learning Representations, ICLR 2014, 2014.
(74) C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z.. Wojna. Rethinking the inception archi- tecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[75] M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference Machine Learning, pages 6105-6114, PMLR, 2019.
14

---

[76] M. Terzi, A. Achille, M. Maggipinto, and G. A. Susto. Adversarial training reduces information and improves transferability. arXiv preprint arXtv:2007.11259, 2020.
[77] F. Tramèr, D. Boneh, A. Kurakin, I. Goodfellow, N. Papernot, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. In 6th International Conference on Learning Representations, ICLR 2018- 101-feoneeee Track Proceedings, 2018.
[78] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. In International Conference Learning Representations, 2019.
[79] J. Uesato, B. O' O'donoghue, P. Kohli, and A. Oord. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning, pages 5025-500 5034. PMLR, 2018.
[80] Utrera, E. Kravitz, N. B. Erichson, R. Khanna, and M. W. Mahoney. Adversarially-trained deep nets transfer better. arXiv preprint arXiv:2007.05860 2020.
[81] G. Valle-Pérez, C. Q. Camargo, and A. A. Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions. arXiv preprint arXiv:1805.08522, 2018.
[82] H. Wang. Wu, Z. Huang, and P. Xing. figh-freeuency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8684 8694, 2020.
[83] D. Warde-Farley and I. Goodfellow. Adversarial perturbations of deep neural networks. Pertur- bations, Optimization, and Statistics, 311, 2016.
[84] K.A. A. Wei. Understanding non-robust features in image classification. PhD thesis, Mas- sachusetts Institute of Technology, 2020.
[85] D. Wu, Wang, S.-T. Xia, J. Bailey, and X. Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. arXiv preprint arXiv:2002.05999, 2020.
[86] L.. Wu, Z. Zhu, et Towards understanding generalization of deep learning: loss landscapes. arXiv preprint arXiv:
Perspective of
2017.
[87] C. Xie, 7. Zhang, Y. Zhou, S. Bai, J. Wang, Z. Ren, and A. L. Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2730-2739, 2019.
[88] Z. Yang. L. L. Xu, S. Zuo, Q. Chen, Rubinstein, C. Zhang, and B. Li. Trs: Transferability reduced ensemble via encouraging gradient diversity and model smoothness. arXiv preprint arXiv:2104.0m 00671, 2021.
[89] Q-s. Zhang and S.-C. Zhu. Visual interpretability for deep learning: survey, Frontiers of Information Technology & Electronic Engineering, 19(1):27-39, 2018.
[90] Z. Zhao, D. Dua, and S. Singh. Generating natural adversarial examples. arXiv preprint arXiv: 11342, 2017.
[91] Z. Zhao, Z. Liu, and M. Larson. On success and simplicity: A second look at transferable targeted attacks. arXiv preprint arXi::2012.11207, 2020.
[92] W. Zhou, X. Hou, Y. Chen, M. Tang, X. Huang, X. Gan, and Y. Yang. Transferable adversarial perturbations. In Proceedings of the European Conference Computer Vision (ECCV), pages 452 462-467,22 2018
[93] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018
15


Reviews:
[{"review_id": "yLX2rlnApm1", "reviewer": "Reviewer_gLyV", "summary": "The paper demonstrates that strong transferability across models can be achieved by using slightly-robust source networks through a comprehensive study and few analyses.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n- Originality: \n \n  * This work seems to be a follow up study of [67] with a more comprehensive study, including analysis on CNN and transformer architectures, different hyperparameters, and representation similarity analysis etc. The novelty seems a bit limited.\n\n- Clarity:\n  * I found the paper a bit hard to read. \n    * There are several terms that are not precise or confusing to me, e.g., transferability, universal, $\\epsilon$-robustness, slightly-robust, and overlap. I found it a bit confusing in the context of adversarial robustness as it could be unclear whether those terms are for attackers or defenders.\n    * There are also several typos: e.g., L157 fined -> find. L266 testset -> test set.\n    * I also found it confusing that there are several sentences using future tense. It is not clear to me when it is going to happen. \n    * I also found pages 7--8 a bit hard to read as there are not enough structures  but with many long paragraphs. \n\n\n- Quality:\n  * The paper looks at 16/255 perturbation for imagenet models. As shown in [1], 16/255 is a very large perturbation radius and it could potentially yield visible class changes.\n  * I am not sure about the previous literature on transfer attack. I found it a bit suspicious to rely on FGSM type of attacks only. I wonder whether autoattack or other pgd types of methods [2] could more reliably justify the robustness of the transfer models. \n\n- Significance:\n  * I found the results and findings are interesting to know but it is mostly empirical. There is not enough explanation on why this phenomena happens, how this could be useful for defending transfer attacks, etc. The conclusion seems to expand too broad. \n\n[1] Daniel Zoran, et al.; Towards Robust Image Classification Using Sequential Attention Models, CVPR 2020, pp. 9483-9492\n[2] https://arxiv.org/abs/1910.09338", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "T3L0lsd8wxB", "reviewer": "Reviewer_QJSC", "summary": "This work studies how adversarial robustness of a source network can affect the transfer of adversarial examples to a different target network. The main finding is that slightly robust networks are ideal source networks; training the source networks with no adversarial perturbations, or with large perturbations, is less effective. The authors analyze why this might be the case, and propose that slightly robust networks have increased universality of representations.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- This fits the story of adversarially trained networks transferring better to downstream tasks (a different kind of transfer).\n- The result that highly robust networks don't work as well is surprising and interesting\n- The writing is engaging and clear\n- The main experiments are well-designed\n\nCons:\nSection 4 is circular. Consider the following snippets.\n- “Universality Explains Targeted Transferability” (the main claim)\n- “When a neural network has substantial representation transferability to every other network of a certain type, we say that the neural network has features that are universal with respect to these destination networks” (definition of universality)\n- “We refer to the degree to which adversarial examples of a source model can analogously affect the features that are computed by the representation layer of the destination model as the representation transferability from source to destination.” (definition of representation transferability)\n\nIf you follow the logic, this is basically saying that the degree to which adversarial examples can perturb features in a bunch of destination models explains targeted transferability, which is tautological. This is nitpicking, but it does matter for clarity. Basically, it's currently unclear what section 4 adds. Indeed, Table 2 feels like the same thing as Table 1, except it shows cosine similarity instead of attack success rate.\n\nI /think/ the authors were going for universality meaning that representations are similar, separate from adversarial transferability. If this is the case, then there are a number of ways to measure the similarity of two networks. If the authors could show that slightly robust networks have representations that are closer to all the destination networks according to an independent metric, that would give the analysis more internal validity.\n\nTypos:\n- 126: “using linear interpolation” → “using bilinear interpolation”\n\nOverall, the main results are interesting, but the issues in section 4 need to be addressed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ScrfPNYZpVA", "reviewer": "Reviewer_ZTym", "summary": "This paper proposes that slightly robust neural networks contain universal features that are transferable across different classifiers trained on the same dataset. The authors demonstrate empirically that adversarial examples of slightly robust networks can transfer across network architectures relative to other levels of robustness. Moreover, the authors empirically show the universality of features in slightly robust neural networks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\nThe main idea of transferability in slightly robust neural networks appears mostly novel: as noted by the authors, [67] proposes the idea that slightly robust networks are transferable. However, [67] primarily focuses on showing the entanglement of non-robust and robust features- by contrast this work focuses primarily on transferability of slightly robust neural networks. The extensive experiments provide new insights and evidence beyond what is reported in previous literature. Specifically, the experiments reveal that the internal representations of slightly robust networks are more transferable, while prior works only considered transferability at the classifier output.\n\n**Quality**\nExperiments are generally extensive and convincing. However, one question that is inadequately addressed is what the significance of targeted vs untargeted attacks is with respect to transferability. Can the authors provide an explanation for why targeted attacks are more difficult to transfer?\n\n**Clarity**\nThe paper is generally clearly written and well organized. However, some of the links with prior works are not made very clear. In particular, the paper discusses transferability of non-robust features, which is a concept also discussed in [30], for example. The authors may want to further discuss the nature of features in slightly robust networks in terms of being robust vs non-robust features, and highlight how their contributions differ from prior findings.\n\n\n**Significance**\nIf the authors can provide answers to some unanswered questions as noted above, this work could provide significant insight into the nature of transferable features in neural networks. This may lead to the development of better adversarial attacks and defenses.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately describe the limitations of the work in section 6.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "OREk_dd1Ot_", "reviewer": "Reviewer_YRoB", "summary": "This paper studies the targeted adversarial examples. They show that making the source classifier just a little robust to small-magnitude adversarial examples can improve the transferability of targeted attacks even between CNNs and transformers.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\n\nThough there exists a plethora of papers studying the transfer attacks, this paper provides a rather interesting perspective. The observation that making the source classifier just a little robust to small-magnitude adversarial examples can improve the transferability of targeted attacks even between CNNs and transformers is quite novel. This is motivated by the previous work (https://arxiv.org/abs/2102.05110) showing that slightly-robust features exhibit the universality principle and that slightly-robust networks can be used to increase transferability. This NeurIPS submission further provides a more comprehensive study investigating the transferability across a variety of architectures. \n\nIn summary, the originality is not about the idea of using slightly robust features, but the comprehensive study and experiments, which provides a rather solid and novel perspective on more effective targeted transfer attacks. \n\n## Quality\n\nThe hypothesis that making the source classifier just a little robust to small-magnitude adversarial examples can improve the transferability of targeted attacks is reasonable and the empirical (both qualitative and quantitative) experiments support it. \n\nI quite enjoy Section 4 which tries to explain why class-targeted adversarial examples transfer via the lens of universality. They measure the similarity between the representations produced by different networks. \n\nHowever, I'm not fully convinced the conclusion regarding transformer classifiers is correct, as the authors only tried one particular transformer, ViT. One example is https://arxiv.org/abs/2104.05704, and the code can be found at https://github.com/lucidrains/vit-pytorch. Also, it might be more informative if they can provide the number of parameters for each model in Table 1. \n\n## Clarity\n\nThe paper is well-written and very easy to follow. \n\nOne suggestion: the authors can indicate in the main submission that TMDI-FGSM is described in the appendix. \n\n## Significance\n\nAlthough the universality principle has been introduced and utilized for training slightly-robust networks before, this submission provides more comprehensive experiments and analysis showing that they can be used for effective targeted transfer attacks.  This is a solid contribution to the study of adversarial attacks and many researchers interested in adversarial examples will find it insightful and useful. However, the tarted audience is also thus limited. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitation and potential negative societal impact in Section 6. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper observes that classifiers with a certain degree of robustness lead to more transferable (targeted) adversarial examples. The manuscript shows how features learned by robust classifiers transfer better to other classifiers. This provides some further insights (building on https://arxiv.org/abs/2102.05110) into transferability, a phenomenon which remains poorly understood in the adversarial ML community. Thus, there is merit to the work proposed here. I encourage the authors to take into account discussions from the reviews while preparing the camera ready of their manuscript, and to open-source their code. In particular, the authors acknowledged that their discussion of universality is hard to follow, if not a bit circular, so I recommend following suggestions made in the author response to rewrite Section 4. Finally, the authors conducted their experiments using a single optimizer (see line 71), I would encourage them to confirm their findings on additional optimizers to obtain a more complete result.
Decision: Accept (Poster)