Learning Transferable Adversarial Perturbations
Year: 2021

Abstract:
While effective, deep neural networks (DNNs) are vulnerable to adversarial attacks. In particular, recent work has shown that such attacks could be generated by another deep network, leading to significant speedups over optimization-based perturbations. However, the ability of such generative methods to generalize to different test-time situations has not been systematically studied. In this paper, we therefore investigate the transferability of generated perturbations when the conditions at inference time differ from the training ones in terms of target architecture, target data, and target task. Specifically, we identify the mid-level features extracted by the intermediate layers of DNNs as common ground across different architectures, datasets, and tasks. This lets us introduce a loss function based on such mid-level features to learn an effective, transferable perturbation generator. Our experiments demonstrate that our approach outperforms the state-of-the-art universal and transferable attack strategies.

heading: 1 Introduction
text: In recent years, deep neural networks (DNNs) have achieved great success in a wide range of applications [1, 2, 3, 4]. However, DNNs have been demonstrated to be vulnerable to adversarial examples [5] crafted by adding imperceptible perturbations to clean images. In particular, two broad categories of attacks have been studied. The first one consists of iterative algorithms [5, 6, 7], which optimize the perturbation for each instance, and thus tend to be computationally expensive. The second one encompasses generative methods [8, 9, 10], which train a deep network to produce perturbations. As such, attacking the target network only involves a forward pass through the generator, typically resulting in much faster attacks than iterative methods. However, speed is not the only important factor to assess the strength of an attacker; its ability to generalize to different situations is also key to its success.
In this paper, we therefore study the transferability of perturbations obtained with generative methods. Specifically, we investigate the transfer of such perturbations when the conditions at inference time differ from the training ones in terms of (i) target architecture, e.g., the generator was trained to attack a VGG-16 but the target network is a ResNet152; (ii) target data, e.g., the generator was trained using the Paintings dataset but the test data comes from ImageNet; (iii) target task, e.g., the generator was trained to attack an image recognition model but faces an object detector at test time. To the best of our knowledge, for generative methods, our work constitutes the first attempt at transferability across tasks, and only [11] has studied generalization across architectures and data, by introducing a loss function acting on the relative class probabilities of the attacked and unattacked examples.
Here, by contrast, we improve the transferability of a perturbation generator across architectures, data, and tasks by exploiting the mid-level features of DNNs. The key motivation behind our approach is our observation that the mid-level features extracted by DNNs with different architectures, different data, or for different tasks bear strong similarities. This is illustrated in Figure 1, where we visualize the features extracted by different backbones and for different tasks, and thus different datasets, using
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
the method of [12]. Our analysis suggests that adversarial perturbations that significantly affect the mid-level features of one sample in one architecture also affect them in a different architecture, even for a different task and with different data.
We therefore propose to train a perturbation generator by maximizing the distance between the normal features of a sample and their adversarial counterparts extracted in the intermediate layers of a pretrained classifier. The resulting perturbations are then transferable across architectures, datasets and tasks because they similarly affect the mid-level features of the corresponding filters in the target setup. The perturbed mid-level features are then propagated to the top layers of the network, and, as a result, lead to incorrect predictions.
Contributions. Our contributions can be summarized as follows: 1. We identify the intermediate features of CNNs as common ground across different architectures, different data distributions and different tasks. 2. We introduce an approach that exploits such features to learn an effective, transferable perturbation generator. 3. We systematically investigate the effect of target architecture and target data distribution on the transferability of adversarial attacks. Our experiments demonstrate that our approach yields higher fooling rates than the state-of-the-art universal [8] and transferable [11] attacks. Our code is available at https://github.com/krishnakanthnakka/Transferable_ Perturbations.

heading: 2 Related Work
text: Below, we review the recent works on adversarial attacks, with a focus on generator-based approaches.
Adversarial Attacks. Adversarial attacks were first investigated in [5] to identify the vulnerability of modern deep networks to imperceptible perturbations in the context of image classification. Since then, several attack strategies have been studied, including single-step fast gradient descent [6, 13], and computationally more expensive optimization-based attacks, such as CW [14], JSMA [15], and others [16, 17, 7].
While the above methods are image-dependent, the existence of Universal Adversarial Perturbation (UAP) was first shown in [18], considering the task of learning a single perturbation that can fool a classifier independently of the input image. Such a UAP was iteratively updated based on its effectiveness to move the individual data samples across the decision boundary. Other UAPs based on the computation of the singular vectors of the Jacobian matrices of feature maps have been studied
in [19]. In parallel to UAPs, several works have shown that iterative adversarial attack strategies could be transferred across architectures [20, 21, 22]. In particular, the recent work of [23] proposed to improve targeted attack transferability across architectures by training auxiliary classifiers at each intermediate layer to attack a CNN’s feature maps. The above-mentioned methods, however, either are computationally expensive at inference time [16, 7, 15, 14, 21, 23], or suffer from low transferability rates [18, 20, 22]. Furthermore, most of the attacks aiming for transferability [18, 19, 23] strongly depend on the availability of data from the target domain. By contrast, we introduce an efficient generative method to produce adversarial perturbations that generalize not only across architectures, but also across training data and across tasks.
Generator-based Attacks. Generative adversarial perturbations (GAP) were first introduced in [10]. In particular, [10] showed that a generator network can be used to craft a UAP that transform an input image to an image-dependent perturbation. Similarly, [24] introduced the advGAN generative framework to learn to produce adversarial perturbations, and further proposed to make use of distillation to perform black-box attacks. Based on the observation that the effectiveness of these two methods strongly depends on the availability of data from the target domain during training, [11] introduced the relativistic cross-entropy loss, which was shown to better generalize across datasets. Furthermore, [8] proposed to generate UAPs by transforming the generator’s output using a regional norm layer that enforces perturbation homogeneity. While effective, the above-mentioned works explicitly rely on the classification boundary of the attacked model, which we will show to tend to make them overfit to the source data. By contrast, we identify the mid-level features as a more robust signal shared across not only different architectures and datasets, but even across different tasks. Our experiments will showcase the superiority of exploiting this information over the losses used in previous approaches to train a transferrable perturbation generator.

heading: 3 Methodolgy
text: Let xi ∈ RH×W×3 be a color image of size H ×W , and yi be the associated ground-truth label. Furthermore, let f denote the task-related convolutional neural network that takes x as input and extracts, at layer l = {lj}Lj=1, a feature map f`(xi) ∈ RN`×D` , which we assume to be reshaped in matrix form, such that N` = H` ×W`, with H` and W` the spatial dimensions of the feature map, and D` its number of channels.
Our goal is to train a generator G that produces a perturbation δi ∈ RH×W×3, which, when added to the clean image xi, results in predicting a different label from yi. To this end, we feed the input image xi to the generator to synthesize an unbounded adversarial image G(xi), which is then clipped to be within an bound of xi under the `∞ norm. Let x̂i be the adversarial image obtained after such a clipping of G(xi). In contrast to [10, 11], which rely on the final classification boundary of the task-related network f to train the generator with a cross-entropy-based loss, we exploit the mid-level features of f . Specifically, we maximize the L2 distance between the normal feature map f`(xi) and adversarial feature map f`(x̂i) at layer l using a feature separation loss term defined as
Lfeat(xi, x̂i) = ||f`(xi)− f`(x̂i)||2F , (1)
where ‖ · ‖F denotes the Frobenius norm. The overall training scheme of our perturbation generator is provided in Algorithm 1. In practice, we observe the feature separation loss of Eq. 1 to encourage the generator to learn low-level, quasiimperceptible structured patterns that affect that activations of a few filters while either suppressing the other ones or leaving them unaffected. More importantly, these perturbations are highly transferable.
Reason for transferability. Given the simplicity of our formulation, a natural question that arises is what affects the transferability of perturbations among CNNs. To understand this, we analyze the internal workings of a CNN, whose filters in each layer encode different levels of information. In particular, in Figure 2, we visualize the filters of various CNNs, sampled at different layers, using the technique of [12], which maximizes the mean activation of the each filter. This allows us to discover common characteristics across the different architectures. On one hand, the bottom filters, close to the input image, extract color and edge information, as shown in the left block of Figure 2. On the other, the top-level filters, close to the output layer, shown in the right block, are more focused on the object representation, and thus more task specific. By contrast, the mid-level filters learn more nuanced features, such as textures, and therefore tend to display similar patterns across architectures,
datasets tasks. As a consequence, and as indicated by our experiments, attacking the initial layers requires large perturbation strengths. Furthermore, while attacking the top-level features works well in the white box scenario, it does not transfer well to different data or architectures, because the features are already too architecture- and task-specific. Attacking mid-level features thus comes as a natural choice, which we will show to yield highly transferable perturbations.
Let us now focus on understanding the reason for the transferability when the target data is unavailable to the attacker. In Figure 3, we plot t-SNE visualizations of different datasets at 4 layers of DenseNet121. The proximity of the features extracted from the Comics or Paintings datasets, containing images from a few classes particularly focused on humans, to those extracted from ImageNet suggests these datasets as good candidates to train a generator that will fool an ImageNet-trained network (as shown in Tables 1 and 2). By contrast, the ChestX dataset presents a larger domain gap. Nevertheless, we will show that, albeit for a drop in fooling rate, our use of mid-level features still make a ChestX-trained generator reasonably effective on ImageNet-based classifiers. Furthermore, in Figure 4, we plot the top 80 activations of ImageNet, Paintings and ChestX at layer 8 of DenseNet121. This shows that the peaks of the ImageNet dataset (orange) are more strongly correlated to those of the Paintings dataset (left) than to those of ChestX (right). This is confirmed by our experiments, where a Paintings-trained generator is more effective than a ChestX-trained one. In short, while mid-level features lead to generator transferability, the resulting effectiveness remains affected by the similarity of the data.
Algorithm 1 Training a transferable adversarial perturbation generator Input: xi: clean images; f : pretrained classifer; : perturbation `∞ bound
Initialize generator G; Load classifier f and freeze its parameters; repeat
Get a clean image xi and feed xi to f to obtain f`(xi); Generate an unbounded adversarial image G(xi) and clip it within an bound of xi to get x̂i; Pass the adversarial image to f to obtain f`(x̂i); Compute the feature separation loss Lfeat; Compute the gradient of Lfeat w.r.t. the weights of G and update these weights using Adam.
until convergence return trained Generator G∗

heading: 4 Experiments
text: We evaluate the effectiveness of our attack strategy in diverse settings. Below, we first discuss our experimental setup and then compare our attacks with the state-of-the-art generator-based ones: CDA [11], RHP [8], and GAP [10]. To further demonstrate the generalizability of our approach, we also report its performance on adversarially-trained models and on the SSD [25] object detector with 4 different backbones. We discuss the experimental details in the supplementary material.
Datasets. To train the generator, similarly to [11], we use data from either ImageNet [26], Comics [27], Paintings [28] or ChestX [29] as source domain, containing 1.2M, 40K, 80K, and 8K images, respectively. We then randomly select 5000 images from the ImageNet [26] validation set as target domain to evaluate the transferability of our attacks. As in [11, 18, 10], we report the fooling rate and the absolute difference in top-1 error between before and after the attack. The fooling rate is the percentage of images for which the label is changed after the attack.
Models. For the target models, we make use of the publicly available PyTorch [30] versions of VGG16 [31], VGG-19 [31], ResNet152 [32], DenseNet121 [33], Inception-V3 [34] and Squeezenet [35] pretrained on ImageNet. We chose this family of networks to understand the deeper impact of transferability across diverse architectures. As in [11], to evaluate transfer across datasets, we use ChestXNet [29] pretrained on ChestX. For the perturbation generators, we use the same architectures as in [11, 10]. To train them, we use the Adam optimizer with a learning rate of 2e−4 and a batch size of 16.
Attack settings. We perform attacks in four settings: 1. White-box attacks, where the attacker has access to the exact target model and target data distribution; 2. Standard black-box attacks, where the attacker has access to a substitute model from a different family of architectures trained on the target data and to the target data itself; 3. Strict back-box attacks, where the attacker also uses a substitute model trained on the target data, but without having access to the target data itself; 4. Extreme black-box scenario, we perform attacks without any knowledge of the target model or the target data.

heading: 4.1 Transferability to Unknown Target Model
text: Let us first study the standard white box and black-box settings. In this experiment, we train the generator with target data containing 1.2M ImageNet training samples, using one of the 5 ImageNet pretrained models as target architecture. We then evaluate the transferability of the attack to the remaining 4 models.
Figure 5: Transferability within the same family. We report the fooling rates when transferring attacks to other networks within the same family. The generators were trained with DenseNet121 and ResNet152, with = 10.
Table 1 compares the effectiveness of our attacks to the CDA and GAP ones. We outperform these baselines on average by 10.5 and 13.5 percentage points (pp) (i.e., absolute difference of fooling rates), respectively. The differences are particularly pronounced when the generator is trained with SqueezeNet. Furthermore, in the DenseNet121 case, the difference between CDA and Ours is 13.6pp.
A similar trend can be seen when transferring attacks within the same family of networks. For instance, Figure 5 (a) shows the transferability when the generator is trained with DenseNet121 and evaluated on other networks of the same family, namely, DenseNet161, DenseNet169, and DenseNet201. Figure 5 (b) shows the results for similar experiments on the ResNet family. Our method performs on average 1.68pp, 6.4pp better than CDA and GAP in the case of ResNets, which demonstrates that our feature separation loss generalizes well for attacks within the same family. Importantly, the performance gap between Ours and the baselines increases with the difference in depth from training – from 0.02pp on DenseNet121 to 6.3pp on DenseNet201 w.r.t. GAP.
In Figure 6, we further study the transferability of attacks as the number of samples from the target data to train the generator varies. Our method (blue bar) consistently maintains higher fooling rates, even for a small number of training samples, than GAP (red bar) and CDA (yellow bar). Note that all methods tend to saturate when around 5K samples are available, with our method consistently outperforming the baselines.

heading: 4.2 Transferability to Unknown Target Data
text: Let us now turn to the strict black-box setting, where the target data is unavailable to the attacker, who then relies on a different dataset to train the generator. Note that, here, the attacker still has access to a substitute model trained on the target data, therefore we also study the impact of transferability to an unknown target model. Following [11], we consider two synthetic datasets, Comics and Paintings, depicting objects similar to the ImageNet ones, and one dataset, ChestX, containing a completely different type of data, thus suffering from a large domain gap.
Table 2 compares the results of our approach with those of the baselines. Our method yields clearly superior results for all 3 datasets. For instance, with Comics, Ours outperforms GAP and CDA on average by 14.9pp and 10.3pp, respectively, and with Paintings, by 14.9pp and 11.0pp, respectively. Note that, while we still outperform the baselines when using the ChestX data, by around 22pp on average, the fooling rates of all methods drop significantly in this challenging scenario. Nevertheless, altogether, these results evidence that one can learn to generate adversarial examples with high fooling rates despite not having access to the target data, even by using data from a completely
different domain. Furthermore, the resulting generator generalizes particularly well to the unseen target domain when trained with our feature separation loss.

heading: 4.3 Extreme Cross-Domain Transferability
text: We further study the transferability of adversarial attacks in the extreme scenario where neither the target architecture nor the target data are available, and where even the classifier used to learn the generator was trained on data that differs from the target domains. To this end, we use ChestXNet [29] pretrained on ChestX dataset as classifier. We train the generator with different datasets on denseblock12 and evaluate its transferability in Table 3. Despite the challenging nature of this setting, our method still yields satisfying fooling rate; 68.5pp on average vs 87.0pp in the standard black-box scenario. Furthermore, our approach outperforms the state-of-the-art CDA by 18.5% on average.

heading: 4.4 Transferability to Robust Models
text: In this section, we study the transferability of our generators to five state-of-the-art defenses, i.e., the high-level representation guided denoiser (HGD) [36], the input preprocessing defense through random resizing and padding (R&P) [37], Feature Denoising (FD) on ResNeXt-101 [38], Projected Gradient Descent (PGD) [39] on ResNet50, and the average of three ensembles of adversarially trained Inception models [40] (EnsembleAdv). In addition to GAP and CDA, we compare our method to RHP, which constitute the state-of-the-art universal attack in terms of effectiveness against defenses. As shown in Table 4, Ours achieves the overall best performance on HGD, R&P and EnsembleAdv. However, we observe that none of the methods are successful in attacking Feature denoising and PGD defenses. Note, however, that with PGD and FD, the error rate on the clean samples is significantly higher than with other defenses, making these strategies ill-suited for practical applications.
Nework Method EnsembleAdv [40] HGD [36] RP [37] FD [38] PGD [39] Error on normal samples 24.6 19.5 21.0 51.1 53.7
Inc-v3 RHP 29.5 26.8 23.3 2.38 2.40 Inc-v4 RHP 25.1 23.4 20.2 1.90 2.20 IncRes-v2 RHP 28.8 26.9 25.1 2.20 2.20 SqueezeNet GAP 27.2 33.1 26.7 4.24 5.84 SqueezeNet CDA 25.6 31.4 26.5 4.06 5.42 SqueezeNet Ours 33.7 43.9 32.7 7.06 3.30

heading: 4.5 Cross-Task Transferability Analysis
text: To demonstrate the transferability of our approach across tasks, we attack object detectors using perturbation generators trained with image classifiers. Specifically, we choose the SSD framework with 4 different backbones, namely, VGG16, ResNet50, EfficientNet, and MobileNet-v3 pretrained on PASCAL VOC [41]. In Table 5, we report the mAP on the PASCAL VOC test-set, containing 4952 images, for perturbation generators trained with different image classifiers and with = 16. The performance of the SSD detector on clean images is 68.1, 66.1, 61.1 and 55.4 using VGG16, ResNet50, EfficientNet and MobileNetv3 as backbone, respectively. Our attacks significantly decrease these score and yield lower mAPs than the baselines in most cases. These results highlight, for example, that a generator trained on the synthetic Comics dataset with an image classifier can fool an SSD detector trained on a different domain, with a different architecture, and for a different task. As evidenced by the comparisons with CDA and GAP, learning to generate perturbations that affect the mid-level features is more effective than focusing on the classification boundary.

heading: 4.6 Additional Analysis
text: To further analyze our method, we visualize the resulting perturbed images using t-SNE plots. Specifically, in Figure 7(a), we provide t-SNE plots obtained using the final latent representation of 1000 normal (blue) and attacked (yellow) ImageNet images for the strict black-box and extreme black-box scenarios. In both cases, our method yields features that clearly separate the normal and adversarial images with large margins, whereas CDA fails to do so. In Figure 8, we visualize the unbounded adversarial images obtained with different methods and also show the bounded adversarial images computed with = 10 with our approach in the last column. As can also be seen in Figure 9, the perturbations follow repeating patterns corresponding to the patterns of the most disrupted midlevel filters. Note that RHP [8] explicitly enforces perturbation homogeneity using a predefined pattern, such as vertical or horizontal patterns. Our filter visualizations for multiple networks reveals that the bottom layers learn such horizontal and vertical stripes. While we tried training the generator to perturb such layers, we observed that larger strength are required in this case. Overall, our results evidence that attacking mid-level filters that have strong correlations across architectures consistently improves the transfer rates.
Limitations. In our experiments, we choose a single layer l at a time. To this end, we sweep over each block of layers, typically around 5 on average for the studied architectures. We empirically found that the best-performing layer is independent of the choice of the target model to attack. For example, a generator trained with a feature separation loss on the conv4-1 layer in VGG16 transfers to all attack settings at inference time, avoiding repeated sweeps and computational costs. For all our experiments, we set the layer l to attack to relu after conv4-1, layer3, mixed6b, denseblock8, and fire10 for VGG16, ResNet152, Inception-v3, DenseNet121, and SqueezeNet, respectively. We report additional results with generators trained at different layers in the supplementary material. Nevertheless, we have no guarantees that such layers truly are optimal; in particular, better results might be achievable by attacking ensembles of layers, but this would require tuning the layer combination.

heading: 5 Conclusion
text: In this paper, we have explored the use of mid-level features in conjunction with generative networks to learn transferable perturbations. We have shown that a generator trained with feature separation loss can successfully fool models across architectures and tasks, and that a proxy dataset from a different domain can be leveraged to learn effective perturbations. Our experiments demonstrate that our method outperforms the state-of-the-art attacks across a wide range of architectures and tasks. We have limited our experiments to undefended models due to the unavailability of publicly available robust models across architectures but will study this in future work. Furthermore, we believe that a deeper analysis of learned filter banks with respect to changes in architectures can shed light on building better black-box models.

heading: 6 Broader Impact
text: Understanding the strength of adversarial attacks is critical to building robust defenses in the future. The practical implications of adversarial examples in safety-critical applications, such as autonomous driving and biometrics, are huge. Our extensive set of experiments highlights the vulnerability of modern networks to adversarial attacks even in the extreme black-box case, which has a high potential for societal impact. We hope that our work will motivate the community to further study the internal workings of neural networks in the context of adversarial attacks, and in particular exploit such analysis to design effective defense mechanisms.

heading: Acknowledgement
text: This work was supported by the Swiss National Science Foundation.

Review summaries:
The paper proposes an adversarial attack generated by another deep neural network (DNN), that shown to have potential to transfer the attack to different network architectures, different datasets and different tasks. The authors inspire the attack from an observation that mid-level features of a DNN share similarities with different architectures, datasets and tasks. Learning the adversarial perturbation generator network to attack the mid-level features of a DNN shown to transfer to different attack settings. The effectiveness of the attack is justified on different attack settings (white box and variants of black box attacks) that emphasize the transferrable nature of the attack.

The work studied the generalization of adversarial attacks that are induced by generative methods. Specifically, the transferability of the perturbations for different testing conditions are evaluated. The mid-level features of DNNs are used to learn a transferable perturbation generator. Evaluations of transferable attacks are performed on tasks with a different network architecture, and/or data distributions. 

This paper shows that mid-level features show a common pattern across different domains/tasks/architectures. Motivated by this, this paper introduced to train a generative model which leveraging the mid-level features to produce adversarial examples. Experimental results demonstrated that the proposed method is effective under different settings.