BEYOND IMAGENET ATTACK: TOWARDS CRAFTING ADVERSARIAL EXAMPLES FOR BLACK-BOX DOMAINS
Year: 2022

Abstract:
Adversarial examples have posed a severe threat to deep neural networks due to their transferable nature. Currently, various works have paid great efforts to enhance the cross-model transferability, which mostly assume the substitute model is trained in the same domain as the target model. However, in reality, the relevant information of the deployed model is unlikely to leak. Hence, it is vital to build a more practical black-box threat model to overcome this limitation and evaluate the vulnerability of deployed models. In this paper, with only the knowledge of the ImageNet domain, we propose a Beyond ImageNet Attack (BIA) to investigate the transferability towards black-box domains (unknown classification tasks). Specifically, we leverage a generative model to learn the adversarial function for disrupting low-level features of input images. Based on this framework, we further propose two variants to narrow the gap between the source and target domains from the data and model perspectives, respectively. Extensive experiments on coarse-grained and fine-grained domains demonstrate the effectiveness of our proposed methods. Notably, our methods outperform state-of-theart approaches by up to 7.71% (towards coarse-grained domains) and 25.91% (towards fine-grained domains) on average. Our code is available at https: //github.com/Alibaba-AAIG/Beyond-ImageNet-Attack.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) have achieved remarkable success in the image classification task in recent years. Nonetheless, advances in the field of adversarial machine learning (Szegedy et al., 2014; Goodfellow et al., 2015) make DNNs no longer reliable. By adding a well-designed perturbation on a benign image (a.k.a adversarial attack), the resulting adversarial examples can easily fool state-of-the-art DNNs. To make the matter worse, the adversarial attack technique can even be applied in the physical world (Sharif et al., 2016; Kurakin et al., 2017a; Xu et al., 2020; Duan et al., 2021), which inevitably raises concerns about the stability of deployed models. Therefore, exposing as many “blind spots” of DNNs as possible is a top priority.
Generally, deployed models are mainly challenged with two threat models: white-box and blackbox. For white-box threat model (Kurakin et al., 2017b; Moosavi-Dezfooli et al., 2016; Carlini & Wagner, 2017; Shi et al., 2019), the attacker can obtain complete knowledge of the target model, such as the gradient for any input. However, deployed models are usually opaque to unauthorized users. In this scenario, prior black-box works (Poursaeed et al., 2018; Dong et al., 2018; Xie et al., 2019; Inkawhich et al., 2020; Gao et al., 2020b; Wang et al., 2021) mostly assume that the source data for training the target model is available and mainly explore the cross-model transferability among models trained in the same data distribution. Specifically, perturbations are crafted via accessible white-box model (a.k.a substitute model), and resulting adversarial examples sometimes can fool other black-box models as well. Yet, these works still ignore a pivotal issue: A model owner is ∗Work done when Qilong Zhang interns at Alibaba Group, China †Corresponding author
unlikely to leak the relevant information of the deployed model. To overcome this limitation, querybased black-box attacks (Papernot et al., 2016; Brendel et al., 2018; Chen et al., 2020; Li et al., 2021) are proposed, which adjust adversarial examples just according to the output of the target model. However, the resource-intensive query budget is extremely costly and inevitably alerts the model owner.
Therefore, we need a more “practical” black-box threat model to address this concern, i.e., without any clue about the training data distribution as well as the pre-trained model based on it, and even querying is forbidden. Intuitively, this threat model is more challenging to build for attackers and more threatening to model owners. To the best of our knowledge, a recent work called CDA (Naseer et al., 2019) is the first to attempt such an attack. Specifically, it learns a transferable adversarial function via a generator network against a different domain (training data and pre-trained model are all from ChestX-ray (Wang et al., 2017) domain). During inference, it directly crafts adversarial examples for benign ImageNet images to fool target ImageNet pre-trained models. However, its cross-domain transfer strength is still moderate. Besides, relying on small-scale datasets to train a generator may limit the generalization of the threat model.
Considering that ImageNet is a large-scale dataset containing most of common categories in real life and there are various off-the-shelf pre-trained models, one can easily dig out much useful information to build a strong threat model. Therefore, in this paper, solely relying on the knowledge of the ImageNet domain, we introduce an effective Beyond ImageNet Attack (BIA) framework to enhance the cross-domain transferability of adversarial examples. To reflect the applicability of our approach, we consider eight different image classification tasks (listed in Table 1). Figure 1 illustrates an overview of our method. Particularly, we learn a flexible generator network Gθ against ImageNet domain. Instead of optimizing the domain-specific loss function like CDA, our method focuses on disrupting low-level features following previous literature to ensure the good transferability of our BIA. Furthermore, we propose two variants based on the vanilla BIA to narrow the gap between source and target domains. Specifically, from the data perspective, we propose a random normalization (RN ) module to simulate different data distributions; from the model perspective, we propose a domain-agnostic attention (DA) module to capture essential features for perturbing. In the inference phase, our Gθ accepts images of any domain as the input and crafts adversarial examples with one forward propagation. Extensive experiments demonstrate the effectiveness of our proposed methods. Towards the coarse-grained and fine-grained domains, we can outperform state-of-the-art approaches by up to 7.71% and 25.91% on average, respectively. Besides, our methods can also enhance the cross-model transferability in the source domain.

heading: 2 RELATED WORKS
text: Iterative Optimization Approaches. Under the black-box threat model, iterative attack methods are a popular branch, which usually adopt domain-specific loss or intermediate feature loss to craft adversarial examples. For the former, Madry et al. (2018) extend Goodfellow et al. (2015) to perform projected gradient descent from randomly chosen starting points inside -ball. Dong et al. (2018) introduce momentum term to stable the update direction. Xie et al. (2019) apply random transformations of the input at each iteration, thus mitigating overfitting. Gao et al. (2020a) propose patch-wise perturbation to better cover the discriminative region. Wu et al. (2020a) explore the security weakness of skip connections (He et al., 2016; Huang et al., 2017) to boost attacks.
Different from the methods mentioned above, intermediate feature-based methods focus on disrupting low-level features. For example, Zhou et al. (2018) maximize the Euclidean distance between the source image and target image in feature space and introduce regularization on perturbations to reduce variations. Inkawhich et al. (2019) make the source image close to the target image in feature space. Lu et al. (2020) propose a dispersion reduction attack to make the low-level features featureless. Naseer et al. (2020) design a self-supervised perturbation mechanism for enabling a transferable defense approach. Wu et al. (2020b) compute model attention over extracted features to regularize the search of adversarial examples.
Generator-oriented Approaches. Compared with iterative optimization approaches, generatororiented attacks are more efficient (i.e., only need one inference) to generate adversarial examples. In this branch, Baluja & Fischer (2017) propose an adversarial transformation network to modify the output of the classifier given the original input. Poursaeed et al. (2018) present trainable deep neural networks for producing both image-agnostic and image-dependent perturbations. Naseer et al. (2019) leverage datasets from other domain instead of ImageNet to train generator networks against pre-trained ImageNet models, and inference is performed on ImageNet domain with the aim of fooling black-box ImageNet model. They also attempt a practical black-box threat model (from ChestX-ray to ImageNet), and the attack success rate can outperform the result of Gaussian noise.

heading: 3 TRANSFERABLE ADVERSARIAL EXAMPLES BEYOND IMAGENET

heading: 3.1 PROBLEM FORMULATION
text: Given a target deep learning classifier ft(·) trained in a specific data distribution χt, we aim to craft a human-imperceptible perturbation for the benign image xt ∼ χt from the target domain with the only available knowledge of source ImageNet domain (including pre-trained model fs(·) and data distribution χs). Formally, suppose we have a threat modelMθ∗ whose parameter θ∗ is solely derived from the source domain, our goal is to craft adversarial examples for xt from target domain so that they can fool the ft(·) successfully:
ft(Mθ∗(xt)) 6= ft(xt) s.t. ||Mθ∗(xt)− xt||∞ ≤ , (1)
where is the maximum perturbation to ensure xt is minimally changed. Intuitively, crafting adversarial examples for the black-box domain is very challenging. As shown in Table 1 and Figure 6 of Appendix, images from different domain vary greatly.

heading: 3.2 PRELIMINARY
text: Iterative/Single-step optimization methods (Goodfellow et al., 2015; Madry et al., 2018; Zhao et al., 2020; Gao et al., 2021; Mao et al., 2021; Li et al., 2021) and generator-oriented methods (Baluja & Fischer, 2017; Poursaeed et al., 2018; Naseer et al., 2019) are two popular branches for building the threat model. Since the attacker has the large-scale ImageNet training set at hand, there is no reason not to take full advantage of them. Therefore, in this paper, we adopt the generator-oriented framework which learns a transferable adversarial function via a generative model Gθ. Given that the threat model aims at crafting transferable adversarial examples for black-box domains, relying
on the last layer with domain-specific loss functions (e.g., relativistic cross-entropy loss adopted by Naseer et al. (2019)) is less effective since this might lead to overfitting to source domain. In contrast, the intermediate layers of the DNN presumably extract general features (Yosinski et al., 2014) which may share across different models. Hence, as a baseline for the new black-domain attack problem, our Beyond ImageNet Attack (BIA) turns to destroy the low-level features of the substitute model at a specific layer L to generate transferable adversarial examples according to existing literature (Yosinski et al., 2014; Zhou et al., 2018; Inkawhich et al., 2019).
As illustrated in Figure 1, Gθ is learned to decrease the cosine similarity between adversarial example x′s and benign image xs ∈ RN×Hs×Ws (sampled from χs) to make the feature featureless:
θ∗ = argmin θ Lcos(fLs (x′s), fLs (xs)). (2)
In the inference phase, our generator Gθ∗ can directly craft adversarial examples for input images xt ∈ RN×Ht×Wt from the target domain:
x′t = min(xt + ,max(Gθ∗(xt),xt − ). (3) The resulting adversarial examples x′t are depicted in Figure 8 of Appendix. Compared with CDA, our BIA is more effective in both source (white-box) and target (black-box) domains. Yet, as shown in Figure 2, crafting more transferable adversarial examples still has some challenges:
• Data Perspective: The distribution (i.e., mean and standard deviation) of source domain is largely different from the target domain. For example, the standard deviation of ImageNet is about twice that of SVHN.
• Model Perspective: Although some feature map of fLs (·) can capture the object (∈ χt) for feature representation (e.g., the first feature map in Figure 2), there are also some feature maps that are significantly biased (e.g., the second feature map in Figure 2).
To alleviate the concern of generating poor transferable adversarial examples that may arise from the above limitations, we propose two variants, equipped with random normalization (RN ) module or domain-agnostic attention (DA) module, respectively.

heading: 3.3 RANDOM NORMALIZATION MODULE
text: Generally, DNNs (Simonyan & Zisserman, 2015; He et al., 2016; Huang et al., 2017) are usually equipped with normalization for input images1 so that they can be modeled as samples from the standard normal distribution. However, as illustrated in Figure 2 (left), the distribution of dataset from the different domain can vary dramatically. Thus, training against a specific domain may limit the generalization of the resulting Gθ∗ . Besides, the commonly used strategy of label-preserving data augmentation (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015) is less effective because it has little effect on changing the distribution of the inputs (more details are shown in Appendix A.3).
To overcome this limitation, fusing knowledge from different domains might be helpful. As shown in Table 3 of Naseer et al. (2019), using training data from other domains against the pre-trained
1For Pytorch pre-trained ImageNet model, input images should be normalized using mean = [0.485, 0.456, 0.406] and standard deviation = [0.229, 0.224, 0.225]
model in the source domain usually enhances the transferability of adversarial examples towards the target domain. However, this setup is not feasible because the available knowledge for a more practical threat model may be limited, i.e., restricted to one domain. Therefore, we instead propose a random normalization (RN ) module to simulate different data distribution in the training phase:
RN (xs) = σ · xs − µ′
σ′ + µ, (4)
where σ and µ are default standard deviation and mean vectors for ImageNet, and µ′ ∼ N (µ′mean, µ′std) and σ′ ∼ N (σ′mean, σ′std) are two random scales2 sampled from Gaussian distribution. Combined withRN , and the object function can be expressed as:
θ∗ = argmin θ Lcos(fLs (RN (x′s)), fLs (RN (xs))). (5)

heading: 3.4 DOMAIN-AGNOSTIC ATTENTION MODULE
text: Unlike the random normalization module, the domain-agnostic attention module aims to narrow the domain gap from the model perspective. Our inspiration is from prior works (Hansen & Salamon, 1990; Caruana et al., 2004; Dong et al., 2018), which demonstrate that the ensemble strategy can avoid getting trapped in the local optimum and improve performance. Since there are many feature maps at layerL and each of them can model the input, i.e., extracts the features, we can also integrate them to produce a more robust feature representation AL, thus mitigating the impact of several biased feature maps, e.g., the second feature map of Figure 2. Specifically, we apply cross-channel average pooling to the feature maps at layer L:
AL = | ∑C i=0 f L s (xs)|
C , (6)
where C is channel number of fLs (xs).
As depicted in Figure 3, even if our source model is not trained in the target domain, the robust feature representation is still able to capture the essential feature of object very well. Surprisingly, it is even similar to the one that derived from a completely different target model. Therefore, this robust feature representation can serve as a domain-agnostic attention (DA) to enhance the cross-domain transferability
of adversarial examples. Specifically, in the training phase, we leverage AL to assign weights for each pixel of feature maps at the same layer. The resulting object function can be written as:
θ∗ = argmin θ Lcos(AL fLs (x′s),AL fLs (xs)), (7)
where is Hadamard product. After the training, even if the inputs are not from the source domain, our generator Gθ∗ is supposed to own an ability to capture the essential feature to disrupt.

heading: 4 EXPERIMENTS
text: Source (white-box) Domain. Our training data is the large-scale ImageNet (Russakovsky et al., 2015) training set which includes about 1.2 million 3×224×224 images. Generators are trained against four ImageNet pre-trained models including VGG-16, VGG-19 (Simonyan & Zisserman, 2015), ResNet152 (Res-152) (He et al., 2016) and DenseNet169 (Dense-169) (Huang et al., 2017). In this domain, we also consider three other models, i.e., DenseNet121 (Dense-121) (Huang et al.,
2µ′mean, µ′std, σ ′ mean and σ′std are all hyper-parameters.
2017), ResNet50 (Res-50) (He et al., 2016) and Inception-v3 (Inc-v33) (Szegedy et al., 2016), to analyze the cross-model transferability. All models are available in the Torchvision library4.
Target (black-box) Domain. Generally, the image classification tasks can be divided into coarsegrained and fine-grained tasks in terms of label granularity (Touvron et al., 2021). Thus, in addition to ImageNet (white-box domain), we consider seven other black-box domains (shown in Table 1) including four coarse-grained (CIFAR-10, CIFAR-100 (Krizhevsky, 2009), STL-10 (Coates et al., 2011) and SVHN (Netzer et al., 2011)) and three fine-grained (CUB-200-2011 (Wah et al., 2011), Stanford Cars (Krause et al., 2013) and FGVC Aircraft (Maji et al., 2013)) classification tasks. For the fine-grained classification, we use DCL framework (Chen et al., 2019) with three different backbones: ResNet50 (Res-50) (He et al., 2016), SENet154 and SE-ResNet101 (SE-Res101) (Hu et al., 2018). The pre-trained models for the coarse-grained classification are from Github5.
Implementation Details. Our generator Gθ adopts the same architecture as (Naseer et al., 2019), which is a composite of downsampling, residual (He et al., 2016) and upsampling blocks (more details can be found in Figure 7 of Appendix). The output size of Gθ is equal to the input size. For CDA and our methods, we use Adam optimizer (Kingma & Ba, 2015) with a learning rate of 2e-4 and the exponential decay rate for first and second moments is set to 0.5 and 0.999, respectively. All generators are trained for one epoch with the batch size 16. For the layer L, we attack the output of Maxpool.3 for VGG-16 and VGG-19, the output of Conv3 8 for Res-152 and the output of DenseBlock.2 for Dense-169 (ablation study can be found in Appendix A.1). For brevity, we only refer the output of a specific layer/block using its layer/block name. For our RN module, we set µ′ ∼ N (0.50, 0.08) and σ′ ∼ N (0.75, 0.08), and the ablation study is shown in Appendix A.2. Competitors. We compare our proposed methods with projected gradient descent (PGD) (Madry et al., 2018), diverse inputs method (DIM) (Dong et al., 2018; Xie et al., 2019), dispersion reduction (DR) (Lu et al., 2020), self-supervised perturbation (SSP) (Naseer et al., 2020) and cross-domain attack (CDA) (Naseer et al., 2019). The maximum perturbation ε is set to 10. Follow Lu et al. (2020), we set the step size α = 4 and the number of iterations T = 100 for all iterative methods. For DIM, we set the default decay factor µ = 1.0 and the transformation probability p = 0.7. The Gaussian smoothing (gs) for CDA is applied by 3× 3 Gaussian kernel. Evaluation Metrics. We use the top-1 accuracy on the whole test set (test size is shown in Table 1) after attacking to evaluate the performance of different methods. We also report standard deviation across multiple random runs in Appendix A.7.

heading: 4.1 TRANSFERABILITY COMPARISONS
text: In this section, we first conduct experiments for the black-box domain in Section 4.1.1 (coarsegrain) and Section 4.1.2 (fine-grain), then we report the results for the white-box (source) domain in Section 4.1.3. For the discussion of generator mechanism, changing source domain and ensemblemodel attacks, we leave them in Appendix A.4, Appendix A.5 and Appendix A.6, respectively.

heading: 4.1.1 RESULTS ON COARSE-GRAINED DOMAIN
text: In this section, we craft transferable adversarial examples for coarse-grained classification tasks. The results are shown in Table 2, where we leverage four ImageNet pre-trained models, including VGG-16, VGG-19, Res-152 and Dense-169, to train generators, respectively.
From Table 2, a first glance shows that our proposed methods consistently surpass state-of-the-art approaches. For example, if the substitute model is VGG-16, the most effective CDA remains a top-1 accuracy of 66.41% for CIFAR-10 after attacking, while our vanilla BIA can effectively bring down it to 57.38% and DA variant can further drop the top-1 accuracy to 55.16%. Among all tasks, the STL-10 and SVHN domains are the most difficult to attack, and the performance gap among existing attacks is moderate. Nonetheless, we can significantly enhance the transferability with the help of ourRN variant in these domains. Notably, if the substitute model is Res-152,RN
3Inc-v3 expects inputs with a size of 3× 299× 299. Therefore, the adversarial examples are also craft on the same size inputs. Note the size of training data for our generator is still 3×224×224.
4https://pytorch.org/vision/stable/models.html 5https://github.com/aaron-xichen/pytorch-playground
variant can further decrease the top-1 accuracy from 89.46% (vanilla BIA) to 85.79% on SVHN. Compared with state-of-the-art CDA on all domains, RN variant significantly outperforms it by 7.71% on average. This demonstrates that our proposed RN module is effective in coping with different distributions of inputs, thus improving the generalization of the resulting generator.

heading: 4.1.2 RESULTS ON FINE-GRAINED DOMAIN
text: We also analyze the transferability of adversarial examples towards fine-grained classification tasks. For each domain, three black-box models with different backbones trained via the DCL framework are the target. The results are summarized in Table 3, where the leftmost column is the substitute model and the top row shows the target model.
In this scenario, the performance gap between the existing state-of-the-art algorithms and our proposed methods is further enlarged. Remarkably, when attacking against Dense-169, even the vanilla BIA can drop the average top-1 accuracy to 29.23%, while CDA+gs, CDA, SSP, DR, DIM and PGD are still with the high average top-1 accuracy of 68.75%, 61.91%, 48.65%, 80.84%, 68.06% and 84.06% after attacking, respectively. Furthermore, by adding DA or RN modules in the training phase, the generator Gθ∗ is capable of crafting more transferable adversarial examples. On average, ourRN variant can reduce the top-1 accuracy from 46.52% (vanilla BIA) to 39.33%, and DA variant can further drop it to 36.42%, which remarkably outperforms SSP by 25.91%. This demonstrates our proposed DA module can effectively alleviate the bias caused by several feature maps, thus focusing on disrupting essential features.

heading: 4.1.3 RESULTS ON SOURCE DOMAIN
text: Although our proposed methods are mainly designed to improve the threat of adversarial examples towards black-box domains, they are also effective for enhancing the cross-model black-box transferability in the white-box domain. For example, by training against Dense-169, CDA still remains a top-1 accuracy of 43.78% on Inc-v3, while our BIA can achieve a relatively low top-1 accuracy of 38.58% on it. Besides, RN variant can further decrease the top-1 accuracy on Dense-169 (whitebox model) and Inc-v3 (black-box model) by 2.4% and 24.34%, respectively. This demonstrates that our proposed RN module is also able to avoid getting stuck in the local optimum of a specific model when training on a large-scale dataset. For DA variant, since the target domain and source domain are identical, it is naturally able to improve the transferability as well. As shown in Table 4, compared with vanilla BIA,DA variant can further degrade the top-1 accuracy from 18.45% (vanilla BIA) to 13.47% on average.

heading: 4.2 COMBINATION OF DOMAIN-AGNOSTIC ATTENTION AND RANDOM NORMALIZATION
text: In this section, we report the results for BIA equipped with bothRN and DA. The following is the update rule:
θ∗ = argmin θ Lcos(AL fLs (RN (x′s)),AL fLs (RN (xs))). (8)
As illustrated in Figure 4, RN module and DA module are not always mutually reinforcing. For example, when training against Res-152 and transferring adversarial examples to fine-grained or source domains, combining RN module and DA module can further decrease the average top-1 accuracy to 35.75% and 13.11%, respectively. However, if the black-box domain is coarse-grain,
only applyingRN module is better than applying both RN and DA modules. We speculate that it may be because the RN module affects the low-level features extract by the substitute model, and thus be incompatible with the DA module sometimes.
Since Figure 4 shows that usingDA and RN in tandem is less effective when training against Dense-169 while they reinforce each other in VGG-16 in most cases, we visualize the cross-channel average pooling of intermediate features for VGG-16 and Dense-169 to better explain this phenomenon. As illustrated in Figure 5, it can be observed that the RN module reinforces the discriminative features in VGG-16. However, it in-
hibits the response of objects’ essential features extracted by Dense-169. Consequently,DAmodule may cause the resulting generator to reduce the ability to attack essential features, thereby making it challenging to use these two techniques in tandem.

heading: 5 CONCLUSION
text: In this paper, we present a practical black-box threat model for the cross-domain attack. Specifically, we train a generator network in the large-scale ImageNet domain to disrupt low-level features better, thus generating transferable adversarial examples for the black-box domain. Based on this framework, we further propose two variants to narrow the gap between the source and target domains from the data and model perspectives, respectively. Extensive experiments demonstrate the effectiveness of our proposed methods. This also reminds the model owner that “Your deployed model is not safe even you do not leak any information to the public”. We hope our proposed approaches can serve as a benchmark for evaluating the stability of various deployed models.

heading: 6 ACKNOWLEDGE
text: This work was supported by the National Natural Science Foundation of China (Grant No. 62020106008, No. 61772116 and No. 61872064) and Alibaba Group.

heading: A APPENDIX
text: D ro po ut C on v Re LU C on v Ba tc hN or m Ba tc hN or m ⨁ C on v Re LU Ba tc hN
or m
Tr an sp os eC on v Ba tc hN or m Re LU
C on v
Downsampling UpsamplingResidual𝑥! 𝑥! "
CDA
Benign
BIA
Figure 8: Adversarial examples crafted by CDA and our vanilla BIA ( = 10). Both generator networks are trained against ImageNet pre-trained VGG-16 (Simonyan & Zisserman, 2015). Red highlighted labels represent misclassification.
A.1 SELECT LAYER FOR ATTACKING
In this section, we analyze the impact of different intermediate layers of the substitute model on the transferability of resulting adversarial examples. The results are illustrated in Figure 10.
In general, training against the shallow and middle layers yields more cross-domain transferable but less cross-model transferable adversarial examples than against the deep layer. For example, if the substitute model is Res-152, disrupting shallow layer Conv2 3 is more effective for transferring towards coarse-grained domain, and perturbing middle layer Conv3 8 is more effective in reducing the average top-1 accuracy of fine-grained models. In contrast, attacking deep layers like Conv5 3 can yield more transferable adversarial examples in the source domain. This demonstrates that lowlevel features are more similar across domains and high-level features are more domain-specific.
A.2 SELECT GAUSSIAN DISTRIBUTION FOR RANDOM NORMALIZATION
For DA module, it is parameter-free. Therefore, we only conduct the experiment to select an optimal distribution for RN module, i.e., the µ′ and σ′ in Equation 4. Here we tune the mean of µ′ and σ′ from 0.25 to 0.75 with a granularity of 0.25. For the standard deviation of them, we fix it to 0.08 so that the sampled µ′ and σ′ can basically take values ranging from 0.0 to 1.0 (according to the three-sigma rule).
The results are shown in Figure 9, where we craft adversarial examples via VGG-16 and report the average top-1 accuracy in both source and black-box domains. From the results, we observe that a bigger σ′mean can effectively improve the transferability. For example, if we increase σ ′ mean from 0.25 to 0.75, the top-1 accuracy on fine-grained models can be further decreased by 3.16% on average. Although the influence of µ′mean is relatively moderate when σ ′ mean = 0.75, setting µ ′ mean to 0.5 is usually better. Therefore, we set µ′ ∼ N (0.50, 0.08) and σ′ ∼ N (0.75, 0.08) in our paper.
A.3 DATA AUGMENTATION VS. RANDOM NORMALIZATION
Data augmentation (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015) is a widely used strategy for improving the generalization of the model. Nonetheless, these label-preserving transformations are less effective for training a generator to craft more transferable adversarial examples.
As demonstrated in Figure 11, where we report the results for our vanilla BIA, BIA with data augmentation (BIA+AUG)6 and BIA with RN (BIA+RN ), BIA+AUG is less effective than our proposed BIA+RN and might even degrade the performance of our vanilla BIA. For example, when training against VGG-19, BIA gets an average top-1 accuracy of 58.71% on the fine-grained domain, yet BIA+AUG degrades it to 62.29%. In contrast, our proposed BIA+RN can significantly decrease the result to 47.78%. This is mainly because that the common data augmentation cannot effectively change the distribution of the training dataset, thus decreasing the generalization towards the black-box domain.
A.4 INSIGHT INTO THE GENERATOR
Although many prior works (Poursaeed et al., 2018; Naseer et al., 2019) have leveraged the generator to craft adversarial examples, they hardly analyze the role of each block of the generator. This section will give an insight into the generator and understand how it processes an input. In Figure 12, we feed an image into the generator trained with the vanilla BIA (against Res-152) and visualize the
6Specifically, we use three common data augmentation techniques including “RandomResizedCrop”, “RandomHorizontalFlip” and “ColorJitter”.
output of each block. As we can observe, these blocks play different roles in crafting adversarial examples:
• Downsampling block: it mainly extracts discriminative features of the input image. As the size of the down-sampled images gets smaller, there is no significant noise overall.
• Residual block: Unlike the downsampling block, this block is responsible for adding noise and its behavior looks very similar to the iterative algorithm.
• Upsampling block: This block gradually reconstructs the adversarial example from abstract features.
Since the residual block mainly works as suppressing or reversing the features extracted by the downsampling module, the downsampling module has an essential impact on generating transferable adversarial examples.
To better understand the effectiveness of our proposed modules, here we investigate them from the perspective of the generator mechanism. Specifically, we apply crosschannel average pooling to the output of the downsampling block and calculate the difference map between vanilla BIA and our proposed variants. Without loss of generality, here we only show the definition of Diff(RN variant, BIA), and Diff(DA variant, BIA) can be easily de-
duced. Specifically, we first apply cross-channel average pooling to the output of the downsampling
block:
ALRN = | ∑C i=0 GdRN θ∗ (x)|
C , (9) ALBIA = | ∑C i=0 GdBIAθ∗ (x)|
C , (10)
where GdRN θ∗ and G d BIAθ∗ denote the output of the downsampling block forRN variant and vanilla BIA, respectively. Then our difference map can be expressed by:
Diff(RN variant,BIA) = { 1, ALRN −ALBIA > 0, 0, else.
(11)
From the result of Figure 13, we can observe that the generators derived from our proposed variants concentrate more on the body of the object (especially for DA variant) than that of vanilla BIA (which pays more attention to the background, i.e., the black region in the difference map). This demonstrates that our proposedRN andDA variants do narrow the domain gap, and thus be capable of yielding more transferable adversarial examples.
A.5 DISCUSSION ON CHANGING SOURCE DOMAIN
Since all the experiments are only regarding from ImageNet domains to target domains, it is unclear how well the method will perform if the source dataset is different (especially when the source dataset is small). Therefore, in the following Table 5, we report the results for transferability from CUB-200-2011 to other domains. Generators are learned against CUB-200-2011 domain (the substitute model is DCL (backbone: Res-50) and training data is CUB-200-2011 testing data (5794 images)). For results of fine-grained domains, we average top-1 accuracy of backbone SENet-154 and SE-Res101. For the result of the ImageNet domain, we average top-1 accuracy of all models introduced in our manuscript. We can observe that our method consistently outperforms our main competitor CDA by a large margin. Besides, we also notice that transferring from CUB-200-2011 to CIFAR is very challenging (compared with our reported results in Table 2). Therefore, we highlight the necessity of using a large-scale dataset such as ImageNet to train the adversarial examples generator.
A.6 DISCUSSION ON ENSEMBLE-MODEL ATTACKS
As demonstrated in prior works (Liu et al., 2017; Dong et al., 2018; Mopuri et al., 2018), attacking against an ensemble of models can yield more transferable adversarial examples. However, it is not clear whether the ensemble-model attack is also effective in our cross-domain attack scenario. To investigate this, we conduct an experiment in Table A.6, which shows the results for training against VGG-16 and an ensemble of VGG-16, Vgg-19, Res-152 and Dense-169, respectively.
From the result, We can observe that ensemble-based training can also improve the transferability of adversarial examples towards fine-grained domains significantly.
A.7 DISCUSSION ON STANDARD DEVIATION ACROSS MULTIPLE RANDOM RUNS
To ensure the stability and credibility of the evaluations, experiments are repeated several times for our methods. In Table 7, we report the results for each random seed and standard deviation across these random runs.
A.8 EFFECTS OF RN AND DA ON COARSE-GRAINED AND FINE-GRAINED TASKS
From Table 2 and Table 3, we observe that RN module is more effective than DA module for coarse-grained models, but not as well as DA module for fine-grained models. There may be two reasons:
On the one hand, the default normalization of coarse-grained classification models is different from ImageNet and fine-grained classification models. Specifically, coarse-grained classification models use mean = [0.5,0.5,0.5] and std = [0.5,0.5,0.5], but ImageNet and and fine-grained classification models use mean = [0.485,0.456,0.406] and std = [0.229,0.224,0.225]; Therefore, usingRN module can narrow normalization gap between ImageNet and coarse-grained domains.
On the other hand, the resolution of coarse-grained domains such as CIFAR-10 and CIFAR-100 is much lower than the ImageNet domain and fine-grained domains. Therefore, the intermediate features of images from coarse-grained domains are more coarse than those from ImageNet and fine-grained domains, which may enlarge the gap between ImageNet and coarse-grained domains.

Review summaries:
This paper tackles the challenge of generating adversarial perturbation for a target model - with no access to the model, or the model's training data (i.e. target domain). Using a trained model and data from a source domain (ImageNet), the authors train a generator to craft perturbations which maximize the cosine distance between the intermediate features of clean and adversarial images. This generator is then assisted by two techniques - random normalization of the input image, and spatial attention on Intermediate-layer features (used for cosine distance). Experiments show that this method outperforms prior methods in black-box setting (no access to target domain or model) as well as white-box setting. 

This work first identifies a more practical threat model for black-box transfer adversarial attack, where the target model's domain remains unknown, and the attacker's surrogate model may be trained in another domain. Then, the BIA attack is proposed to enhance transferability, whose key idea is to distort low-level features captured by DNN's intermediate layers instead of perturbing the domain-specific features in the output layer. Two modules, DA and RN, are further proposed to improve attack success rate. Experimental results demonstrate that BIA is more effective than existing methods.

This paper focuses on the transferability of black-box domains. In real life, we do not know the relevant information of the deployed model and transfer attacks on black-box domains can better evaluate the vulnerability of deployed models. Therefore, Beyond ImageNet Attack (BIA) is proposed to investigate the transferability towards black-box domains (unknown classification tasks) with the only knowledge of the ImageNet domain. From the perspective of data and model, the authors propose random normalization (RN) module and domain-agnostic attention (DA) module to narrow the gap between the source and target domains. Finally, BIA achieves state-of-the-art performance in black-box domains settings.