Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the challenge of generating adversarial perturbation for a target model - with no access to the model, or the model's training data (i.e. target domain). Using a trained model and data from a source domain (ImageNet), the authors train a generator to craft perturbations which maximize the cosine distance between the intermediate features of clean and adversarial images. This generator is then assisted by two techniques - random normalization of the input image, and spatial attention on Intermediate-layer features (used for cosine distance). Experiments show that this method outperforms prior methods in black-box setting (no access to target domain or model) as well as white-box setting. 
main_review: ### Strengths:

1) The problem setting (no access to target data) is of importance - in practice, access to data is as hard, if not harder, than access to model.
2) The experiments are extensive, and clearly show a significant improvement in black-box attack capability.
3) The code provided with the paper, along with the appendix, help gaining a clearer understanding of the method (conversely, they further emphasize readability issues of the manuscript).

### Weaknesses:

1) The manuscript is poorly written - grammatical mistakes and semantic mistakes are aplenty. Some phrases are the opposite of what the method actually does. Section 3.4 states "Specifically, we apply a channel-wise average pooling to the feature maps at layer L" where as the actually operation is cross-channel average pooling (Refer to [1]). Other mistakes are highlighted below.

2) The novelty of the method is limited. In Wen Zhou et al., 2018 [2], Intermediate feature disruption is used to increasing black-box transferability. In Weibin Wu et al., 2020 [3], attention is used for increasing transferability. The paper does not mention these works.

3) The claims of section 4.2 are only weakly supported. Statement: "The downsampling module has an essential impact on the resulting adversarial examples" I fail to see how this can be inferred from visualizing the cross-channel attention outputs.
 
### Other weaknesses:

1) Since all the experiments are only regarding ImageNet -> target datasets, it is unclear how well the method will perform if the source dataset is different (especially if the source dataset is small). 
2) The metrics do not include standard deviation across multiple random runs. Evaluating the standard deviation in at least one setting will elucidate the significance of the results.
3) Results with combining the two proposed techniques (RN and DA) *should* be present in the main draft. This is an important question that the manuscript only deals with in the appendix. The manuscript with benefit from a discussion on the fact that using these two techniques in tandem is challenging, and fails to consistently out-perform using just a single module. 

### Text Errors: 

#### Abstract:
1) transferability nature ==> transferable nature.
2) the only knowledge ==> only the knowledge
3) the coarse-grained domain ==> coarse-grained domains

#### Introduction:

1) possible to the spotlight ==> possible
2) transparent ==> opaque 
3) the query ==> querying
4) but more threatening ==> and more threatening
5) the generator ==> a generator

#### Method:
1) they can subject to the ==> they can be modeled as samples from the standard normal distribution.
2) even the inputs are not ==> even if the inputs are not

#### Experiments:
1) in the Torchvision. ==> in the Torchvision library.
2) another seven ==> seven other

### Questions for the authors: 

1) How will the generator network perform with its trained with all source models at once? (See experiments - Table 3 in Konda Reddy Mopuri et al., 2017 [4]) I suspect that it should further increase the transferability.

2) Have the authors tried to increase the RGB jittering when comparing to existing methods? I suspect that with significant jittering, augmentation may perform similar to random normalization.

### References: 

[1] Network In Network, Min Lin, Qiang Chen, Shuicheng Yan, arXiv 2013.

[2] Transferable Adversarial Perturbations, Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, Yong Yang, ECCV 2018.

[3] Boosting the Transferability of Adversarial Samples via Attention, Weibin Wu, Yuxin Su, Xixian Chen, Shenglin Zhao, Irwin King, Michael R. Lyu, Yu-Wing Tai, CVPR 2020.

[4] NAG: Network for Adversary Generation, Konda Reddy Mopuri, Utkarsh Ojha, Utsav Garg, R. Venkatesh Babu, CVPR 2018
summary_of_the_review: The method proposed in this paper out-performs existing methods, and targets an important setting (no access to target domain or model). However, the writing is error-ridden, and the proposed method is only marginally novel w.r.t. existing works. Therefore, I rate the paper as marginally above accept threshold, conditional on the authors correcting the mistakes highlighted above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work first identifies a more practical threat model for black-box transfer adversarial attack, where the target model's domain remains unknown, and the attacker's surrogate model may be trained in another domain. Then, the BIA attack is proposed to enhance transferability, whose key idea is to distort low-level features captured by DNN's intermediate layers instead of perturbing the domain-specific features in the output layer. Two modules, DA and RN, are further proposed to improve attack success rate. Experimental results demonstrate that BIA is more effective than existing methods.
main_review: See the pros/cons below.

### Pros
1. Considering more practical threat model is certainly helpful and important for the transfer attack research.
2. The results indeed demonstrate large improvement of BIA in terms of error rate.

### Cons
1. I'm a little concerned about the "cross-domain" statement made in this work. To me, the target datasets considered in this work (CIFAR, STL, CUB, Stanford Cars) are still coming from the same natural imagery "domain" as ImageNet, despite they have different label spaces. In particular, CUB is known to have overlap with ImageNet [1], where the "cross-domain" claim certainly does not hold. An example case that is more "cross-domain" would be to transfer from ImageNet model to a ChestX-ray model (in a similar sense to Naseer et al.).

2. The specific methodology of BIA seems not new. It is known that perturbing intermediate layer features can yield more transferable adversarial examples (e.g., [2]). In fact, the formulation of BIA appears very similar to the one proposed in [2] (while BIA minimizes the cosine similarity between intermediated layer features of the clean and adversarial examples, [2] maximizes the euclidean distance, which essentially is the same). Feature space attacks are also shown to be more powerful than decision space attacks in more strict black-box transfer scenarios [3], but Sec. 3.2 fails to recognize these existing works. Clearly identifying the difference between BIA and [2] might help address this concern.

3. If my above judgement of BIA not being new is correct, then my further concern comes from the result side. In Table 2, it seems that the performance gain can be largely attributed to the BIA (or essentially feature space attack) itself rather than the DA and RN module. This hurts the empirical novelty to some extent, as previous works have shown the superiority of feature space attacks in either standard or more strict transfer settings ([2,3]).


[1] http://www.vision.caltech.edu/visipedia/CUB-200-2011.html

[2] Feature Space Perturbations Yield More Transferable Adversarial Examples

[3] Perturbing Across the Feature Hierarchy to Improve Standard and Strict Blackbox Attack Transferability 
summary_of_the_review: This paper indeed identifies a more practical threat model, but the experiments do not closely match the proposed "cross-domain" scenario, and the performance gain seems to largely come from existing technique (perturbing feature space instead of decision space). These issues prevent me from recommending for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the transferability of black-box domains. In real life, we do not know the relevant information of the deployed model and transfer attacks on black-box domains can better evaluate the vulnerability of deployed models. Therefore, Beyond ImageNet Attack (BIA) is proposed to investigate the transferability towards black-box domains (unknown classification tasks) with the only knowledge of the ImageNet domain. From the perspective of data and model, the authors propose random normalization (RN) module and domain-agnostic attention (DA) module to narrow the gap between the source and target domains. Finally, BIA achieves state-of-the-art performance in black-box domains settings.
main_review: Strengths:
1. BIA focuses on disrupting low-level features to improve transferability.
2. This work proposes random normalization (RN) module to handle the various distributions between source domains and target domains.
3. This work proposes domain-agnostic attention (DA) module to produce a more robust feature representation.

Weaknesses:
1. RN module and DA module are not always mutually reinforcing. The reason behind this has not been analyzed.
2. In competitors, diverse inputs method (DIM) is not new. From this perspective, why not use the more powerful MI-DI-TI-FGSM or a newer transfer attack？
3. Table 2 and Table 3 show the transferability comparisons on classification tasks. However, the effects of DA and RN seem to depend on different models. To understand more deeply, it is necessary to analyze why different modules have different effects in different models.

Minor questions:
1. What are the experiments on the fine-grained and coarse-grained classification to prove? Why distinguish between fine-grained and coarse-grained? There is no clear explanation in this work.
2. Does the comparison method differ greatly in training costs?

summary_of_the_review: I tend to accept this paper because it focuses on more realistic black box attack settings and proposes two modules to improve performance. The design of the module is insightful and effective, but the module proposed under some models is not always effective, which limits its application and requires more adequate analysis.
review_title: null

Meta-review: This paper considers that the model's training data may be not accessible when learning the attacking model, and thus a more practical blackbox attack scheme, Beyond ImageNet Attack (BIA) framework, is designed. All the reviewers agreed that the setting in this paper is important and helpful when designing attack methods. However, the method is not totally new. Nevertheless, considering the importance of the problem investigated in this paper, the nice design of the overall framework, and the extensive experiments, the AC recommends accept for this paper.
Decision: Accept (Poster)