Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the transferability of adversarial examples crafted by attacking one model to another model. The main idea is to employ a generator to produce adversarial examples and maximize feature divergence on a latent space.
main_review: The core idea of this work is to run an attack on mid-features at the intermediate layer of the source network. The main intuition is that mid-layers tend to learn common patterns (e.g., textures) across various deep nets trained on the same dataset or similar datasets.

However, the arguments in the paper regarding why maximizing feature divergence on mid-layers can boost transferability have not been addressed comprehensively and thoroughly. The current arguments are too general to me. It would be better if the authors delve into what really happens with mid-layer features and what changes in the pattern of adversarial examples when maximizing feature divergence on mid-layers. This would help insightful demonstrate the usefulness of the proposed method.

--Post rebuttal---
The responses from the authors have addressed my concerns so I have increased my score.

limitations_and_societal_impact: Although the experiments are quite comprehensive, the proposed method is restrictively novel. Moreover, the arguments of the proposed method are too general, hence needing a more comprehensive study.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an adversarial attack generated by another deep neural network (DNN), that shown to have potential to transfer the attack to different network architectures, different datasets and different tasks. The authors inspire the attack from an observation that mid-level features of a DNN share similarities with different architectures, datasets and tasks. Learning the adversarial perturbation generator network to attack the mid-level features of a DNN shown to transfer to different attack settings. The effectiveness of the attack is justified on different attack settings (white box and variants of black box attacks) that emphasize the transferrable nature of the attack.
main_review: Originality: 
Learning the generator to attack the mid-level features and demonstrating such generator to transfer to different attack settings is interesting and novel. Previous works are cited and differences are discussed sufficiently.

Quality: 
The paper provides an interesting finding and technically sound. The proposed method is appropriate and the claims are well supported by the extensive experimental results. This is a complete work and the strengths of the method are reveled by crafting the experiments carefully. The weakness of the method is not sufficiently addressed (see below). 

Clarity: 
The paper is clearly written, well organized and also provided details to reproduce the results.

Significance: 
The paper addresses an important topic of transferring the attack to different test settings.
The results of the proposed attack outperforms the previous attacks in all the different attack settings. Future works would benefit from the unique findings made in this work. 

I appreciate that authors study the effectiveness of their method on few of the adversarially trained models. However, I wonder that the authors missed the PGD based adversarially trained model as a baseline to test their method. CK Mummadi et al. has shown that PGD based adversarially trained models are also robust to universal perturbations and thus raises a question how their method performs on such defense method. 

Besides the interesting finding that mid-level features are shared across different architectures, my major concern about this work is the assessment of the proposed attack against potential defenses. How strong is the proposed attack when a defense is crafted to address it specifically? It seems the attack objective could be easily get integrated into PGD based adversarial training to achieve a robust model. If so, I wonder whether the proposed attack would circumvent such defense mechanism. I believe this is an important discussion to be addressed in order to evaluate the significance of the proposed attack. Finally, I rate this paper as marginally above acceptance threshold. However, I am willing to reconsider my rating based on the author responses and other reviews.

CK Mummadi et al.  Defending Against Universal Perturbations With Shared Adversarial Training, ICCV 2019.

-----------------------------------
Post rebuttal remarks:
I read the other reviews and authors responses. I thank the authors for conducting additional experiments on defended models. The results show that the proposed attack is effective against HGD and R&P defended models, but not effective against PGD and Feature Denoising based defended models (same as other attacks). However, my major concern of crafting a defense that specifically target to circumvent this attack still remains open and hence I keep my rating same as before.
limitations_and_societal_impact: I appreciate the authors for covering the limitations and broader impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The work studied the generalization of adversarial attacks that are induced by generative methods. Specifically, the transferability of the perturbations for different testing conditions are evaluated. The mid-level features of DNNs are used to learn a transferable perturbation generator. Evaluations of transferable attacks are performed on tasks with a different network architecture, and/or data distributions. 
main_review: originality: the work presents an interesting finding that by maximizing the difference of mid level features of neural networks between the input image and the perturbed image, the generated perturbation can be transferred to another task setting such as a different data distribution or a different network structure. This finding itself is a useful contribution. However, the method of using generative method to generate perturbation is not new. The results indicate that there is a large gap between the white-box setting and the black-box setting. This means that when the target data distribution is hugely different and/or target network architecture is different, the transferability of the attack degrades. 

quality: the paper is technically sound and the claims are solid. However, the results indicate that there is still a large gap between the non-transfer case and the transfer case. In order to train an attacker that can transfer or generalize well to different settings, there could be other methods such as using meta learning to train the attacker on multiple data sources or multiple network architectures. It would be great if the author could provide results on meta learning based attack to see how good is the proposed attack comparing with meta learning based attack. 

clarity: the paper is presented in a clear way except for some tiny fonts in some of the figures. 

significance: the results are important and the insight from the paper is useful for other applications. 
limitations_and_societal_impact: The authors mentioned their limitations pretty well, in terms of only testing on a single layer. However, it would be great if the authors could add additional experiments comparing with meta learning or domain randomization based attack for improving generalization of attacks.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper shows that mid-level features show a common pattern across different domains/tasks/architectures. Motivated by this, this paper introduced to train a generative model which leveraging the mid-level features to produce adversarial examples. Experimental results demonstrated that the proposed method is effective under different settings. 
main_review: ---
Strength:
- The introduced methods show improvements over existing methods (CDA) in several different settings, including a challenging black-box attack setting. The attacker does not know any information about the target architecture/dataset/task. 
- The experiments results are well designed, comprehensive and outperform existing methods. Analysis using t-SNE shows the introduced method can produce well-separated features even in cross-domain/architecture attacks.
- The writing and presentations of this paper are easy to follow.
---
Concerns and Questions:
- Although the visualization of mid-level features in Figure 2 shows certain similarities, a qualitative metric would be more convincing. Such as CKA/CCA [1].
- Why not include RHP in Table 1,2,3? It is also a baseline method.
- In Table 3, why is the fooling rate lower for ChestX10k than others? Using ChestX data to train generators against ChestXNet should have a higher fooling rate, and this result is a bit puzzling. 
- Figures 3 and 4 in supplementary is the missing label for the x and y-axis. Presumably, x for layers and y for fooling rate, there is no need for >100 for the y axis. Is Figure 3/4 only considered the mid-level layers for different architectures? For ResNet models, layers are chosen from 1-4, while others choose from a wide range of layers. Is there any method/reason for how to choose these layers? The introduced methods rely on the choice of intermediate layers for the attack. There should be a more comprehensive ablation study on this. Such as a simple comparison for introduced method applies to low/high-level features with mid-level features. 

[1] Similarity of Neural Network Representations Revisited, ICML 2019.



limitations_and_societal_impact: Authors have adequately addressed the limitations and potential negative societal impact
review_title: null

Meta-review: The paper studied the generalization of adversarial attacks induced by generative methods. The authors found that, by maximizing the difference of mid-level features of neural networks between the clean and perturbed images, the generated perturbation can be better transferred to another task setting such as a different data distribution or a different network structure. The finding is interesting and useful to the community. However, there is still a gap between the attack performance under the white-box and black-box settings. Furthermore, the proposed attack is effective against HGD and R&P defenses, but not effective against PGD and feature-denoising-based defenses. We suggest the authors discuss these in more detail in future revisions.
Decision: Accept (Poster)