Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work studies how adversarial robustness of a source network can affect the transfer of adversarial examples to a different target network. The main finding is that slightly robust networks are ideal source networks; training the source networks with no adversarial perturbations, or with large perturbations, is less effective. The authors analyze why this might be the case, and propose that slightly robust networks have increased universality of representations.
main_review: Pros:
- This fits the story of adversarially trained networks transferring better to downstream tasks (a different kind of transfer).
- The result that highly robust networks don't work as well is surprising and interesting
- The writing is engaging and clear
- The main experiments are well-designed

Cons:
Section 4 is circular. Consider the following snippets.
- “Universality Explains Targeted Transferability” (the main claim)
- “When a neural network has substantial representation transferability to every other network of a certain type, we say that the neural network has features that are universal with respect to these destination networks” (definition of universality)
- “We refer to the degree to which adversarial examples of a source model can analogously affect the features that are computed by the representation layer of the destination model as the representation transferability from source to destination.” (definition of representation transferability)

If you follow the logic, this is basically saying that the degree to which adversarial examples can perturb features in a bunch of destination models explains targeted transferability, which is tautological. This is nitpicking, but it does matter for clarity. Basically, it's currently unclear what section 4 adds. Indeed, Table 2 feels like the same thing as Table 1, except it shows cosine similarity instead of attack success rate.

I /think/ the authors were going for universality meaning that representations are similar, separate from adversarial transferability. If this is the case, then there are a number of ways to measure the similarity of two networks. If the authors could show that slightly robust networks have representations that are closer to all the destination networks according to an independent metric, that would give the analysis more internal validity.

Typos:
- 126: “using linear interpolation” → “using bilinear interpolation”

Overall, the main results are interesting, but the issues in section 4 need to be addressed.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the targeted adversarial examples. They show that making the source classifier just a little robust to small-magnitude adversarial examples can improve the transferability of targeted attacks even between CNNs and transformers.
main_review: ## Originality

Though there exists a plethora of papers studying the transfer attacks, this paper provides a rather interesting perspective. The observation that making the source classifier just a little robust to small-magnitude adversarial examples can improve the transferability of targeted attacks even between CNNs and transformers is quite novel. This is motivated by the previous work (https://arxiv.org/abs/2102.05110) showing that slightly-robust features exhibit the universality principle and that slightly-robust networks can be used to increase transferability. This NeurIPS submission further provides a more comprehensive study investigating the transferability across a variety of architectures. 

In summary, the originality is not about the idea of using slightly robust features, but the comprehensive study and experiments, which provides a rather solid and novel perspective on more effective targeted transfer attacks. 

## Quality

The hypothesis that making the source classifier just a little robust to small-magnitude adversarial examples can improve the transferability of targeted attacks is reasonable and the empirical (both qualitative and quantitative) experiments support it. 

I quite enjoy Section 4 which tries to explain why class-targeted adversarial examples transfer via the lens of universality. They measure the similarity between the representations produced by different networks. 

However, I'm not fully convinced the conclusion regarding transformer classifiers is correct, as the authors only tried one particular transformer, ViT. One example is https://arxiv.org/abs/2104.05704, and the code can be found at https://github.com/lucidrains/vit-pytorch. Also, it might be more informative if they can provide the number of parameters for each model in Table 1. 

## Clarity

The paper is well-written and very easy to follow. 

One suggestion: the authors can indicate in the main submission that TMDI-FGSM is described in the appendix. 

## Significance

Although the universality principle has been introduced and utilized for training slightly-robust networks before, this submission provides more comprehensive experiments and analysis showing that they can be used for effective targeted transfer attacks.  This is a solid contribution to the study of adversarial attacks and many researchers interested in adversarial examples will find it insightful and useful. However, the tarted audience is also thus limited. 
limitations_and_societal_impact: The authors discussed the limitation and potential negative societal impact in Section 6.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper demonstrates that strong transferability across models can be achieved by using slightly-robust source networks through a comprehensive study and few analyses.

main_review: 
- Originality: 
 
  * This work seems to be a follow up study of [67] with a more comprehensive study, including analysis on CNN and transformer architectures, different hyperparameters, and representation similarity analysis etc. The novelty seems a bit limited.

- Clarity:
  * I found the paper a bit hard to read. 
    * There are several terms that are not precise or confusing to me, e.g., transferability, universal, $\epsilon$-robustness, slightly-robust, and overlap. I found it a bit confusing in the context of adversarial robustness as it could be unclear whether those terms are for attackers or defenders.
    * There are also several typos: e.g., L157 fined -> find. L266 testset -> test set.
    * I also found it confusing that there are several sentences using future tense. It is not clear to me when it is going to happen. 
    * I also found pages 7--8 a bit hard to read as there are not enough structures  but with many long paragraphs. 


- Quality:
  * The paper looks at 16/255 perturbation for imagenet models. As shown in [1], 16/255 is a very large perturbation radius and it could potentially yield visible class changes.
  * I am not sure about the previous literature on transfer attack. I found it a bit suspicious to rely on FGSM type of attacks only. I wonder whether autoattack or other pgd types of methods [2] could more reliably justify the robustness of the transfer models. 

- Significance:
  * I found the results and findings are interesting to know but it is mostly empirical. There is not enough explanation on why this phenomena happens, how this could be useful for defending transfer attacks, etc. The conclusion seems to expand too broad. 

[1] Daniel Zoran, et al.; Towards Robust Image Classification Using Sequential Attention Models, CVPR 2020, pp. 9483-9492
[2] https://arxiv.org/abs/1910.09338
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes that slightly robust neural networks contain universal features that are transferable across different classifiers trained on the same dataset. The authors demonstrate empirically that adversarial examples of slightly robust networks can transfer across network architectures relative to other levels of robustness. Moreover, the authors empirically show the universality of features in slightly robust neural networks.
main_review: **Originality**
The main idea of transferability in slightly robust neural networks appears mostly novel: as noted by the authors, [67] proposes the idea that slightly robust networks are transferable. However, [67] primarily focuses on showing the entanglement of non-robust and robust features- by contrast this work focuses primarily on transferability of slightly robust neural networks. The extensive experiments provide new insights and evidence beyond what is reported in previous literature. Specifically, the experiments reveal that the internal representations of slightly robust networks are more transferable, while prior works only considered transferability at the classifier output.

**Quality**
Experiments are generally extensive and convincing. However, one question that is inadequately addressed is what the significance of targeted vs untargeted attacks is with respect to transferability. Can the authors provide an explanation for why targeted attacks are more difficult to transfer?

**Clarity**
The paper is generally clearly written and well organized. However, some of the links with prior works are not made very clear. In particular, the paper discusses transferability of non-robust features, which is a concept also discussed in [30], for example. The authors may want to further discuss the nature of features in slightly robust networks in terms of being robust vs non-robust features, and highlight how their contributions differ from prior findings.


**Significance**
If the authors can provide answers to some unanswered questions as noted above, this work could provide significant insight into the nature of transferable features in neural networks. This may lead to the development of better adversarial attacks and defenses.
limitations_and_societal_impact: The authors adequately describe the limitations of the work in section 6.
review_title: null

Meta-review: This paper observes that classifiers with a certain degree of robustness lead to more transferable (targeted) adversarial examples. The manuscript shows how features learned by robust classifiers transfer better to other classifiers. This provides some further insights (building on https://arxiv.org/abs/2102.05110) into transferability, a phenomenon which remains poorly understood in the adversarial ML community. Thus, there is merit to the work proposed here. I encourage the authors to take into account discussions from the reviews while preparing the camera ready of their manuscript, and to open-source their code. In particular, the authors acknowledged that their discussion of universality is hard to follow, if not a bit circular, so I recommend following suggestions made in the author response to rewrite Section 4. Finally, the authors conducted their experiments using a single optimizer (see line 71), I would encourage them to confirm their findings on additional optimizers to obtain a more complete result.
Decision: Accept (Poster)