Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper tries to investigate and understand if and how adversarial training helps the models trained on the source domain transfer easier and faster to target domains. With extensive different configurations (such as fine-tuning strategies) in experiments, the authors show that robust models transfer better than natural models with less training data from the target domain. Also they demonstrate the intuition behind through experiments, such as capturing shapes than textures or using influence functions. 

Strengths
- The idea is interesting and have a potential for impacts in the community.
- Extensive experiments and investigations how and when the robust models works better than natural models is good to demonstrate the main ideas of this paper.
- Paper is easy to understand.

Weaknesses
- Even though it was shown by the experiments, it might need to have more theoretical understanding why the robust models transfer better or have better representations than natural models. 
- Even though it seems to provide some explanations, it lacks more thorough investigation why the specific configuration choices yield better performances than others.
- The presented dataset choices seem limited, which could limit its potential impacts and applications in real-world problems (see the comments below).

Detailed comments:
- If the shape is indeed more important than texture for human-like performance, is it possible make the model even works on par with the natural models? 
- Why specific configuration works better than others, such as fine-tuning three conv. blocks and ∥δ∥2 ≤ 3?
- In CIFAR-100 and (especially) CIFAR-10, fine-tuning one conv. block is better than zero conv. block and why?
- The target domains except CIFAR-100 and CIFAR-10 are all digit datasets, so its application to real-world problems may be limited. How about using different and non-overlapping classes than those in the source domain in other image datasets as target domains, such as CALTECH-256? It could make the paper stronger.
- In Table 5, the accuracy differences seem larger than ~5% as written in the text.

Typo:
Page 3: nx‘on-negative -> non-negative
review_title: Good paper to understand that robust models transfer better, but can be better.

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary of paper:** This paper investigates how "robust" (adversarially trained) models can improve the transfer of representations, finding that they transfer better. Additionally, they investigate some reasons this could be the case, examining the biases robust models appear to induce.

**Pros/strong points:**
 - interesting, well-explained experiments with mostly clear nice figures
 - nice extra investigations giving insight into the bias(s) conferred by adversarial training

**Cons/weak points:**
 - overstatement as though the results apply to any type of data/model, but only image data and convolutional nets are tried
 - potential issue with influence function experiments
 - no analysis of computational cost of adversarial training or other information on tradeoffs
 - background lacking/potential issues with related work

**Summary of review + recommendation:** Good paper with nice, thoughtful experiments, mostly well written, although I think the biggest issue there is over-claiming results applying to all data when only image data/convnets are studied. I'm also a bit worried about the thoroughness of the background research, and would like to see an analysis of the computational cost of adversarial training. If these and my question about influence function experiments are addressed, I would be happy to raise my score.

DETAILED REVIEW:

**Quality:**
 - Generally well written and organized, although the link between successive sections could be made a bit more strongly / flow could be better
 - Overstatement of scope (transfer learning in general, when only image data is studied) combined with misstatements in the first parts (about the origins of transfer learning) makes me skeptical of the depth of background research done and makes me suspect there may be some very relevant things which have not been surveyed/cited.  (see Specific questions/recommendations for details on both points, below)   
 - Comparison of computational cost / other tradeoffs in adversarial vs. 'natural' training not discussed
 - Potential weakness/misleading conclusions in influence function experiments if I've understood correctly (see specifics below)

**Clarity:**
 - The abstract and introduction would benefir from more technical, specific language to avoid ambiguity and establish flow of the sections
 - Clarity within sections is pretty good., some specific suggestions below.
 - Figures are mostly nice and clear, not so much Table 1 and 7a though.

**Originality:**
 - This is the largest potential problem that I am most unsure about. I'm very familiar with work on statistical learning theory and generalization overall, but I'm not an expert in transfer learning or adversarial method and I'm not sure how well these works are reviewed, so I'm not sure how novel this work is. The experiments are well done and well explained though, and I think this is a good contribution even if it is less original than it is made to seem due to the lack of context.

**Significance:**
 - Nice insights and interesting experiments for those wanting to understand the impact of robust training on transfer
 - Limited practical insights without an analysis/discussion of tradeoffs e.g. overall computational time and stability.
 - Directions proposed for future work are concrete and interesting


**Specific questions/recommendations:**
 - the first sentence of abstract and the title talk about DL generically, but the second-sentence is about images specifically. If you add non-image data, I'd suggest rephrasing to make the 2nd sentence generic, and maybe mention this technique has been particularly successful for images. However since all experiments are with images, I'd suggest making the title and abstract specific to that domain. e.g. "Evidence from image data that adversarially-trained deep nets transfer better" or "Adversarially-trained deep nets transfer to new images better"  
 - Overstatement of results: If you want the strong/general claims about transfer learning, I would strongly recommend doing experiments with at least MLPs in addition to convnets, and at least one other type of data in addition to  images. Otherwise the statements in title/abstract/intro should be circumscribed to more accurately reflect the nature of the experiments.  
 - mention what is adversarially vs. naturally trained. although I strongly suggest using a different term than 'naturally', including in diagrams/elsewhere as it is confusing. It makes it seem like you are comparing adversarial training to natural gradient methods.
 - first sentence putting data hogs in quotes misleadingly suggests that it's a commonly used phrase. Suggest removing this i.e. "they are known to require large..." (and suggest adding a reference which quantifies this rather than referring to hearsay).
 - "similarly, "stunning" is an opinion, suggest "remarkable" or something like "excellent" which can be derived from empirical results - Comparison of computational cost / other tradeoffs :even just a reference where this is done with a sentence summarizing those results / other tradeoffs, e.g. "it's usually at least 2x as slow and more likely to be unstable" or something like that might be enough, but I would prefer to see full training curves and computational cost numbers in appendix, with a line or two summarizing these in the main text.
 - Stating that Caruana (1995) proposed transfer learning seems incorrect to me. There was a NeurIPS workshop that same year on the subject, suggesting it was already an established term at that time. I don't know the full history, but from a quick google it seems to have been very common in psychology / education literature in the 70s and 80s, here's a book that talked about it in the context of ML in the 80s :https://pdfs.semanticscholar.org/b547/c5837bff9347dc76330a72fd7cbc517ee08c.pdf and here's Rich Sutton talking about it in 92: https://link.springer.com/content/pdf/10.1007/978-1-4615-3618-5.pdf
 - Related work:    - covariate shift could also mention risk extrapolation https://arxiv.org/pdf/2003.00688.pdf (how does the extrapolation done there differ from the adversarial training?)    - it seems like a lot of works on adversarial and contrastive training and the relationship to generalization are missing; I'm not an expert in this but starting way back hard negative mining and other contrastive methods (e.g. word2vec) have been used and their properties discussed 
 - The first subheading in section 4 is the conclusion drawn from that paragraph: "Adversarially-trained models transfer better and faster.", but subsequent headings do not have the same 'syntax' (they are more like titles than conclusions to be drawn). I like the conclusion-as-title format; I find it very engaging and helpful for skimming especially since there are many experiments. But most of all I would strongly suggest that all titles have the same 'syntax' i.e. if you can't think of conclusions-as-titles for the other bold p headers (although I think you can and should!), I would recommend rephrasing this one to be like the others (e.g. Comparison of adversarial and non-adversarial transfer)
 - formatting of table 1, with captions both below and above the tables, is hard to read. Put it all above or all below.
 - Could have been a nice opportunity to investigate whether robust models are more or less susceptible to the types of bias people worry about in real-world image datasets (e.g. face recognition); maybe worth mentioning this in future work 
 - Unless I'm mistaken, the experiments with influence functions do not distinguish the effect of performance from that of training (bias toward the "human prior"). To do so, the robust and natural methods would have to have the same accuracy (i.e. this might involve very early stopping of the robust method to match the natural model performance). Without this, the qualitative and quantitative results could just be due to the higher accuracy of the robust method, not the particular form of prior it induces.
review_title: Nice and clear with interesting experiments, but lacking context, a bit over-claimed, and a few other potential issues I'm optimistic can be addressed

final_score: rating: 6: Marginally above acceptance threshold
review_content: 1, Summary of contribution:
This paper claims that the pre-trained model trained adversarially can achieve better performance on transfer learning, and conducted extensive experiments on the efficacy of the adversely trained pre-trained models.
Also, the paper conducts an empirical analysis of the trained models and shows that the adversarially pre-trained models uses the shape of the images rather than the texture to classify the images. 
Using the influence function (Koh 2017), the paper reveals that each influential image on the adversarially trained model is much more perceptively similar to its test example.
 

2, Strengths and Weaknesses:
The paper is well-written and organized, and the experiments look fair and well support the claim.   The analysis is interesting and insightful. 
Meanwhile, the transfer is done to the domain of lower complexity, and some important comparative ideas are not extensively investigated. 


3, Recommendation:
While the paper’s empirical results are solid, there seems to be a substantial room left for comparative studies.  More ablation studies shall be done for other regularization methods.
I believe that the paper is marginally above the acceptance threshold. 

4, Reasons for Recommendation:
The reader will benefit more from the paper if the authors can justify their use of adversarial training as the regularization in the pretraining process.  I believe that this research warrants some comparative study for dropout, weight decay, as well as random perturbations.  I think the paper can be more insightful if it shows whether the other classical regularization performs better or worse on transfer learning than the proposed approach. 

5,  Additional feedback:
In addition to the suggestions made in 4, I also believe that comparison shall be made against the model trained without pretraining. 

\
---Post rebuttal---

Thank you for the response, and thank you for checking the performance comparison against the white-noise perturbation. It would be interesting to see a future work involving means other than Adversarial training (e.g. including other simple mechanisms like weight decay and dropout) to help reduce the overfitting effects in the pretraining phase. I would like to keep my score as is. 

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: ### Contributions ###
* The paper proposes that models that were adversarially trained transfer better to other datasets in that they increase _clean_ performance on this target dataset if there are only few labeled datapoints for the target task or only few training epochs are conducted
* The authors test their hypothesis for ResNets pretrained on Imagenet in different threat models and transfer these models to 6 different target datasets. Generally, results provide sufficient evidence for the paper's main hypothesis (robust models transfer better)
* Additional experiments provide evidence that better transferability of robust models is partly due to relying more on shape rather than texture cues. Moreover, an additional analysis using influence functions leads to the hypothesis that  robust neural networks might have learned to classify using example-based concept learning like in human beings.

### Significance ###
Transfer learning is a topic of high relevancy for practitioners since it can reduce both data label effort and training time. Improving upon the baseline of transferring models pretrained on Imagenet with a non-adversarial loss is thus a potential significant result. 
However, the paper's review of the transfer learning literature is superficial and misses some relevant related work such as "Rethinking ImageNet Pre-training" by He et al., ICCV 2019.  Additionally, Geirhos et al. (ICLR 2019) also showed that models pretrained on stylized ImageNet (and thus having a stronger shape bias) transfer better to object detection tasks. This should be mentioned in Section 5.  And more generally: if stronger transferability is mainly due to increased shape bias, wouldn't it make sense to pretrain explicitly for strong shape bias rather than achieving this indirectly via adversarial training as proposed in this paper?
A more thorough review of the transfer learning literature and relating the obtained results to this would generally strengthen the paper.

### Originality ###
The work is a purely empirical work studying the stated hypotheses, no novel methods are proposed. Originality can thus only come from the hypotheses.
The main hypothesis (robust models transfer better) was also proposed by Salman et al. (2020). However, this work should be seen as concurrent since it was released on arXiv only four months ago. 
The main prior work is Shafahi et al. (ICLR 2020), which also studies transferring adversarially pretrained models to other tasks. However, their focus is on the robustness gains of transferred models rather than on the effect on clean performance.
In summary, I think the main hypothesis studied in the paper is original. However, it is also clearly only a relatively small incremental step beyond what Shafahi et al. 2020 did.

### Clarity ###
Experimental setup, training pipeline, and analysis are outlined clearly. Releasing code for finetuning and replicating the experiments would further strengthen reproducibility

### Quality ###
Generally, the experiments are well conducted, covering a broad range of threat models, target datasets, training image and epochs regimes, and finetuning strategies. Additionally, Section 5 and 6 shed additional light onto why robust models might transfer better, and by this further strengthening the main message of the paper.
One shortcoming is that all target tasks are image classification tasks. Whether robust models also transfer better to task such object detection or semantic segmentation remains unclear. 

### Recommendation ###
In summary, I think the paper is a nice experimental study of a clearly stated hypothesis with potential practical impact. I thus lean towards acceptance, even though novelty is clearly borderline.

### Final Recommendation after Author Response###
The authors have addressed several of my main concerns. It would have been helpful to study transferability to tasks beyond image recognition, but overall, I think the paper has been considerably improved. I increase my score to 7.
Two remarks regarding new content:
 * I find it misleading to denote PGD as a targeted adversary and additive Gaussian noise as a random adversary Section 7. "Targeted" usually refers to an adversary that aims at achieving a specific misclassification (target class). Gaussian noise is not really an adversary, rather a distortion/image corruption. I would recommend clarifying the naming to avoid confusion of readers.
 * Is there any particular reason to use PGD(3) in Table 1b for evaluation? Would the effect hold also against stronger attacks (more iterations etc.)?

review_title: Review for "Adversarially-Trained Deep Nets Transfer Better"

Meta-review: The premise of the work is simple enough: investigate if networks that are trained with an adversarial objective end up being more suitable for transfer learning tasks, especially in the context of limited labeled data for the new domain. The work uncovers the fact that shape-biased representations are learned this way and this helps for the tasks they considered.

There was rather robust back and forth between the authors and the reviewers. The consensus is that this work has merit, has good quality experiments and investigates something with high potential impact (given the importance of transfer learning in general). I hope that most of the back and forth findings are incorporated in the final version of this work (especially the discussion and comparison with Shafahi et al., as well as all the nuances of the shape bias).
Decision: Accept (Poster)