Adversarially-Trained Deep Nets Transfer Better: Illustration on Image Classification
Authors: Francisco Utrera
Evan Kravitz
N. Benjamin Erichson
Rajiv Khanna
Michael W. Mahoney
Venue: ICLR 2021 Poster (2021)
Keywords: transfer learning
adversarial training
influence functions
limited data

Abstract: Transfer learning has emerged as a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains. This process consists of taking a neural network pre-trained on a large feature-rich source dataset, freezing the early layers that encode essential generic image properties, and then fine-tuning the last few layers in order to capture specific information related to the target situation. This approach is particularly useful when only limited or weakly labeled data are available for the new task. In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models, especially if only limited data are available for the new domain task. Further, we observe that adversarial training biases the learnt representations to retaining shapes, as opposed to textures, which impacts the transferability of the source models. Finally, through the lens of influence functions, we discover that transferred adversarially-trained models contain more human-identifiable semantic information, which explains -- at least partly -- why adversarially-trained models transfer better.

Published as a conference paper at ICLR 2021
ADVERSARIALLY-RRANER DEEP NETS TRANSFER BETTER: ILLUSTRATION ON IMAGE CLASSIFICATION
Francisco Utrera"
Evan Kravitz" UC Berkeley
N. Benjamin Erichson ICSI and UC Berkeley e www.cchenneeeeeeeccceee edu
UC Berkeley otrerf@berkeley edu
kravitz@berkeley edu
edu
Rajiv Khanna
Michael W. Mahoney
UC Berkeley ra rajivak@berkeley.ed
ICSI and UC Berkeley
mmahoney@sta: wtt.ttttt.eeeeeeee..o
ABSTRACT
Transfer learning has emerged as powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains. This pro- cess consists of taking neural network pre-trained on a large feature-rich source dataset, freezing the early layers that encode essential generic image properties, and then ainettunigg the last few layers in order to capture specific information re- lated to the target situation. This approach is particularly useful when only limited or weakly labeled data are available for the new task. In this work, we demonstrate andversarialllytrainee
that seveessaiilll-ttainne models, especially if only limited data are available for the new domain task. Further, we observe that adversarial training biases the learnt representations retaining shapes, as opposed to textures, which impacts the transferability of the source models. Finally, through the lens of influence functions, we discover that transferred eeersarially-trainne models contain more -innt-iieeniifialle seman- tie information, which explains - at least partly why adversarially-traiee mod- els transfer better.
models transfer better than
INTRODUCTION
While deep neural networks (DNNs) achieve state-of-tth--at performance in many fields, they are known to require large quantities of reasonably high-quality labeled data, which can often be ex- pensive to obtain. As such, transfer learning has emerged as a powerful methodology that can significantly ease this burden by enabling the user to adapt pre-trained DNN range of new sit- uations and domains (Bengio 2012: Yosinski et al. 2014). Models that are pre-trained on ImageNet DDng al. 2009) have excellent transfer learning capabilities after fine-tuning only few of the last layers (Kombllihh et al. 2019) on the target domain.
motivated by the observation that humans apply previously learned knowledge to solve new problems with ease (Caruana 1995). With this motivation, learning more source tasks and apply the knowledge to target task (Pan & Yang 2009). The main benefits include a reduction in the number of required labeled data points in the target domain (Gong et 2012 Pan & Yang, 2009) and reduction in training costs as compared to training model from scratch. However, in practice transfer learning remains an "art" that requires domain expertise to tune the many knobs the transfer process. An important consideration, for example, is which concepts or features are transferable from the source domain to the target domain. The features which are unique to a domain cannot transferred, and so an important goal of transfer learning is to hunt for features shared across domains.
Early work in transfer learning
aims to extract knowledge from
has recently been shown that adversarially-traine trained models (henceforth denoted as robust models) capture more robust features that are more aligned with human perception, compared to the seem- ingly patternless features (to humans, at least) of standard models (Ilyas et al. 2019). Unfortunately,
Equull contribution

---

Published as a conference paper at ICLR 2021
(a)
(b)
1. Train Source Model on ImageNet
2. Fine-tune
3. Accuracy
Seed
Robust
Natural
(FT) on Target Dataset
on Target Dataset
Robust
Train
FT Robust
Farger Dataset Test
Better
ImageNer
Target Dataser
FT
Natural
Baseline
Natural
We demonstrate that adversarially-traines (i.e., robust) DNNs transfer better and faster to new domains with the process shown in (a): A ResNet50 is trained adversarially or non-adversariall) (i.e., naturally) on the source dataset. Then, we fine-tune both of these source models on the target dataset. We hypothesize the robust features in robust models that encode more humanly percep- tible representations, such as textures, strokes and lines, as seen in (b), are responsible for this phenomenon. See Appendix A.I I.
Figure I:
for details on how generated the images in (b).
these models typically have lower generalization performance on the source domain, as compared to non-ad non-adversarially -rrainnne trained (henceforth denoted as natural, as in previous works (Tsipras et al., 2019 SSafahi al. 2019) Salman et al. 2020)) model. Hence, [[yys et al. (2019) hypothesize that non-robust features that are lost during adversarially training may have a significant positive impact on generalization within given dataset or domain. This inherently different feature representation between models constructed with adversarial training and models trained with standard methods would also explain why accuracy and robustness are at odds (Tsipras et al.) 2019). This leads to the question of whether models that robust representations generalize better across domains. This the main question we address.
In this work, demonstrate that robust models transfer better to domains than natural models. To demonstrate this, we conduct an extensive number of transfer learning experiments across mul- tiple domains (i.e., datasets), with various numbers of fine-tuned convolutional blocks and random subset sizes from the target dataset, where the critical variable is the constraint used to adversarially train the source model. (Described in detail Sections 3 and Appendix A..3) Importantly, note that we do not use an adversarial training procedure for the actual transfer learning process. Our findings indicate that robust models have outstanding transfer learning characteristics across configura- tions, where we measure the performance in terms of model accuracy on target datasets for varying numbers of training images and epochs. Figure | provides a summary of our approach.
Our focus in this work is to show that robust source models learn representations that transfer better new datasets on image recognition tasks. While adversarial training was proposed combat adversarial attacks, our experiments discover an unintended but useful application. Adversarial training retains the robust features that are independent of the idiosyncrasies present in the source training data. Thus, these models exhibit worse generalization performance the source domain, but better performance when transferred. This observation novel, and we undertake extensive empirical studies to make the following contributions:
We discover that adversarially trained source models obtain higher test accuracy than natu- source models after fine-tuning with fewer training examples on the target datasets and over fewer training epochs.
We notice that the similarity between the source and target datasets affects the optimal number of fine-tuned blocks and the robustness constraint.
We show that adversarial training biases the learned representations to retain shapes instead of textures, impacting the source models' transferability.
We interpret robust representations using influence functions and observe that adversarially-train source models better capture class-level semantic properties of the images, consistent with human concept learning and understanding.
2

---

Published as a conference paper at ICLR 2021
2 RELATED WORKS
ImageNet transfers. Our focus is on studying the transfer of all but the last few layers of trained DNNs and fine-tuning the last non non-transferred layers. For ease of exposition, we restrict our attention to ImageNet models (Deng et al. 2009). Kornblith et al. (2019) study the transfer of natural models to various datasets and is thus a prequel to our work. Yosinski et al.] (2014) also study transferring natural models but focus on the importance of individual neurons on transfer learning. |Recht et al. (2019) study the generalization of natural and robust models to additional data generated using a process similar to that of generating ImageNet. They conclude that models trained on ImageNet overfit the data. However, they study the models' generalization as-is without fine-tuning.
Covariate shift. A significant challenge in transfer learning is handling the data distribution change across different domains, also called covariate shift. It's widely recognized in successful domain adaptations (Yosinski al. 2014]] Glorot et al.) 2011) that the representations in earlier layers more " "generic" and hence more transferable than the ones in later layers. This hierarchical disen- tanglement is attributed to the properties of the data itself, that the later layers are more closely associated with the data and do not transfer as well. This motivated studies for shallow transfer learning (Yosinski al., 2014; Ghifary al.) 2014) and more general studies to extract features that remain invariant across different data distributions (Arjovsky et al. (201)). In Section S we see that adversarial training biases the learned representations to retain shapes instead of textures, which may be more desirable invariant across the datasets.
Transfering dddversarilll-trrinne models. There are mainly two works directly associated with ours. First, subsequent to this paper's initial posting (in a non-anonymized form in public forum), Salman et al. (2020) posted a related paper. They arrived at broadly similar conclusions, confirming our main results that robust models transfer better; and they do so by focusing somewhat different experiments, c.g., they focus on the effects of network architecture width, fixed feature transfer, and seeing if models without texture bias transfer better than robust models. Second, Shafahi eet al. (2020) mainly find that models lose robustness as more layers are fine-tuned. It might seem to contradict thesis that they also notice that ImageNet robust model with || I 5 constraint has lower accuracy on the target datasets, CIFAR-10 and CIFAR-100, compared to a natural ImageNet model. However, we show that the robust model transfers better than the natural one when we use a ||0||2 < 3 constraint to adversarially train the source model.
There has been significant interest in interpreting blackbox models using salient examples from the data. A line of research focuses on using influence func- tions (Koh & Liang] 2017 Koh et al., 2019 Khanna et al., 2019) to choose the most indicative data points for a given prediction. In particular, Khanna et al. (2019) discuss the connection of influence functions with Fisher kernels; and Kim et al.] (2016) propose using criticisms in addi- tion to representative examples. Complimentary lines of research focus on interpretability based on human-understandaall concepts (Bau et al. (017) and feature saliency metrics (M. Ancona 2017).
Example based interpretability.
3 BRIEF OVERVIEW OF THE ADVERSARIAL TRAINING PROCESS
Adversarial training modifies the objective of minimizing the average loss across all data points by first maximizing the loss produced by each image with a perturbation (i.e., a mask) that may not exceed a specified magnitude. Here, we describe this process, similar to Madry et al. (2018).
Let (xi, yi) be m data points for i E [m]. where I; E Rd is the 7th feature vector, and yi E 3 is the corresponding response value. Typically, we model the response as a parametric model he : Rdd + y with a corresponding loss function é: y x y + R20. The objective is minimize the loss {(j. y), where ÿ he (x) is the predicted response. Adversarial training replaces the above minimization problem of training the model by a minimax optimization problem to make the model resilient to arbitrary perturbations of inputs. The goal of adversarial training is to solve problem of the form
min m
max ((ho(x; + 8.). 1
(1)
That is, the goal is to find the parameters of the model hp that minimize the average maximum loss obtained by perturbing every input v, with d; constrained such that its C, norm does not exceed
3

---

Published as a conference paper at ICLR 2021
some non-negative c O, then 6, 0, in which case there is no perturbation to the input, which what we call natural training. As increases, the magnitude of the perturbation also increases. For more details on how solve this problem, and few examples, Appendix A.2
TRANSFERRING ADVERSARIALLY-TRAA
MODELS
In this study, we train four ResNet50 source models on ImageNet. We train one of them naturally (non-adversarially), and train each of the remaining three adversarially with one of the following constraints: (i) ||ö]|2 I 3, (ii) || S 255 (iii) ||0|oo S 255 Next, we fine-tune some convo- lutional blocks in the source models to each of the six target datasets separately using a subset of the training data. We repeat each of these trials for various seed values and report the mean and 95% confidence interval. Altogether, that considers four ImageNet source models, four fine-tuning configurations, six target datasets, ten random subset sizes, and average of fifteen random seeds for total of 14,400 fine-tuned models. For more details, see Appendix and A.4
A.
have comprehensive and replicable experimental setup
(a)
(b)
GFAR-100
75
GFAR-10
50
Accuracy
60
50
FMNIST
25
Test Accuracy
15
KMNN ST
MNIST
100
90
80
70
100
400
1600
6400
23600
100
400
1600
6400
25600
400
1600
6400
25000
100
200
400
900
1600
3200
6400
12800
25e0d
50000
Number of training images
Number of training images
Figure 2: Robust models generalize hetter to new domains, especially with fewer training images in the target domain. (a) Shows the test accuracy the six target datasets (color-coded in (b)) for various subset sizes. (b) Shows the test accuracy delta, defined as the robust model test accuracy minus natural model interval. Both the robust and natural models are ResNet50s' trained on ImageNet. The robust model uses ||81|2 < constraint. Both models fine-tur three convolutional blocks on the target dataset.
accuracy. The solid line is the mean and its shade is the 95% confidence
(a)
(b)
60
90
90
CIFAR-100
50
85
60
CIFAR-10
25
Accuracy
40
80
60
FMIIST
KMNIST
50
90
90
98
96
10
Test
70
94
92
60
23
81
150
Number training epochs
Number of training epochs
Figure 3: Robust models transfer faster. (a) Shows the test accuracy during the fine-tuning process on the six target datasets (color-coded as in (b)). (b) Shows the test accuracy delta, defined as the robust model test aceuracy minus the natural model test accuracy. The solid line the mean and its shade is the 95% confidence interval. Both the robust and natural models are ResNet50s' trained on ImageNet. The robust model uses ||00/ S 3 constraint. Both models fine-tune three convolutional blocks using random subset of 3,200 images (~ 5%) of the target dataset.
4

---

Published as a conference paper at ICLR 2021
Ad adversriillly-taiinee models transfer better and faster. For ease of comparison, we select the robust and natural models that transfer with the highest test accuracy across all datasets (fine- tuning three convolutional blocks and the robust model using the |öا|| S 3 constraint), shown in Figures 2 and See Appendix [A.5 for additional results. Figure 2 shows that the test accuracy delta between robust and natural models is above zero for all six target datasets. Thus, robust models obtain higher test accuracy on the target dataset than the natural model, especially with less training data in the target domain. Robust models also learn faster, as shown by the positive test accuracy delta in Figure B]b for all target datasets after only 11 and 21 fine-tuning epochs. See Appendix AA.6 for additional information on different random subset sizes. Fine-tuning cost is the same for both robust and natural models, but training the source model is considerably more expensive. For more detail on computational complexity see A.8 Also, our code available at ea ri.ninnngiit
https: //g http:///....tteeeerrootttttttta.
Best results achieved with &g constraint and fine-tuning one to three convolutional blocks. Ro- bust models achieve the highest test accuracy the target datasets when an optimal number of con- volutional blocks are fine-tuned, and when these models are trained with an appropriate constraint type. In particular, fine-tuning zero (only the fully-connected layer) nine convolutional blocks leads to lower test accuracy than fine-tuning one or three blocks, as shown in Figure 4a) for all six target datasets. The natural model and the other two robust models exhibit the same behavior, as shown in Appendix [.7]] To analyze the best constraint type, we select the fine-tuning configuration that yields the highest test accuracy on the target datasets (fine-tuning three convolutional blocks). We see that the &2 constraint outperforms the Loo constraint, as shown by the positive accuracy delta between the Eg and Lo models in Figures d) and (c), respectively.
Similarity effect on transfer learning configurations.
Besides noticing that robust models achieved better performance on the target dataset than natural models, we also observe trends in how well they transfer to different datasets. When transferring from ImageNet, we find that CIFAR- 10 and CIFAR- 100 have interesting transfer properties, compared to the other datasets. In particular, even though all other datasets transfer better when fine-tuning one or three blocks, it seems that models transfer better to CIFAR-10 and CIFAR- 100 when fewer blocks are fine-tuned, as shown in Figure b). This suggests that because these datasets are close to ImageNet, fine-tuning of early blocks is unnecessary Yosinski et al. 2014). Along similar lines, it is better to use a smaller c for CIFAR-10 and CIFAR-100 datasets than the other datasets when transferring from ImageNet, as seen from Figure (ic). This is because a larger perturbation would destroy low-level features, learned from ImageNet, which are useful to discriminate between labels in CIFAR-10 and CIFAR- 100. Finally, for datasets that are most distinct from ImageNet (SVHN and KMNIST), we find that robustness yields the largest benefit classification accuracy and learning speed, as seen in Fig- ure (bb and Figure Bib), respectively. These discrepancies are even more noticeable when smaller fractions of the target dataset are used.
(a)
(b)
100
75
Test Accuracy Delta %
Accuracy
25
++00
50
40
25
0-
100
0.
0000
80
60
+88
60
1680
-5
OFAR-100
OFAR-10
KMNIST
FMNIST
25
40
-10
SVHN
MNIST
100
400
1600
6400
25500
100
400
1600
6400
23600
100
400
1600
6400
25600
200
200
400
900
1600
3200
6400
12800
25600
50000
Number of training images
Number of training images
Figure 4: The optimal number of fine-tuned blocks is somewhere between one and three. (a, b)
Shows the test accuracy of the robust model trained on ImageNet using the ||0]| S 3 constraint with various numbers of fine-tuned blocks (0. 1, 3, or 9). (a) Shows the test accuracy on each of the six target datasets (color-coded as in (b)). (b) Shows the test accuracy delta, defined as the test accuracy of the model with three fine-tune tuned blocks minus the test accuracy of the model with one fine-tuned block. The solid line is the mean and its shade is the 95% confidence interval.
5

---

Published as a conference paper at ICLR 2021
5 BIAS TOWARDS RECOGNIZING SHAPES AS OPPOSED TO TEXTURES
In this section, we explore the effect of texture and shape bias, as described by Geirhos et al. 2019). on the robust models' transferability. As pointed out by Geirhos al. (0119) natural models are more biased towards recognizing textures than shapes. This is in stark contrast to the human bias of recognizing shapes over textures (Landau al. 1988). However, Engstrom et al. (2019) showed that robust models encode humanly-aligned representations, and we observe (e.g see Figure Tb) that these representations persist even after fine-tuning on CIFAR-10.
Seversarially-trainne models are less sensitive to texture variations. Table la shows that the robust model outperforms the natural one when only tested on Stylized Imagenet (SIN) and also after fine-tuning only the last connecco layer SIN. Both models are ResNet50s pre-trained on ImageNet (IN), and the robust model uses a | I 3 constraint.
Models trained standard and stylized ImageNet are less sensitive to adversarial attacks. Table Tb shows that the ResNet50 model trained on both IN and SIN (IN+SIN) outperforms the models trained on just IN PGD(3) adversarial test accuracy on IN for various levels.
Adversarially-traiine models are biased towards low resolution and low frequencies. We observe that the transferability of robust models is also affected by two input perturbations that destroy, or at least damage, textures. Namely, lowering the resolution images and applying low pass filters. To demonstrate this, use the Caltech101 dataset (Li Fei- et al.) 2004). This dataset has 101 labels with 30 high-resolution (224x224 pixels or more) images per label. The results in Table 2 support our conjecture that robust models use shapes more than textures for classification by
(a)
(c)
(d)
12
CIFAR- 100
FMNIST
TANUST
CIFAR
KHNIST
101
Test Accuracy Delta
-SHN
MNIST
Test Accuracy Delta
Test Accuracy
(b)
3-
(e)
0-
10 x3
-3
50
100
200
600
Ruo
1600
3200
6d00
12800
25600
50/00
100
200
400
800
1600
3200
6400
12800
25600
50000
100
200
400
800
1600
3200
6400
12800
25500
50000
Number of training images
Number of training images
Number training images
Figure 5: Shows the test accuracy on target datasets, (a) CIFAR-IO and (b) SVHN, of three robust
models. (c) Shows that source models trained with lower transfer better to target domains that are
similar to ImageNet, as shown by the test accuracy delta between the | I 253 model and the |1 < 8
205 model. (d, e) Shows the test accuracy delta between the Eo norm with 3 and each 355 in (d), and outperforms both of the Ex constraints.
of the too models:
255 in (e). In both (d, e) the la norm constraint
Table 1: Adversarially-trainne models are less biased towards recognizing textures than natural ones. Models trained on both ImageNet and Stylized ImageNet are more robust to adversarial attacks.
(a) Shows SIN test accuracy of robust and natural IN source models before (i.e., SIN (Test)) and af- ter fine-tuning (i.c., SIN (FT)) three convolutional
(b) Shows IN PGD(3) adversarial test accuracy with various magnitudes for ResNet50 models naturally-trained (non-adversarially) on IN, SIN, and both IN and SIN.
blocks. Robust constraint: ||0]|2 S 3.
Model
SIN (Test)
SIN (FT)
(Test)
IN
SIN
IN+SIN
Robust
20.1
64.2
3/32
52.1
39.6
53.9
Natural
11.4
36.1
3/16
34.7
25.5
38.5

---

Published as a conference paper at ICLR 2021
showing that the robust model obtains higher test accuracy, in both the low- low-resolution and low-pass versions of Caltech101, than the natural one.
INTERPRETING REPRESENTATIONS USING INFLUENCE FUNCTIONS
In this section, we use influence functions (ooh & Liang 2017) to show that robust representa- tions hold semantic information, i.e., robust DNNs classity images like human would, through similar-looking examples, Engstrom al. (2019) observed that moving the image in carefully cho- sen directions the latent space allows for high-level human-unddersaaddabe feature manipulation in the pixel space. They suggest that the bias introduced by adversarial training as a human prior on the representations, so that these representations are extractors of high-level human-interpretable features. It has long been established that humans learn new concepts through concept-rerresentative or lnoilar-looking examples (Cohen et al. 1996] [Nweel] 1972). Our focus in the present work is study whether these representations aid the neural network to learn new concepts (namely image labels) akin to how humans learn concepts.
be viewed
To study this, we use influence functions as described by Koh Liang 2017 (see Appendix A.9 for an overview). For each test image in the CIFAR-10 dataset, influence functions allow us to answer the following: What the influence of each training image on the model prediction for given test image? We ask this question for both the robust and natural models, and compare the results. In our experiments, we fine-tune the last three blocks with the same 3,200 randomly selected training images. Also, the robust model uses ||0||2 S 3 as the constraint.
Adversarially-trained models have more similar-looking influential images. Figure shows that the robust models' most inffuential image is often more perceptibly similar to the test image than the natural models' most influential image. Consider, for example, the test image of the blue (on the second column). The robust models' corresponding top influential training image is similar-looking blue car, while the natural model has a red truck. As second example, the robust
low resolution and low pass filtered variants of Caltech101. This table shows the test accuracy after fine-tuning three convolutional blocks of the robust model (115l22 3) and the natural model a low resolution (Low-Res) low pass filtered (Low-Pass) version of Caltech101. We lowered the resolution to 32x32 and zeroed the top 1,024 (~~ 14%) frequencies, respectively.
Table 2: Robust models transfer better than natural
Model
Low-Res
Low-Pass
Robust Natural
83.7
85.3
79.1
80.7
Plane
Car
Bird
Deer
Prog
Hoese
Truck
Test
Robust
Natural
Figure 6: Adversarially-trancc (i.e, robust) models have more similar-looking influential images in the target dataset than the non-adversarially -esirrallllyyytt -rainndd (i.c., natural) model. The top shows randomly selected test image, for each of the ten categorics. The middle and bottom rows display the most influential images, for the robust and natural models, respectively.
7

---

Published as a conference paper at ICLR 2021
(a)
(b)
10000
Match target % (train)
80
rebust
Test Image Index
8000
70
natural
6000
60
4000
50
40-
2000
30
1000
3000
1000
2000 3000
60 Influential training image ranking
100
Train image Index
Figure 7: Top influential image labels in the robust model match test image labels more often than in the natural model. (a) Shows the influence values (standardized by their matrix norm) of each training image on each test image, sorted by their label as in Figure [6] for both the natural (left) and robust (right) models. (b) Displays the percentage of times that the label of the top-k influential image in the training set matches the label in the test image being evaluated.
models' top influential training image for the orange truck (on the far right) is a similar-looking orange truck, while the natural model has blue and white truck.
Inftuential image labels match test image labels more often in adversarially To quantify the visual similarity described above, we show the influence values (standardized by their matrix norm) of each training image on each test image, sorted by their label as in Figure 6 for both the natural (left) and robust (right) models in Figure(7[a) Darker and better-defined blocks across the diagonal signal that the influence values are more consistent with the test image label index in the y-axis because darker colors represent higher influence values. The robust model (right) has a slight advantage over the natural model.
models.
Figure 7b)) further accentuates the difference between the robust model and the natural model. It displays the percentage of times that the label of the top-k influential image in the training set matches the label in the test image evaluated. To better understand this figure, consider the leftmost point in Figure ( for both models. This point represents the proportion of the training images corresponding to the darkest dots in each horizontal line (i.e., top-1 influential training image) in (a) that matc the label of the given test image, for robust and natural models separately. 78.6% of the robust model's top-1 influential images match the label of the given test image vs 55.1% for the natural counterpart. We also consider the case when the category of at least three of the top-5 influential training images match that of the test image. This happens in 77.3% of the cases for the robust model, but only for 53.8% of the cases for the natural model. This vast gap is not explainable solely from only ~5% difference in target test accuracy, shown in Table 7 in Appendix A.S
From the qualitative and quantitative analysis, we that the robust model has learned represen- tations with more -imaa-iientifiable semantic information than the natural model, while the latter relies on less interpretable representations. In other words, the robust neural network has learned the image labels by creating strong associations to semantically-simite examples (akin to example- based concept learning human beings) its internal representations. Thus, reinforeing the human prior bias hypothesis in robust representations observed by [Egstrom et al. (2019).
7 Do OTHER ADVERSARIAL ATTACKS IMPROVE TRANSFERRABILITY?
Prior works show that there is connection between the sensitivity of neural network to Gaussian noise and its robustness to adversarial perturbations (Weng et al. 2018 Gilmer et al. 2019). It has also been suggested that Gaussian perturbations can improve or even replace adversarial training (Kannan al., 2018). Further, it has been shown that often only few PGD iterations sufficient to obtain a robust model (Madry et al. 2018 Shafahi et al. 2019 Wong et al. 2020).
To better understand these trade-offs, in this section we further explore the transferrability of models trained on ImageNet with random Gaussian noise and one-step of PGD (i.e., PGD(()) using the same methodology described in Section 4 For all models, including the Gaussian one, we contraint the
8

---

Published as a conference paper at ICLR 2021
(a)
(b)
(c)
25
CIFAR-10U
CIFAR-10
Accuracy Delta
20
SVHN
15
15
KMNIST
MNIITT
10
10
10
5
5
0-
0-
200
800
3200
12800
50000
200
800
3200
12800
50000
200
800
3200
12800
50000
Number of training images
Number of trainingg images
Number of training images
Figure 8: Shows the test accuracy delta, defined the transfer test accuracy of the souree models trained with PGD(20) in (a), PGD(1) in (b), and Gaussian in (c), minus the naturally-trained model. Comparing (a) and (b) shows that more attack steps (i.e., PGD(20) vs PGD(1)) slightly improves transferability. Comparing (a) and (b) to (c) shows that a targeted adversarial attacks (i.e. PGD vs Gaussian) significantly improves transferability relative to random ones. (c) shows us that a random adversarial attack can improve transfer accuracy.
perturbation, namely 8, to be ||61|2 < 3 in order make a fair comparison across models. See A.10 for more details our experimental setup.
In the following we discuss our experimental results, summarized in Figure 8 which shows the test accuracy delta of each of the three adversarialll-ttrinee models versus the natural one.
Training with more steps is marginally better. Figure 8 (a) and (b) show that more PGD iterations (i.e., PGD(20) vs PGD(1)) slightly improve transferability. This is evidenced by the slightly higher test accuracy delta in (a), relative to (b) across all target datasets. Our emperical result agrees with previous works that are showing that more attacker steps typically only improve adversarial robustness slightly (Madry et al. 2018 Shafahi et al. 2019) Wong et al. 2020).
targeted adversary better than random one. Comparing Figure (a) and (b) to (c) shows that a targeted adversarial attacks (i.e. PGD vs Gaussian) significantly improves transferability relative random perturbation. This evidenced by the slightly higher test accuracy delta in (a), and (b) relative to (c) across all target datasets.
random adversary better than no adversary. Figure shows us that random adversarial attack can improve transferability. This is evidenced by the significantly positive accuracy delta in (c) across all target datasets. Our results agree with prior works showing that training a model by perturbing inputs with Gaussian noise can improve adversarial robustness (Kannan et al. 2018
CONCLUSION AND FUTURE WORKS
We show that robust models transfer very well to new domains, even outperforming natural models. This may be surprising since robust models generalize worse than natural models within the source domain, and since they were originally designed to protect against adversarial attacks. We show that robust DNNs can be transferred both faster and with higher accuracy, while also requiring fewer images to achieve suitable performance on the target domain. We observe that adversarial training biases the learned representations to retain shapes instead of textures, which impacts the source models' transferability. We also show that the improved classification accuracy is due to the fact that robust models have an implicit bias that enables them to comprehend human-aligned features. Given the widespread use DNNs, there is great potential for robust networks to be applied to a variety of high-tech areas such facial recognition, self-driving cars, and healthcare, but understanding the issues we have addressed crucial to deliver upon that potential. Please see Appendix A. for details on future works.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We are grateful to the generous support from Amazon AWS and Google Cloud. NBE and MWM would like to acknowledge IARPA (contract W911NF20C0053 NSF, ONR and CLTC for provid- ing partial support of this work. Our conclusions do not necessarily reflect the position the policy of our sponsors, and no official endorsement should be inferred.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. CoRR, 0tss/1977.02892. 2019.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretabilityy of deep visual representations. In Computer Vision and Pattern Recognition (CVPR), 2017.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Inter- national Conference of Machine learning (ICML), 2012.
Rich Caruana. Learning many related tasks at the same time with backpropagation. In Neural Information Processing Systems (NeurIPS). 1995.
Marvin S. Cohen, Jared Freeman, and Steve Wolf. Metarecognition in time-stressed decision making: recognizing, critiquing, and correcting, Human Factors, 1996.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR), 2009.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness prior for learned representations, 2019.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. International Conference on Learning Representations (ICLR), 2019.
Muhammad Ghifary, W. Bastiaan Kleijn, and Mengjie Zhang. Domain adaptive neural networks for object recognition. In Pucific Rim International Conferences Artificial Intelligence (PRICAI), 2014.
Justin Gilmer, Nicolas Ford, Nicholas Carlini, and Ekin Cubuk. Adversarial examples are natural consequence of test error in noise. In International Conference on Machine Learning (ICML), 2019.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sentiment classification: A deep learning approach. In International Conference on Machine Learning (ICML), 2011.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In Computer Vision and Pattern Recognition (CVPR), 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Computer Vision and Pattern Recognition (CVPR), 2016.
Andrew Ilas,, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Neural Information Processing Systems (NeurIPS), 2019.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing, 2018.
Rajiv Khanna, Been Kim, Joydeep Ghosh, and Oluwasanmi Koyejo. Interpreting black box predic- tions using fisher kernels. In Artificial Intelligence and Statistics (AISTATS), 2019.
10

---

Published as a conference paper at ICLR 2021
Been Kim, Rajiv Khanna, and Oluwasanmi Koyejo. Examples not enough, learn to criticize! criticism for interpretabilityy Neural Information Processing Systems (NeurIPS), 2016.
Pang Wei Koh and Percy Liang, Understanding black-box predictions via influence functions. In International Conference Machine Learning (ICML), 2017.
Pang Wei Koh, Kai-Siang Ang. Hubert Teo, and Percy Liang. the aceuracy of influence functions for measuring group effects. In Neural Information Processing Systems (NeurIPS). 2019.
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better? In Computer Vision and Pattern Recognition (CVPR), 2019.
Barbara Landau, Linda Smith, and Susan Jones. The importance of shape in early lexical learning. Cognitive development, 3(3):299-321, 1988.
Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training exam- ples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPR), pp. 178-178, 2004.
A. C. Öztireli M. Gross M. Ancona, E. Ceolini. A unified view of gradient-based attribution methods for deep neural networks. In Neural Information Processing Systems (NeurIPS) workshops, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Allen Newell. Human problem solving. Prentice-Hall, USA, 1972.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. Transactions on knowledge and data engineering (TPAMI), 2009.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In International Conference on Machine Learning (ICMI), 2019.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? arXiv preprint arXiv:2007.08488, 2020.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint arXiv:2008. 04859, 2020.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Neural Information Processing Systems (NeurIPS), pp. 3358-3369, 2019.
Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David Jacobs, and Tom Goldstein. Adversarially robust transfer learning. In International Conference on Learning Rep- resentations (ICLR), 2020.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representa- tions (ICLR), 2019.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. From imagenet image classification: Contextualizing progress benchmarks. In International Con- ference on Machine Learning (ICML), 2020.
Tsui-Wer Wei Weng, Huan Zhang, Pin- Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. Evaluating the robustness of neural networks: An extreme value theory approach. In International Conference on Learning Representations, 2018. httpttss::/ //tp:////peeevvveeeeee
net / Eorum?id 2orum?iddBBUHLMM20 ObU
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training.
In International Conference on Learning Representations (ICLR), 2020.
11

---

Published as a conference paper at ICLR 2021
Baoyuan Wu, Weidong Chen, Yanbo Fan, Yong Zhang, Jinlong Hou, Jie Liu, and Tong Zhang. Tencent ml-images: A large-scale multi-label image database for visual representation learning. IEEE Access, 177772683-772699 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- 077747, 2017.
ing machine learning algorithms. arXiv preprini arXiv: (ann::7788077777
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Neural Information Processing Systems (NeurIPS), 2014.
A APPENDIX
A.I DETAILS ON FIGURE I(A)
We generated Figure I(a) by maximizing the activations in the penultimate layer on out of distri- bution images (i.e., images that following the "feature visualization" technique as implemented in Engstrom et al. (2019). The natu- ral and robust models are pre-trained ResNet-50 models on ImageNet after fine-tuning only the last fully-connected layer on CIFAR-10. The robust model used the ||01|2 S 3 constraint.
part of the training and test data) our starting seeds,
A.2 MORE DETAILS ON ADVERSARIAL TRAINING
In practice, we solve equation| using stochastic gradient descent (SGD) over 0. More concretely, include random sample of training examples in a set B, specify non-neee non-negative learning rate a, calculate the gradient with respect to the parameters the model Vg. and update as follows:
Vo max E(ho, (I; + 61), ys).
16:1,
(Z..D:)
This training process has sequential nature: it first finds the worst possible perturbation o for each training example (x,, y/) in B before updating the parameters of the model 0, where
8 arg max (ho + 65),
Problem equation 3 is typically solved using projected gradient descent with k update steps, which what we call PG)(k). this work, we use PGD(20), which means that we take 20 update steps to solve (3). Each step iteratively updates 5, by projecting the update onto the lp ball of interest:
6; = P(;; + a E(ho + +d),,
As an example, consider the case of the & norm and let (ō,) ((ho(x; + v1). If we want to meet the restriction that ||0 l2 e, can pick update value for ó, whose Lg norm will be at most the learning rate. This yields the problem:
Vo,((8.)
arg max v V., = 'V,, (66))l2 S0
(5)
And in the case of the l.o norm we have
arg max o (o. = sign( V., f(.))) Sa
Thus, we set the learning rate to be equal that we reach the boundary condition for 6;. Also we must clip 6, according to the Ep norm in case that it exceeds the boundary condition.
num. of steps for 1.5 in order to ensure
A.3 TRANSFER LEARNING EXPERIMENTAL SETUP DETAILS
Source models. For all of our experiments, we use four residual networks (ResNet-50) (He et al.) 2016) pre-trained the ImageNet dataset (Deng et al. 2009), naturally-trained (without
12

---

Published as a conference paper at ICLR 2021
an adversarial constraint), and the others use PGD(20) and the following adversarial constraints: (1) S where is a matrix that contains represents the pertur- bation applied to the input image described in Section B equation (I). In addition to the natural ResNet-50, considered as baseline, we also use three robust networks with various constraints. For speed, transparency and reproducibility, do not re-train the source models ourselves) |
4 ||6||2 I 3, (2) |] loc S 255 (3) ||ō]|ce I 255
Fine-tuning procedure. To transfer our models we copy the entire source model to the target model, freeze the all but the last & convolutional blocks, re-initialize the last fully-connected (FC) layer for the appropriate number of labels, and only fine-tune (re-train) the last FC layer plus 0, 1, 3, or 9 convolutional blocks. Freezing layers in the neural networks entails permitting forward propagation, but disabling the prop-pronn of gradients used during SGD training. We have four different fine-tuning configurations, one for each number of fine-tuned convolutional blocks. Note that the ResNet model that we consider has residual blocks that are composed of three convolutional layers, i.e., we fine-tune 27 layers plus the fully connected layer when the number of fine-tuned blocks is equal to 9. (See Section A.4 for visualization of our fine-tuning process).
Random subsets. One of the most interesting parts of our experimental setup is that we also explore the test accuracy of our fine-tuned models using randomly chosen subsets of 100, 200, 400, and 25,600 images from the target dataset. These subsets are constructed using random sampling without replacement, with minor constraint: all labels must have at least one training image. For each run of model training, we fix the training data to be randomized subset of the entire training data. As the number of images in a random subset decreases, the variance in the validation accuracy of the transferred models increases. Thus, we repeat the fine-tuning procedure using 20 seeds for every subset with at most 1,600 images, and using 10 seeds for all larger subsets. We reduce the number of seeds for larger subsets because the inherently lower variance in the validation accuracy doesn't t justify paying the computational cost associated to fine-tuning more seeds.
Target datasets. We transfer our models to a broad set of target datasets, including (1) CIFAR- 100, (2) CIFAR-10, 10, (3) SVHN, (4) Fashion MNIST (Xiao al., 2017). (5) KMNIST and (6) MNIST. Since all of these datasets have images at lower resolution than ImageNet, we up-scale our images with bi-linear interpolation. In addition, we use common data transform techniques such as random cropping and rotation that are well-known to produce high-quality results with certain datasets.
A.4 FINE- TUNING DETAILS
Figure illlsstrates all four fine-tuning configurations in our experiments. Notice how in Subfigure d) we unfreeze more than half of the ResNet-50 architecture, thereby testing what occurs as we fine-tune lot of blocks.
All source models are fine-tuned to all datasets using stochastic gradient descent with momentum using the hyperparameters described in Table
Table 3: Hyper- parameerr summary for all fine-tuned source models
Learning
Batch
Weight
LR
LR decay
Fine-tuned adversarially?
rate
size
Momentum 0.9
decay
decay
schedule
0.1
128
5 x 10
10x
1/3, 2/3 epochs
No
The learning rate decays to a tenth of it's current value every 33 or 50 epochs, which corresponds to 1/3 of the total fine-tuning epochs, as shown in Table 4 Also, the test accuracy frequency refers to how often is the test accuracy computed, in epochs. So, for example, if the test accuracy frequency 20, then we check the test accuracy after epoch 1, 21, 41, ..., 81, and 100.
With regards to the random seeds, we have the following formula to define the set of seeds used, Sk, as function of the total number of random seeds used, k:
'The models that we are provided as part of by the following repository: Madryliab/rrbustnesss
htttp:://gtthbs
13

---

Published as a conference paper at ICLR 2021
Table 4: Batch summary for every target dataset and source model
Number of Fine-tuning epochs
Number of random seeds
Test accuracy frequency (epochs)
LR decay
images
schedule
100
100
20
20
33/66
200
100
20
20
33/66
400
100
20
20
33/66
800
100
20
20
33/66
1,600
100
20
20
33/66
3,200
150
10
10
50/100
6,400
150
10
10
50/100
12,800
150
5
10
50/100
25,600
150
10
50/100
All
150
50/100 10
S₁ {20000000 + (1000000)}z (0, 1, , 1)}.
(6)
Thus, when use 20 seeds, as it is the case for the subset of 100 images, we use seeds 20000000, 20100000, ..., 21900000. Large numbers were used to avoid numerical instability issues that arise with small numbers where their binary representation has too many zeroes.
See Table [5] for additional detail with regards to the source models. Notice that although adversarially-trainee models do worse the source dataset, they outperform naturally-trained mod- els on the target datasets, as shown in Table
Table 5: Summary of source models trained ImageNet, which we consider for transfer learning.
Pre-training Procedure
Constraint
ImageNet Test Accuracy
Natural
76.13%
Adversarial
|8||2 3 | < 4/255
57.90%
Adversarial
62.42%
Adversarial
| S 8/255
47.91%
See Table for high-level overview of all datasets used. This should serve as reminder that our source dataset is ImageNet, with an extensive 1.2 million training images and 1,000 labels, it serves as a great starting point our experiments. All other target datasets have considerably lower number of training and test images.
Table 6: Summary of the source and target datasets.
Number of training images
Number of test images
Number of labels
Color or
Source or
Dataset
grayscale
target
ImageNet
1.2 million
150,000
1,000
color
source
CIFAR-100
50,000
10,000
100
color
target
CIFAR-10
50,000
10,000
10
color
target
SVHN
73,257
26,032
10
color
target
FMNIST
60,000
10,000
10
grey-scale
target
KMNIST
60,000
10,000
10
grey-scale
target
MNIST
60,000
10,000
10
grey-scale
target
14

---

Published as a conference paper at ICLR 2021
(a)
(b)
(d)
Toreow 24.22
757 orn
Ter Ser.ee d6
ToT
Gooee 512
Y4
Figure 9: Fine-tuning setup: (a) Zero blocks. (b) One block. (c) Three blocks (d) Nine blocks. Each block has Three convolutional layers.
A.5 ADDITIONAL RESULTS
Table 7 reports the test accuracy of all of our source models after fine-tuning three blocks using different numbers of training images on each of the six target datasets. The rightmost column shows the trantsfnffrrrdd model trained only on the target dataset and trained on the entire network. The average test accuracy is reported for all cases where the model is fine-tuned with less than the entire training set. The bolded numbers represent the highest test accuracy among source models. From this table, we that the robust models consistently outperform the natural models.
15

---

Published as a conference paper at ICLR 2021
(1a)
(1b)
70
CFAR-100
FMNIST
75
60
30
QFAR-10
KMNIST
70
SVHN
MNIST
Accuracy
robust
F00US1
50-
10051
metural
natural
40
natura
20
60
85
80
95
70
60
robust
ro0US1 natural
rooust
5
notural
50
naturo
75
85
100
100
Number training epochs
Number of training epochs
(2a)
(2h)
95.0
70
92.5
15.0
CIFAR-10
CIFAR-10U
90.0
Test Accuracy Delta
12.5
Test Accuracy
60
87.5
80
10.0
FMINIST
70
7.5
92
95
5.0
90
98
2.5
96
0.0
80
150
Number of training epochs
Number of training epochs
Figure 10: Robust models transfer faster. (a) Shows the test accuracy during the fine-tuning process on the six target datasets (color-coded as in (b)). (b) Shows the test accuracy delta, defined as the robust model test accuracy minus the natural model test accuracy. The solid line the mean and its shade is the 95% confidence interval. Both the robust and natural models are ResNet50s' trained on ImageNet. The robust model uses ||ó||2 S 3 constraint. Both models fine-tune three convolutional blocks using a random subset of 800 images (~ 2%) in (1) and 12,800 images (~ 26%) in (2) of the target dataset.
A.6 ADDITIONAL DETAIL. ON LEARNING FASTER
The following subsection contains the additional charts that were omitted in Figure 3 in Section 4 Figure [ shows the same behavior observed in all three figures: It's sub-optimal to fine-tune either nine convolutional blocks, opposed to one three. Consistent with methodology Figure 4 we adversarially-train models on ImageNet and then fine-tune various numbers of convolutional blocks using random sample of images in the target dataset.
A.7 ADDITIONAL DETAIL ON THE EFFECT OF THE NUMBER OF FINE- FINE-TUNED BLOCKS
The following subsection contains the additional charts that were omitted in Figure in Section Figure [ shows the same behavior observed in all three figures: It's sub-optimal to fine-tune either nine convolutional blocks, opposed to for Figure 4 we adversarially-train models on ImageNet and then fine-tune various numbers of convolutional blocks using random sample of images in the target dataset.
three. Consistent with methodology
A.8 COMPUTATIONAL COST
In general, a PGD(k) adversarial training process described by Madry, is k orders of magnitude more expensive than natural training. This can be seen directly from the fact that there are & more iterations in the inner maximization loop of the risk minimization procedure. However, since only the source model must trained adversarially, and these source models can be downloaded from publicly available repositories, the marginal computational cost of fine-tuning target dataset is
16

---

Published as a conference paper at ICLR 2021
(1.9)
(1.b)
100
CIFAR-ICO
75
75
Accuracy
1
50
Ww
Test Accuracy Delta
20
SVHN
0
25
15
KMNIST
80
80
1+80
50
60
1
40
25
100
400
1600
6400
25500
100
400
1600
6400
25600
100
400
1600
6400
25800
200
200
400
800
1600
3200
6400
12800
25600
50on0
Number training images
Number of training images
(2.a)
(2.b)
100
20
75
75
Test Accuracy Delta %
15
1+80
60
Accuracy
50
5
20
100
0.00
0
80
75
75
++00
50
++00
50
1480
SETI
-10
CIFAR-10
CIFAR-100
KMMST
FMNIST
25
SVHN
MNIST
25
-15
100
400
1800
6400
28600
100
460
1600
6400
25600
100
400
1660
6460
25500
100
200
400
8up
1600
3200
6400
12860
25
50000
Number of training images
Number of training images
(3.a)
(3.b)
50
80
75
Test Accuracy Delta %
15
CIFAR-100
CIFAR-10
KMNNST
FMNIST
Test Accuracy
++80
+
50-
MNIST
25
20
100
100
5
80
75
o
50
50
25
100
400
1600
6400
28600
100
400
1600
6400
25000
100
400
1600
6400
25600
100
200
400
800
1eno
3200
6400
128n0
25e00
saano
Number of training images
Number of training images
Figure 11: It's sub-optimal fine-tune either zero (only FC layer) or nine convolutional blocks, as opposed to one three. (a) Test accuracy each of the six target datasets (color-coded as in (b)) for various subset sizes and fine-tuning various number of blocks: zero, one, three, or nine, (b) Test accuracy delta is defined as the test accuracy with three fine-tuneds blocks minus the test accuracy with one fine-tuned block. Adversarial constraints used to train on the source dataset (ImageNet): (1) ||ōl|2 3, (2) || 2 (3) ||]| I 255 The solid line is the mean and the shade is the 95% confidence interval.
perhaps more important. Fortunately, the computational cost fine-tuning both robust or natural models is the same.
17

---

Published as a conference paper at ICLR 2021
A.9 ADDITIONAL DETAILS ON INFLUENCE FUNCTIONS
Suppose that & R x Rd + R is a smooth loss function and x² ,...,,"" x= E I are our given data.
The empirical risk minimization (ERM) problem takes the form
min f(0) SER
0)) j=1
Say 0 is the argmin solution of the above optimization problem. Let's now consider upweighing data point Train with e E I. This modifies the learning problem as:
Σ e(x², w) + el( 0).
min BER
Let Or be the solution of the upweighted problem above. The influence of training data point Train on test data point Tistt approximates the change in the function value [Frest, 0*) + {( 0*) with respect to an infinitesimally small c, i.e., when c -> when Tirain is upweighed by c. This can be calculated in closed form Koh & Liang (2017) as:
-91 92.
where 91 = ( 8T)T_ 92 (areett 8*) and H is the Hessian of the loss function V2 In particular, we first compute the Hessian of the source model fine-tuned with 3,200 CIFAR-10 images as the sum of the Hessians of the loss of batches of five images. Note that we only use the 3,200 images that were used in the fine-tuning process, since it accurately reflects the Hessian of the model. Then get H- using the Pooro-Pen pseudo-inverse, using its singular-valus decomposition and including all singular values larger than le-20.
Koh et. al. Koh & Liang (017)) discuss optimization speedup techniques to determine the most influential Zuain for a given Test at scale. However, finding top- influential images is a combinatorial problem for k > 1. So, typically greedy selection of the next top influential image is made iteratively k times. Further, selecting multiple images also requires consideration of interaction and group effects. As such, the top-5 influential images likely to be less representative of actual influence being asserted than one would expect.
Fisher kernels and influence functions Khanna et. al. Khanna et al. (2019) recently discovered an interesting relationship between Fisher Kernels and Influence functions: if the loss function {(-) can be written negative log-likelihood, then the optimum w*, the Fisher dot product between two points is exactly the same as the influence of those points on each other (note that the influence is symmetric function). In other words, finding the most influential data point to a given data point is equivalent to finding the nearest neighbor of the point in the space induced by the Fisher kernel. As observed in Section [6] for robust training, most influential points for a data point tend to be largely the ones belonging to the same label. This implies that the in the Fisher space, the points with the same label tend to be grouped together.
A. ADDITIONAL DETAILS FOR ADVERSARIAL ATTACKS COMPARISON SECTION
Both the PGD(1) and Gaussian models are ResNet-50's trained ImageNet-1 1K using stochastic gradient descent with momentum and the following hyperparameters: 0.1 learning rate, 128 batch size, 0.9 momentum, 10x learning rate decay, and an equally-spaced (i.e. linear) learning rate decay schedule. The test accuracy of each of these models is 60.29% and 74.02% for the PGD(1) and Gaussian model, respectively. Both models use the ||8]|2 2 I 3 adversarial constraint.
The PGD(1) model uses one attacker step, with step size of 6 and the perturbation is initialized at zero. The Gaussian model adds a perturbation for each pixel drawn from standard Normal distribution ((u 0, o2 1).
A.11 CODEBASE OVERVIEW
The starting point requires downloading the source ImageNet models, and installing the appropriate libraries. Next, the user decide how fine-tune the source models: individually or in batches.
18

---

Published as a conference paper at ICLR 2021
The train. py file will allow individual training, while the tcols/batcio py file allows train- ing in batches.
The train. py file contains 9 parameters that are explained by rumning the following command: python train. py -p. Also, the helpers. py and delete big files. py files un- der the tools folder contain the logic that supports the train. file. This includes the random subset generator, the fine-tuning procedure, and the data transforms.
Separately, note that when rumning the batch. py file, the fine-tuned models won't be saved into the result t / llogs directory. This is due to the fact that models can occupy a significant amount do not plan use these fine-tuned models in the future. However, if the user wants to save the fine-tuned models, then he or she can do so by commenting our line 60 in the bat catch.pp py file: delleteB.ggleefooreeeeeeeeeeeeee
of memory and
()
Lastly all results are stored into the results/ folder by default and can be compiled easily into a csv file using the log.extractor: py script.
A.12 DETAILED FUTURE WORKS
Even though we support our main thesis with extensive empirical evidence and analyze this phe- nomenon through the lens of texture bias and influence functions, why, when, and how robust mod- els transfer better deserves further investigation. In this section we'd like to provide some ideas that we hope will spark research interest.
Different adversarial training constraint type. Prior work only considers &2 and foo adversarial constraint types. However, different adversarial constraints could allow models to retain more trans- ferable features from the source dataset. Two possibilities
(i) constraining with influence functions, and (ii) constraining on the Fourier space instead of the pixel space. For (), as shown by Koh & Liang (017)) for the 7h image I; we can compute the perturbation d; each step of PGD by starting with o; = 0, and then 5 P(o, + assinnppert.ee (x; + 64, x;)).. For (ii), we could instead calculate the gradient w.r.t. each one of the frequencies and constrain the model to only use a subset of all of its frequencies to represent the input image.
the Fischer Kernel
Different source datasets. ImageNet might not be the hest source dataset for two reasons. First, we as shown by (Tsipras al. (2020) there are many labels that overlap with each other, such as rifle and assault rifle. Second, there are many training images containing objects from more than one label, referred to in Tsipras et al. (2020) as multi-object images. Thus, we think it would be worthwhile to use Tencent's Large-Scale Multi-Label Image Database from Wu et al. (2019) as source dataset instead of ImageNet.
Decision- boundayy bias. In line with Section 5 it might be worth looking at the transfcrability of robust models a function of how closely related the labels in the target dataset. Our hy- pothesis is that if the labels in the target dataset are closely related, then the robust model might transfer slightly worse than if the labels were further apart from each other. Although measuring the closeness of labels within dataset is challenging, this could be an interesting extension to Santurkar al. (2020).
New use-cases. As shown in Section 5 robust models are biased towards low resolutions and low frequencies. Thus, it's possible that robust models have lower facial recognition bias than naturally trained models.
19

---

Published as a conference paper at ICLR 2021
Table 7: Summary of the test accuracy on larget datasets after fine-tuning three blocks (nine convo- lutional layers) except for the rightmost column, which shows the non-transferred model trained on the entire network. Reported average test accuracy for all cases where the model is fine-tuned with less than the entire training set.
Target
Training images
Pre-training constraint on source dataset (ImageNet)
Trained From
dataset
Natural
|| &
|| $
233
|| $ 253
Random Tnit
CIFAR-100
100
7.97
8.54
8.24
1.34
CIFAR-10
51.89
49.86
12.29
SVHN
100
23.5
37,9
34.75
35.75
19.14
FMNIST
100
66.11
71.8
71.24
72.69
19.91
KMNIST
100
34.29
51.76
50.12
51.05
12.42
MNIST
100
68.69
83.83
83.03
82.81
12.79
CIFAR-100
200
13.05
13.26
13.71
12.79
1.35
CIFAR-10
200
57 57.44
64.05
62.83
60.11
11.01
SVHN
200
29.37
50.89
46.82
47.77
19.16
FMNIST
200
73.53
78.51
78.33
19.88
KMNIST
200
44.77
62.97
60.36
62.84
12.63
MNIST
200
81.58
90.72
90.44
90.49
15.66
CIFAR-100
400
18.57
21.74
21.32
19.79
1.36
CIFAR-I0
400
66.84
71.65
71.93
68.81
12.01
SVHN
400
37.3
61.97
58.99
61.02
19.36
FMNIST
400
78.8
82.09
81.4
81.58
35.23
KMNIST
400
52.45
71.65
70.18
71.89
13.13
MNIST
400
88.09
94.28
93.9
93.98
22.57
CIFAR-100
800
27.57
33.68
32.55
30.17
2.13
CIFAR-10
800
78.68
78.99
75.95
20.86
SVHN
800
49.17
71.19
68.84
70.19
19.51
FMNIST
800
82.73
85.21
84.6
84,59
61.12
KMNIST
800
63
79.67
77.73
80.68
16.86
MNIST
800
91.81
96.05
95.58
95.92
38.6
CIFAR-100
1,600
37.88
45.64
44.49
40.82
5.13
CIFAR-10
1,600
79.74
83.76
83.47
30.98
SVHN
1,600
77.84
75.24
76.77
19.6
FMNIST
1,600
85,39
87.58
87.09
86.91
78.14
KMNIST
I, 1,600
74.06
84.95
83.66
85,52
30.13
MNIST
1,600
94.47
97.13
97
97.16
85.55
CIFAR-100
3,200
51.46
56.91
55.79
51.34
20.94
CIFAR-10
3,200
83.77
87.56
87.34
85.07
55.99
SVHN
3,200
74.31
85.55
81.57
84.4
22.94
FMNIST
3,200
87.97
89.43
89.11
88.89
83.47
KMNIST
3,200
81.42
89.68
88.26
89.14
81.51
MNIST
3,200
96.4
98.13
97.93
98.1
96.89
CIFAR-100
6,400
62.26
66.1
65.31
60.75
36.74
CIFAR-10
6,400
87.28
90.91
90.23
88.6
71.37
SVHN
6,400
82.63
90.46
88.17
89.09
83.87
FMNIST
6,400
89.63
90.73
90.44
90.11
87.41
KMNIST
6,400
86.7
93.61
91.7
92.86
90.88
MNIST
6,400
97.34
98.73
98.5
CIFAR-100
12,800
69 92
71.61
71.42
67.16
52,32
CIFAR-10
12,800
91.95
93.2
93.11
81.19
SVHN
12,800
88.7
92.86
91.68
91.81
92.44
FMNIST
12,800
91.56
92.09
91.97
91.69
90.32
KMNIST
12,800
94.32
96.21
95.73
95.9
95.7
MNIST
12,800
98.52
99.11
98.98
99.03
99.27
CIFAR-100
25,600
75.08
75.68
75.4
71.52
67.75
CIFAR-10
25.600
93.93
94,92
94.63
93.28
89.92
SVHN
25,600
92.24
94.36
93.76
93.91
94,93
FMNIST
25,600
93.24
93.14
92.79
92.25
KMNIST
25,600
97.01
97.52
97.34
97.44
97.96
MNIST
25,600
99.06
99.27
99.21
99.24
99.53
CIFAR-100
50,000
78.49
79.24
79,51
76.06
79.42
CIFAR-10
50,000
95.36
95.86
95.6
94,58
94.08
FMNIST
60,000
94.51
94.72
94.41
93.98
93.97
KMNIST
60,000
98.07
98.35
98. 19
98.38
99.01
MNIST
60.000
99.19
99.42
99.34
99.39
99.68
SVHN
73.257
95.33
96.02
95.42
95.63
96.53
20


Reviews:
[{"review_id": "mP7RDMIgSq4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of paper:** This paper investigates how \"robust\" (adversarially trained) models can improve the transfer of representations, finding that they transfer better. Additionally, they investigate some reasons this could be the case, examining the biases robust models appear to induce.\n\n**Pros/strong points:**\n - interesting, well-explained experiments with mostly clear nice figures\n - nice extra investigations giving insight into the bias(s) conferred by adversarial training\n\n**Cons/weak points:**\n - overstatement as though the results apply to any type of data/model, but only image data and convolutional nets are tried\n - potential issue with influence function experiments\n - no analysis of computational cost of adversarial training or other information on tradeoffs\n - background lacking/potential issues with related work\n\n**Summary of review + recommendation:** Good paper with nice, thoughtful experiments, mostly well written, although I think the biggest issue there is over-claiming results applying to all data when only image data/convnets are studied. I'm also a bit worried about the thoroughness of the background research, and would like to see an analysis of the computational cost of adversarial training. If these and my question about influence function experiments are addressed, I would be happy to raise my score.\n\nDETAILED REVIEW:\n\n**Quality:**\n - Generally well written and organized, although the link between successive sections could be made a bit more strongly / flow could be better\n - Overstatement of scope (transfer learning in general, when only image data is studied) combined with misstatements in the first parts (about the origins of transfer learning) makes me skeptical of the depth of background research done and makes me suspect there may be some very relevant things which have not been surveyed/cited.  (see Specific questions/recommendations for details on both points, below)   \n - Comparison of computational cost / other tradeoffs in adversarial vs. 'natural' training not discussed\n - Potential weakness/misleading conclusions in influence function experiments if I've understood correctly (see specifics below)\n\n**Clarity:**\n - The abstract and introduction would benefir from more technical, specific language to avoid ambiguity and establish flow of the sections\n - Clarity within sections is pretty good., some specific suggestions below.\n - Figures are mostly nice and clear, not so much Table 1 and 7a though.\n\n**Originality:**\n - This is the largest potential problem that I am most unsure about. I'm very familiar with work on statistical learning theory and generalization overall, but I'm not an expert in transfer learning or adversarial method and I'm not sure how well these works are reviewed, so I'm not sure how novel this work is. The experiments are well done and well explained though, and I think this is a good contribution even if it is less original than it is made to seem due to the lack of context.\n\n**Significance:**\n - Nice insights and interesting experiments for those wanting to understand the impact of robust training on transfer\n - Limited practical insights without an analysis/discussion of tradeoffs e.g. overall computational time and stability.\n - Directions proposed for future work are concrete and interesting\n\n\n**Specific questions/recommendations:**\n - the first sentence of abstract and the title talk about DL generically, but the second-sentence is about images specifically. If you add non-image data, I'd suggest rephrasing to make the 2nd sentence generic, and maybe mention this technique has been particularly successful for images. However since all experiments are with images, I'd suggest making the title and abstract specific to that domain. e.g. \"Evidence from image data that adversarially-trained deep nets transfer better\" or \"Adversarially-trained deep nets transfer to new images better\"  \n - Overstatement of results: If you want the strong/general claims about transfer learning, I would strongly recommend doing experiments with at least MLPs in addition to convnets, and at least one other type of data in addition to  images. Otherwise the statements in title/abstract/intro should be circumscribed to more accurately reflect the nature of the experiments.  \n - mention what is adversarially vs. naturally trained. although I strongly suggest using a different term than 'naturally', including in diagrams/elsewhere as it is confusing. It makes it seem like you are comparing adversarial training to natural gradient methods.\n - first sentence putting data hogs in quotes misleadingly suggests that it's a commonly used phrase. Suggest removing this i.e. \"they are known to require large...\" (and suggest adding a reference which quantifies this rather than referring to hearsay).\n - \"similarly, \"stunning\" is an opinion, suggest \"remarkable\" or something like \"excellent\" which can be derived from empirical results - Comparison of computational cost / other tradeoffs :even just a reference where this is done with a sentence summarizing those results / other tradeoffs, e.g. \"it's usually at least 2x as slow and more likely to be unstable\" or something like that might be enough, but I would prefer to see full training curves and computational cost numbers in appendix, with a line or two summarizing these in the main text.\n - Stating that Caruana (1995) proposed transfer learning seems incorrect to me. There was a NeurIPS workshop that same year on the subject, suggesting it was already an established term at that time. I don't know the full history, but from a quick google it seems to have been very common in psychology / education literature in the 70s and 80s, here's a book that talked about it in the context of ML in the 80s :https://pdfs.semanticscholar.org/b547/c5837bff9347dc76330a72fd7cbc517ee08c.pdf and here's Rich Sutton talking about it in 92: https://link.springer.com/content/pdf/10.1007/978-1-4615-3618-5.pdf\n - Related work:    - covariate shift could also mention risk extrapolation https://arxiv.org/pdf/2003.00688.pdf (how does the extrapolation done there differ from the adversarial training?)    - it seems like a lot of works on adversarial and contrastive training and the relationship to generalization are missing; I'm not an expert in this but starting way back hard negative mining and other contrastive methods (e.g. word2vec) have been used and their properties discussed \n - The first subheading in section 4 is the conclusion drawn from that paragraph: \"Adversarially-trained models transfer better and faster.\", but subsequent headings do not have the same 'syntax' (they are more like titles than conclusions to be drawn). I like the conclusion-as-title format; I find it very engaging and helpful for skimming especially since there are many experiments. But most of all I would strongly suggest that all titles have the same 'syntax' i.e. if you can't think of conclusions-as-titles for the other bold p headers (although I think you can and should!), I would recommend rephrasing this one to be like the others (e.g. Comparison of adversarial and non-adversarial transfer)\n - formatting of table 1, with captions both below and above the tables, is hard to read. Put it all above or all below.\n - Could have been a nice opportunity to investigate whether robust models are more or less susceptible to the types of bias people worry about in real-world image datasets (e.g. face recognition); maybe worth mentioning this in future work \n - Unless I'm mistaken, the experiments with influence functions do not distinguish the effect of performance from that of training (bias toward the \"human prior\"). To do so, the robust and natural methods would have to have the same accuracy (i.e. this might involve very early stopping of the robust method to match the natural model performance). Without this, the qualitative and quantitative results could just be due to the higher accuracy of the robust method, not the particular form of prior it induces.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YBgAeVWzaRF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Contributions ###\n* The paper proposes that models that were adversarially trained transfer better to other datasets in that they increase _clean_ performance on this target dataset if there are only few labeled datapoints for the target task or only few training epochs are conducted\n* The authors test their hypothesis for ResNets pretrained on Imagenet in different threat models and transfer these models to 6 different target datasets. Generally, results provide sufficient evidence for the paper's main hypothesis (robust models transfer better)\n* Additional experiments provide evidence that better transferability of robust models is partly due to relying more on shape rather than texture cues. Moreover, an additional analysis using influence functions leads to the hypothesis that  robust neural networks might have learned to classify using example-based concept learning like in human beings.\n\n### Significance ###\nTransfer learning is a topic of high relevancy for practitioners since it can reduce both data label effort and training time. Improving upon the baseline of transferring models pretrained on Imagenet with a non-adversarial loss is thus a potential significant result. \nHowever, the paper's review of the transfer learning literature is superficial and misses some relevant related work such as \"Rethinking ImageNet Pre-training\" by He et al., ICCV 2019.  Additionally, Geirhos et al. (ICLR 2019) also showed that models pretrained on stylized ImageNet (and thus having a stronger shape bias) transfer better to object detection tasks. This should be mentioned in Section 5.  And more generally: if stronger transferability is mainly due to increased shape bias, wouldn't it make sense to pretrain explicitly for strong shape bias rather than achieving this indirectly via adversarial training as proposed in this paper?\nA more thorough review of the transfer learning literature and relating the obtained results to this would generally strengthen the paper.\n\n### Originality ###\nThe work is a purely empirical work studying the stated hypotheses, no novel methods are proposed. Originality can thus only come from the hypotheses.\nThe main hypothesis (robust models transfer better) was also proposed by Salman et al. (2020). However, this work should be seen as concurrent since it was released on arXiv only four months ago. \nThe main prior work is Shafahi et al. (ICLR 2020), which also studies transferring adversarially pretrained models to other tasks. However, their focus is on the robustness gains of transferred models rather than on the effect on clean performance.\nIn summary, I think the main hypothesis studied in the paper is original. However, it is also clearly only a relatively small incremental step beyond what Shafahi et al. 2020 did.\n\n### Clarity ###\nExperimental setup, training pipeline, and analysis are outlined clearly. Releasing code for finetuning and replicating the experiments would further strengthen reproducibility\n\n### Quality ###\nGenerally, the experiments are well conducted, covering a broad range of threat models, target datasets, training image and epochs regimes, and finetuning strategies. Additionally, Section 5 and 6 shed additional light onto why robust models might transfer better, and by this further strengthening the main message of the paper.\nOne shortcoming is that all target tasks are image classification tasks. Whether robust models also transfer better to task such object detection or semantic segmentation remains unclear. \n\n### Recommendation ###\nIn summary, I think the paper is a nice experimental study of a clearly stated hypothesis with potential practical impact. I thus lean towards acceptance, even though novelty is clearly borderline.\n\n### Final Recommendation after Author Response###\nThe authors have addressed several of my main concerns. It would have been helpful to study transferability to tasks beyond image recognition, but overall, I think the paper has been considerably improved. I increase my score to 7.\nTwo remarks regarding new content:\n * I find it misleading to denote PGD as a targeted adversary and additive Gaussian noise as a random adversary Section 7. \"Targeted\" usually refers to an adversary that aims at achieving a specific misclassification (target class). Gaussian noise is not really an adversary, rather a distortion/image corruption. I would recommend clarifying the naming to avoid confusion of readers.\n * Is there any particular reason to use PGD(3) in Table 1b for evaluation? Would the effect hold also against stronger attacks (more iterations etc.)?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AfnWPt9tDU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to investigate and understand if and how adversarial training helps the models trained on the source domain transfer easier and faster to target domains. With extensive different configurations (such as fine-tuning strategies) in experiments, the authors show that robust models transfer better than natural models with less training data from the target domain. Also they demonstrate the intuition behind through experiments, such as capturing shapes than textures or using influence functions. \n\nStrengths\n- The idea is interesting and have a potential for impacts in the community.\n- Extensive experiments and investigations how and when the robust models works better than natural models is good to demonstrate the main ideas of this paper.\n- Paper is easy to understand.\n\nWeaknesses\n- Even though it was shown by the experiments, it might need to have more theoretical understanding why the robust models transfer better or have better representations than natural models. \n- Even though it seems to provide some explanations, it lacks more thorough investigation why the specific configuration choices yield better performances than others.\n- The presented dataset choices seem limited, which could limit its potential impacts and applications in real-world problems (see the comments below).\n\nDetailed comments:\n- If the shape is indeed more important than texture for human-like performance, is it possible make the model even works on par with the natural models? \n- Why specific configuration works better than others, such as fine-tuning three conv. blocks and ∥δ∥2 ≤ 3?\n- In CIFAR-100 and (especially) CIFAR-10, fine-tuning one conv. block is better than zero conv. block and why?\n- The target domains except CIFAR-100 and CIFAR-10 are all digit datasets, so its application to real-world problems may be limited. How about using different and non-overlapping classes than those in the source domain in other image datasets as target domains, such as CALTECH-256? It could make the paper stronger.\n- In Table 5, the accuracy differences seem larger than ~5% as written in the text.\n\nTypo:\nPage 3: nx‘on-negative -> non-negative", "extra_scores": {}, "extra_text": {}}, {"review_id": "5uGD54LYltl", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1, Summary of contribution:\nThis paper claims that the pre-trained model trained adversarially can achieve better performance on transfer learning, and conducted extensive experiments on the efficacy of the adversely trained pre-trained models.\nAlso, the paper conducts an empirical analysis of the trained models and shows that the adversarially pre-trained models uses the shape of the images rather than the texture to classify the images. \nUsing the influence function (Koh 2017), the paper reveals that each influential image on the adversarially trained model is much more perceptively similar to its test example.\n \n\n2, Strengths and Weaknesses:\nThe paper is well-written and organized, and the experiments look fair and well support the claim.   The analysis is interesting and insightful. \nMeanwhile, the transfer is done to the domain of lower complexity, and some important comparative ideas are not extensively investigated. \n\n\n3, Recommendation:\nWhile the paper’s empirical results are solid, there seems to be a substantial room left for comparative studies.  More ablation studies shall be done for other regularization methods.\nI believe that the paper is marginally above the acceptance threshold. \n\n4, Reasons for Recommendation:\nThe reader will benefit more from the paper if the authors can justify their use of adversarial training as the regularization in the pretraining process.  I believe that this research warrants some comparative study for dropout, weight decay, as well as random perturbations.  I think the paper can be more insightful if it shows whether the other classical regularization performs better or worse on transfer learning than the proposed approach. \n\n5,  Additional feedback:\nIn addition to the suggestions made in 4, I also believe that comparison shall be made against the model trained without pretraining. \n\n\\\n---Post rebuttal---\n\nThank you for the response, and thank you for checking the performance comparison against the white-noise perturbation. It would be interesting to see a future work involving means other than Adversarial training (e.g. including other simple mechanisms like weight decay and dropout) to help reduce the overfitting effects in the pretraining phase. I would like to keep my score as is. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The premise of the work is simple enough: investigate if networks that are trained with an adversarial objective end up being more suitable for transfer learning tasks, especially in the context of limited labeled data for the new domain. The work uncovers the fact that shape-biased representations are learned this way and this helps for the tasks they considered.

There was rather robust back and forth between the authors and the reviewers. The consensus is that this work has merit, has good quality experiments and investigates something with high potential impact (given the importance of transfer learning in general). I hope that most of the back and forth findings are incorporated in the final version of this work (especially the discussion and comparison with Shafahi et al., as well as all the nuances of the shape bias).
Decision: Accept (Poster)