Adversarially robust transfer learning
Authors: Ali Shafahi
Parsa Saadatpanah
Chen Zhu
Amin Ghiasi
Christoph Studer
David Jacobs
Tom Goldstein
Venue:  (2020)

Abstract: Transfer learning, in which a network is trained on one task and re-purposed on another, is often used to produce neural network classifiers when data is scarce or full-scale training is too costly.  When the goal is to produce a model that is not only accurate but also adversarially robust, data scarcity and computational limitations become even more cumbersome.
We consider robust transfer learning, in which we transfer not only performance but also robustness from a source model to a target domain.  We start by observing that robust networks contain robust feature extractors. By training classifiers on top of these feature extractors, we produce new models that inherit the robustness of their parent networks. We then consider the case of "fine tuning" a network by re-training end-to-end in the target domain. When using lifelong learning strategies, this process preserves the robustness of the source network while achieving high accuracy. By using such strategies, it is possible to produce accurate and robust models with little data, and without the cost of adversarial training. Additionally, we can improve the generalization of adversarially trained models, while maintaining their robustness.

Published as a conference paper at ICLR 2020
ADVERSARIALLY ROBUST TRANSFER LEARNING
Ali $ Parsa Saadatpanah" Chen Zhu Amin Ghiasi! Cristoph Studer; (asha fahi, parsa, chenzhu, ami amin) @cs. umd edu: studer&corme edu
David Jacobs} Tom Goldstein
{djacobs, (ong) umd. edu
ABSTRACT
Transfer learning, in which a network trained on one task and re re-purposed on another, is often used to produce neural network classifiers when data is scarce or full-scale training is too costly. When the goal is to produce a model that is not only accurate but also adversarially robust, data searcity and computational limitations become even more cumbersome. We consider robust transfer learn- ing, in which we transfer not only performance but also robustness from a source model to a target domain. We start by observing that robust networks contain ro- bust feature extractors. By training classifiers on top of these feature extractors, we produce new models that inherit the robustness of their parent networks. We then consider the case of "fine tuning" network by re-training end-to-end in the target domain. When using lifelong learning strategies, this process preserves the robustness of the source network while achieving high accuracy. By using such strategies, it is possible to produce accurate and robust models with little data, and without the cost of adversarial training. Additionally, we can improve the generalization of adversarially trained models, while maintaining their robustness.
INTRODUCTION
Deep neural networks achieve human-like accuracy range of tasks when sufficient training data and computing power is available. However, when large datasets are unavailable for training, or pracitioners require a low-cost training strategy, transfer learning methods are often used. This process starts with source network (pre-trained on a task for which large datasets are available), which is then re-purposed to act on the target problem, usually with minimal re-training on a small dataset YYosinski et al.] 2014 Pan & Yang 2009).
While transfer learning greatly accelerates the training pipeline and reduces data requirements target domain, it does not address the important issue of model robustness. It is well-known that naturally trained models often completely fail under adversarial inputs (Biggio et al. 2013 Szegedy al. 2013). As result, researchers and practitioners often resort adversarial training, which adversarial examples are crafted on-the-fly fly during network training and injected into the training set. This process greatly exacerbates the problems that transfer learning seeks to avoid. The high cost of creating adversarial examples increases training time (often by an order magnitude or more). Furthermore, robustness is known suffer when training on small dataset (Schmidt al., 2018). To make things worse, high-capacity models are often needed to achieve good robustness Madry al. 2017 Kurakin et al., 2016 Shafahi et al. 2019b) but these models may over-fit badly on small datasets.
CONTRIBUTIONS
The purpose of this paper is to study the adversarial robustness of models produced by transfer learning. We begin by observing that robust networks contain robust feature extractors, which are resistant to adversarial perturbations in different domains. Such robust features can be used
"equal contribution
University of Maryland
Comell University

---

Published as a conference paper at ICLR 2020
as basis for semi-supervised transfer learning, which only requires re- re-training the last layer of network. To demonstrate the power of robust transfer learning, we transfer a robust ImageNet source model onto the CIFAR domain, achieving both high accuracy and robustness in the new domain without adversarial training. We use visualization methods to explore properties of robust feature extractors. Then, we consider the case of transfer of learning by "fine-tuning." In this case, the source network is re-trained end-to-end using small number of epochs on the target domain. Unfortunately, this end-to-end process does not always retain the robustness of the source domain; the network "forgets" the robust feature representations learned on the source task. To address this problem. we use recently proposed lifelong learning methods that prevent the network from forgetting the robustness it once learned. Using our proposed methods, we construct robust models that generalize well. In particular, we improve the generalization of a robust CIFAR-100 model by roughly 2% while preserving its robustness.
2 BACKGROUND
Adversarial examples fall within the category of evasion attacks-test-time attacks in which a per- turbation is added to natural image before inference. Adversarial attacks most often crafted using a differentiable loss function that measures the performance of a classifier on a chosen im- age. In the case of norm-constrained attacks (which form the basis of most standard benchmark problems), the adversary solves
max 1(x 0.
s.t. 1811 S.
(1)
where 0 are the (already trained and frozen) parameters of classifier c(2z, 0) + 9 that maps an image to class, 2 is the proxy loss used for classification (often cross-entropy), à is the image perturbation, (Ir, y) is the natural image and its true class, and |I-I|p is some fy-norm" The optimization problem
in Eq. [ aims to find a bounded perturbation that maximizes the cross cross-entropy loss given the correct label. There are many variants of this process, including DeepFool (Moosavi-Dezfooll et al. 2016),
L-BFGS (Szegedy et al. 2013). and CW (Carlini & Wagner 2017).
Many researchers have studied methods for building robust network which have been later shown to be ineffecttvve when attacked with stronger adversaries (Athalye et al. (2018). Adversarial training (Szegedy et al. 2013) is one of the defenses that was not broken by Athalye et al. (2018). While ad- versarial training using weak adversary such as the FGSM attack (Goodfellow et al. 2015) can be broken even by single step attacks which add a simple random step prior to the FGSM step (Tramrr let al. 2017), adversarial training using strong attack has successfully improved robustness. Madry et al. 2017) showed that a PGD attack (which is a BIM attack (Kurakin et al.] (2016) with an ini- tial random step and projection) is strong enough attack to achieve promising adversarial training results. We will refer to this training method as PGD adversarial training. PGD adversarial train- ing achieves good robustness on bounded attacks for MNIST (LeCun al. 1998) and acceptable robustness on CIFAR-10 10 (Krizhevsky & Hinton 2009) classifiers.
Tsipras et al. 2018 show that adversarial training with strong PGD adversaries has many benefits in addition to robustness. They also state that while adversarial training may improve generalization in regimes where training data is limited (especially on MNIST), it may be at odds with generalization in regimes where data is available. This trade-off was also recently studied by /Zhag et al. (2019). Su et al. 2018), and Shafahi et al. (019a)).
While, to the best of our knowlegde, the transferability of robustness has not been studied in depth, Hendrycks et al. (2019) studied the case of adversarially training models that were pre-trained on different domains. Our work is fundamentally different in that we seek to transfer robustness with- out resorting to costly and data-hungry adversarial training. We train the target model on natural examples only, which allows to directly study how well robustness transfers. Additionally, this allows us to have better generalization and achieve higher accuracy on validation examples. While as Hendrycks al. (2019) state, fine-tuning on adversarial examples built for the target domain can improve robustness of relatively large datasets such as CIFAR- 10 and CIFAR- 100 compared to adversarial training from scratch on the target domain, we show that in the regimes of limited data (where transfer learning is more common), adversarially robust transfer learning can lead to better results measured in terms of both robustness and clean validation accuracy. By default we will use the foc -oom in this paper.
2

---

Published as a conference paper at ICLR 2020
Table I: Accuracy and robustness of natural and adversarially trained models on CIFAR-10+ and CIFAR-100+. The "+" sign denotes standard data augmentation (c 8).
Dataset
model
validation accuracy
accuracy on PGD-20
accuracy on CW-20
natural
95.01 %
0.00%
0.00%
CIFAR-10+
robust
87.25%
45.84%
46.96%
natural
78. 78.84%
0.00%
0.00%
CIFAR-100+
robust
59.87%
22.76%
23.16%
THE ROBUSTNESS OF DEEP FEATURES
In this section, we explore the robustness of different network layers, and demonstrate that robust networks rely robust deep features. To do so, start from robust classifiers (c(0,)) for the CIFAR-100 100 and CIFAR-10 datasets (Krizhevsky & Hinton 2009). and update 0 by training on natural examples. In each experiment, re-initialize the last & layers/blocks of the network, and re-train just those layers. We start by re-initializing just the last layer, then the last two, and so on until we re-initialize all the layers.
We use the adversarially trained Wide-ResNet 32-10 (Zagoruyko & Komodakis 2016) for CIFAR- 10 from Madry et al. (2017) our robust model for CIFAR-10 10. We also adversarially train our own robust classifier for CIFAR- 100 using the code from Madry et al. (2017). To keep things consistent, we use the same hyper- parametesesserss used by Madry et al. (2017) for adversarially training CIFAR-10 to adversarially train the CIFAR-100 100 model The performance of the CIFAR- 10 and CIFAR-100 models on natural and adversarial examples are summarized in Table [] To measure robustness, we evaluate the models on adversarial examples built using PGD attacks.
We break the WRN 32-10 model into 17 blocks, which are depicted in Fig. ? In each experiment, we first re-initialize the k deepest blocks (blocks through k) and then train the parameters of those blocks on natural images We train for 20,000 iterations using Momentum SGD and a learning rate of 0.001. We then incremenallly unfreeze and train more blocks. For each experiment, evaluate the newly trained model's ' accuracy on validation adversarial examples built with a 20-step PGD foo attack with = 8.
Fig. shows that robustness does not drop only the final layers of the networks are re-trained on natural examples. In fact, there is slight increase in robustness compared to the baseline PGD- 7 adversarially trained models when we just retrain the last batch-normalization block and fully connected block. As we unfreeze and train more blocks, the network's robustness suddenly drops. This leads us to believe that a hardened network's robustness is mainly due to robust deep feature representations and robustness preserved if we re-train on top of deep features.
10mm Hele
33 Linde n0
10
1ir X0
Ja tol Iee
co Sta
10000 Tra
si
3d 1m S20
32 Tamo Se
em ee
30 co 6e
5000mm Ae
mm
x
10 comm.
bat
3d
1:3
Mook 14
Figure 2: Wide Resnet 32-10 and the blocks used for freezing/retraining
Now that we have identified feature extractors as source of robustness, natural investigate whether robustness is preserved when transfer learning using robust feature extractors. We will
We ady. train the WRN 32-10 on CIFAR-100 using 7-step foo PGD attack with step-size=2 and = 8. We train for 80,000 iterations with a batch-size of 128.
In this experiment, we standard data augmentation techniques.
3

---

Published as a conference paper at ICLR 2020
Robust CHFAR-MO retrnining lad bioelo
Rednadtrsss of CIEAR R-102 by setisining blocks
elliiti actunay 100 30
PED 3n cildation WermiNe
1-0 Nambber of deep Marie netraired
Nomber of coup bloeko mitnieee
(a) CIFAR-I0 PGD-20 accuracy
(b) CIFAR-IUS 100 PGD-20 accuracy
Figure Robustness preserved when we retrain only the deepest block(s) of robust CIFAR- 10 and CIFAR- 100 models using natural examples. The vertical axis is the accuracy on PGD-20 generated adversarial examples (i.e. robustness) after re-training deep layers. The robustness of the adversarially trained models if all layers are frozen are shown with dashed lines.
study two different approaches for transferring robustness across datasets: one in which only the last layer is re-trained, and one with end-to-end re-training
TRANSFER LEARNING: RECYCLING FEATURE EXTRACTORS
We study how robustness transfers when the feature extractor layers of the source network are frozen, and we retrain only the last fully connected layer (i.e. the classification layer) for the new task. Formally, the transfer learning objective is:
min 1(=(2r, 0"),, w.
where z is the deep feature extractor function with pre-trained and now "frozen" parameters 0*, and w represents the trainable parameters of the last fully connected layer. To investigate how well robustness transfers, we use two source models: one that is hardened by adversarial training and another that is naturally trained.
We use models trained on CIFAR- 100 as source models and perform transfer learning from CIFAR- 100 CIFAR-10. The results are summarized in Table 2 Compared adversarial/natural training the target model, transferring from a source model seems to result in a drop in natural accuracy (compare first row of Table the first row of Table 2 This difference is wider when the source and target data distributions are dissimilar Yosiiski et al. 2014).
To evaluate our method on datasets with more similar attributes, we randomly partition CIFAR- 100 into two disjoint subsets where each subset contains images corresponding to 50 classes. Table shows the accuracy of transferring from one of the disjoint sets to the other (second row) and to the same set (third row). We can compare results of transfer learning with adversarial training on CIFAR- 100 by averaging the results in the second and third rows of Table get the accuracy across all 100 classes of CIFAR- 100 By doing so, we see that the accuracy of the transferred classifier matches that of the adversarially trained one, even though no adversarial training took place in the target domain. For completeness, we have also included experiments where we use CIFAR-10 as the source and 100 as the target domain.
We make the following observations from the transfer- results in Table 2 transfers: when the source model used for transfer learning is robust, the target model also robust (although less than the source), 2) robustness transfers between models that more similar: If
1) robustness
*The robust CIFAR-100 classifier has 59.87% validation accuracy and 22.76% accuracy on PGD-20 adver- sarial examples. The average validation accuracy of the two Malf-CIFAR--I 100 classifiers on validation examples 64, 98%.+5. 48%
61. 11.22% while the average robustness is 15.16%+15..66 16%
20.51%.
4

---

Published as a conference paper at ICLR 2020
Table 2: Transfer learning by freezing the feature extractor layers (e 8).
Source Dataset
Target Dataset
Source Model
val.
PGD-20
CW-20
natural
83.05%
0.00%
0.00%
CIFAR-I00
CIFAR-10
robust 72.05%
17.70%
17.43%
CIFAR-I00 (50% of classes)
CIFAR-I00
natural
71.44%
0.00%
0.00%
(other 50% of classes)
robust
58.48%
15.86%
15.30%
CIFAR-I00
CIFAR-100
natural
80.20%
0.00%
0.00%
(50% of classes)
(same 50% of classes)
robust 64.96%
25.16%
25.56%
natural 49.66%
0.00%
0.00%
CIFAR-10
CIFAR-100
robust
41.59%
11.63%
9.68%
Table 3: Transfer learning from ImageNet (e 5).
Architecture and Source Dataset
Target Dataset
Source Model
val.
PGD-20
CW-20
natural
90.49%
0.01%
0.00%
CIFAR-10+
robust 5)
88.33%
22.66%
26.01% %
ResNet-50 ImageNet
natural
72.84%
0.05%
0.00%
CIFAR-100+
robust 5)
68.88%
15.21%
18.34%
Robust u-ResNet-50 (E 5) for CIFAR-I0+ 10+
82.00%
53.11%
Robust u-ResNet-50 5) for CIFAR-100+
59.90%
29.54%
the source and target models are trained on datasets which have similar distributions (and number of classes). robustness transfers better, and 3) validation accuracy worst we use robust model as the source compared to using conventionally trained source model: if the source model is naturally trained, natural validation accuracy better, although the target model is then vulnerable to adversarial perturbations.
4.1 TRANSFER LEARNING WITH IMAGENET MODELS
Transfer learning using models trained on ImageNet (Russakovsky er al. 2015) as the source is a common practice in industry because ImageNet feature extractors powerful and expressive. In this section we evaluate how well robustness transfers from these models.
4.1.1 TRANSFER LEARNING USING IMAGENET
Starting from both a natural and robust ImageNet model, we perform the same set experiments we did in section 4 Robust ImageNet models do not withstand untargeted foo attacks using as large an e as those that can be used for simpler datasets like CIFAR. Following the method Shafahi et al. (2019)), "free train" robust ResNet-50 ImageNet using replay nyper-parameter The hardened ImageNet classifier withstands attacks bounded by e 5. Our robust ImageNet achieves 59.05% top-1 accuracy and roughly 27% accuracy against PGD-20 fxo validation examples. We experiment with using this robust ImageNet model and conventionally trained 5o ImageNet model the source models.
4.
attacks
Using the ImageNet source models, we train CIFAR classifiers by retraining the last layer on natural CIFAR examples. We up-sample the 322mmmmmmnniion CIFAR images to 224 x 224 before feeding them into the ResNet-50 source models that are trained on ImageNet. For evaluation purposes, we also train robust ResNet-50 models from scratch using (Shafahi et al., 2019b) for the CIFAR models. To ensure that the transfer learning models and the end-to-end trained robust models have the same capacity and dimensionality, we first upsample the CIFAR images before feeding them to the ResNet-50 model. To distinguish between the common case of training ResNet models on CIFAR images that are 32 x di-dimenniona,, we call our models that are trained on the upsampled CIFAR datasets the upsample-fise ResNets "u-ResNets".
Table 3 iillsstrate that using a robust ImageNet model as a source results in high validation accuracy for the transferred CIFAR target models. Also, given that the ImageNet classifier by itself is 27% robust, the CIFAR-10 10 model maintains the majority of that 27% robustness. When we compare the end-to-end hardened classifiers (robust u-ResNets) with the transferred classifier, can that
5

---

Published as a conference paper at ICLR 2020
Deemiimal petsowacts regaus Chnlli-e
Bdt mn diemet des rigian C1
Frr Ieced Prasina
AT
(a) Clean validation
(b) Adversarial validation
(c) Average validation
When the number of training data per-class very limited (right bars), adversarially robust transfer learning [Transferred] is better in all metrics. However, as the number of training data increases (left bars), fine-tuning with adversarial examples of the target domain [Fine-tuned with AT] results in more robustness. Adversarially robust transfer learning always results in models that work better on natural examples and is 3x faster than fine-tuning with adversarial examples of the larget domain. Using pre-trained robust ImageNet improves both robustness and generalization.
Figure 3:
while the robustness is less for the transferred case, transferred models result in considerably better performance on clean validation examples.
4.2 LOW-DATA REGIME
As touched on before, transfer learning is more common situations where the number of training points in the target domain is limited. Up until now, as a proof of concept, we have illustrated the majority of our experiments on the CIFAR target domains where we have many training points per- class. Hendryykks et al. (2019) show that starting from pre-trained robust ImageNet model and fine-tuning on adversarial examples of the CIFAR domain can improve robustness beyond that of simply adversarial training CIFAR. Here, we illustrate the effect of training data size robustness and natural performance by running various experiments on subsets of CIFAR-100 where we vary the number of training points per-class (N).
We compare three different hardening methods: (1) fire-traininggaavveerrralll training the target do- main (Shafahi et al. 2019b); (2) fine-tuning using adversarial examples of the target task starting from the Free-4 robust ImageNet model similar to (Hendrycks et al. 2019); and (3) training a fully connected layer on top of the frozen feature extractors of the Free-4 robust ImageNet model using natural examples from the target task. For comparing the three different approaches, we look at three metries: (a) clean validation accuracy; (b) robustness against PGD-20 validation adversarial exam- ples; and (c) average of robustness and clean performance (()))) The results are summarized in Fig. 3 In the regimes where transfer learning is more common, adversarially robust transfer learn- ing results in the best overall performance. Adversarially/Fre training the target domain results in less robustness and validation accuracy compared to fine-tuning which highlights the importance of pre-training (Hendrycks et al. 2019). Note that in terms of computational resources required, the cost of fine-tuning on adversarial examples of the target domain is about kx our method since it requires generation of adversarial examples using k-step PGD attacks (we set k 3).
4.2.1 TRAINING DEEPER NETWORKS ON TOP OF ROBUST FEATURE EXTRACTORS
The basic transfer learning setting of section 4.I.I only re-trains one layer for the new task. In section 1.1.I, when we transferred from the robust ImageNet to CIFAR-100, the natural train- ing accuracy was 88.84%. Given the small number of trainable parameters left for the network (x= 2048 x 100) and the fixed feature extractor, the network not capable of completely fitting the training data. This means that there is potential to improve natural accuracy by learning more complex non-linear features and increasing the number of trainable parameters.
To increase representation capacity and the number of trainable parameters, instead of training a 1- layer network on top of the feature extractor, train multi-layer perceptron (MLP) network on top of the robust feature extractor. To keep things simple and prevent bottle-necking, every hidden layer add has 2048 neurons. We plot the training and validation accuracies the natural examples and

---

Published as a conference paper at ICLR 2020
the robustness (i.e. PGD-20 validation accuracy) in Fig. 4 for various numbers of hidden layers. As can be seen, adding one layer is enough to achieve 100% training accuracy. However, doing so does not result in increase in validation accuracy. To the contrary, adding more layers result in slight drop in validation accuracy due to overfitting. As illustrated, we can improve generalization using simple but effective methods such dropout (Srivastava et al. 2014) (with probability 0.25) and batch-normalization (Ioffe& & Szegedy (015)).
However, the most interesting behavior we observe in this experiment is that, as we increase the number of hidden layers, the robustness to PGD-20 attacks improves. Note, this seems to happen even when we transfer from a naturally trained ImageNet model. While for the case where we have no hidden layers. robustness is 0.00% on CIFAR100 when we use a naturally trained ImageNet model as source, if our MI.P has 2, 3, or hidden layers, our robustness against PGD) attacks would be 0.03%, 0.09%, 0.31 % and 6.61% respectively. This leads us to suspect that this behavior may be an artifact of vanishing gradients for adversary as the softmax loss saturates when the data is fit perfectly (Athalye et al.) 2018). Therefore, for this case we change our robustness measure and use the CW attack (Carlini & Wagner 2017) which will encounter fewer numerical issues because its loss function does not have softmax component and does not saturate. Attacking the model from the natural source with CW-20 completely breaks the model and achieves 0.00% robustness. Most interestingly, attacking the model transferred from robust source using the CW objective maintains robustness even when the number of hidden layers increases.
Figure 4: Training an MLP for CIFAR-100 on top of the robust feature extractors from ImageNet. The x- axis corresponds to the number of hidden layers (0 is a linear classifier and corresponds to experiments in sec- tion Robustness stems from robust feature extrac- tors. Adding more layers on top of this extractor does not hurt robustness. Interestingly, simply adding more layers does not improve the validation accuracy and just results in more overfitting (i.e. training accuracy becomes 100%). We can slightly improve generalization using batch norm (BN) and dropout (DO).
Toks
Tedy 18
(W) IV
Number of hiddes MLP banes
ANALYSIS: ROBUST FEATURE EXTRACTORS ARE FILTERS
Our experiments suggest that the robustness of neural networks arises in large part from the pres- ence of robust feature extractors. We have used this observation to transfer both robustness and accuracy between domains using transfer-learning. However, we have not yet fully delved into what it means to have a robust feature extractor. Through visualizations, Tsipras et al. (2018) studied how adversarial training causes the image gradients of neural networks to exhibit meaningful generative behavior. In other words, adversarial perturbations on hardened networks "look like" the class into which the image perturbed. Given that optimization- based attacks build adversarial examples us- ing the image gradient, we also visualize the image gradients of our transferred models to see if they exhibit the same generative behavior as adversarially trained nets.
Fig. 5 plots the gradient of the loss w.r.t. the input image for models obtained by re- re-training only the last layer, and also for the case where we train MLPs on top of robust feature extractor. The gradients for the transfer-learned models with robust source are interpretable and "look like" the adversarial object class, while the gradients of models transferred from natural source not. This interpretatbility comes despite the fact that the source model was hardened against attacks on one dataset, and the transferred model is being tested on object classes from another. Also, we see that adding more layers top of the feature extractor, which often leads over-fitting, does not make gradients less interpretable. This latter observation is consistent with our observation that added layers preserve robustness(Fig.. 4
These observations, together with the success of robust transfer learning, leads us to speculate that robust model's feature extractors act "filter" that ignores irrelevant parts of the image.
7

---

Published as a conference paper at ICLR 2020
Figure 5: Gradients of the loss w.r.t to imput images for the CIFAR-100 transfer learning experiments of sections [4.1. & 4.2.1 The top row contains sample CIFAR-100 images. Other rows contain image gradients of the model loss. The second row is for a model transferred from a naturally trained ImageNet source. Rows 3-5 are for models transferred from robust Im- ageNet source. These rows correspond to an MLP with (row 3). (row 4), and 2 (row 5) hidden layers top of the robust feature extractor. The gradients in the last three rows all show interpretable generative behavior.
END-TO-END TRAINING WITHOUT FORGETTING
As discussed in section transfer learning preserve robustness of the robust source model. However, comes at the cost of decreased validation accuracy on natural examples compared to the case where
naturally trained source model. Consequently, there seems be trade-off between generalization and robustness based on the choice of the source model. For any given clas- sifer. the trade-off between generalization and robustness is the subject of recent research (Tsipras) it al. 2018: Zhang et al. 2019) Shafahi et al. 2019a). In this section, we intend to improve the overall performance classifiers transferred from robust source model by improving their gener- alization on natural images. To do so, unlike previous sections where we froze the feature extractor mainly preserve robustness, we fine tune the feature extractor parameters A. Ideally, we should learn to perform well on the target dataset without catastrophically forgetting the robustness of the source model. To achieve this, we utilize lifelong learning methods.
Learning without Forgetting (LwF) (Li & Hoiem, 2018) is method for overcoming catastrophic forgetting. The method based on distillation. this framework, we train the target model with loss that includes a distillation term from the previous model.
min (2(x 0), y. w) + da d(z(x, (0 07)) 0
(3)
backprop
Initialize 8 0'
Distiliatoon
LwF loss
Image
Classification
loss
True
Figure 6: Our LwF loss has term that enforces the similarity of feature representations (i.e. penul- timate layer activations) between the source model and the fine-tuned model.
where, our method, Aa is the feature representation similarity penalty, and d is some distance metric between the robust model's feature representations 20(2,0)) and the current model's fea- ture representations 2(x, 8). Unlike the original LwF paper that used a distilled loss from Hinton et al. (2015) and applies distillation to the logits, we simply choose d to be the form and apply
8

---

Published as a conference paper at ICLR 2020
distillation the penultimate layer Our loss is designed to make the feature representations of the source and target network similar, thus preserving the robust feature representations (Fig. ( Ideally, 2(2, ()) 2(x 82). To speed up training, given robust feature extractor parameters 07, store (0 07) for the images of the target task and load this from memory (iee. offline) instead of performing forward pass through the robust source network online. Therefore, in the experi- ments related to LwF, we do not train with data augmentation because we have not pre-computed a 0,), where To is the augmented image. Empirically verified that d(=(x; 07), =<xe: 0?)) was not negligible
To improve performance, we follow a warm warm-start scheme and only train the fully connected parame- ters w early in training. We then cut the learning rate and continue fine tuning both feature extractors (0) and ur. In our experiments, we use a learning rate of 0.001, and the warm-start makes up half of the total training iterations. Starting from the pre-trained source model, we train for total of 20,000 iterations with batch-size 128. The results with an adversarially trained CIFAR- 100 model as source and CIFAR-10 target are summarized in Table 4
As can be seen, having a LwF-type regularizer helps maintaining robustness and also results in a considerable increase in validation accuracy. The trade-off between robustness and generalization can be controlled by the choice of Ag. It seems that for some choices of λa such as 0.1, robustness PGD-20 adversarial examples is due to the fact that the validation accuracy
also increases. However, in hindsight, the increase in accuracy is not solely due to improvement in robustness. has increased and we have better classifier overall. For easier comparisons, we have provided the transfer results without LwF at the bottom of Table 4 Note that using LwF, we can keep the robustness of the source model and also achieve clean validation accuracy comparable model that uses naturally trained feature extractors. In the supplementary, we show that similar conclusions can be drawn for the split CIFAR-100 task.
Table 4:
Distilling robust features using learning without forgetting. The bottom rows show results from transfer learnngg with a frozen feature extractor. The '+' sign refers to using augmentation.
Source + Target Dataset
Source Model
Aa
val.
PGD-20 (e 8)
Ie-7
89.07%
0.61%
0.001
86.15%
4.70%
CIFAR-100+ CIFAR-10
robust
0.0025
81.90%
15.42%
0.005
79.35%
17.61 %
0.01
77.73%
17.55%
0.1
73.39%
18.62%
natural robust
NA
83.05%
0.00%
CIFAR-100+ - CIFAR-10+
NA
72.05%
17.70%
DECREASING GENERALIZATION GAP OF ADVERSARIALLY TRAINED NETWORKS
We demonstrated in our transfer experiments that using our LwF-type loss, can help decrease the generalization gap while preserving robustness. In this section, we assume that the source domain the adversarial example domain of dataset and the target domain is the clean example domain of the same dataset. This experiment can be seen as applying transfer learning from the adversarial example domain to the natural example domain while preventing forgetting the adversarial domain.
In the case where the source and target datasets are the same (Transferring from a robust CIFAR- 100 model to CIFAR- 100), by applying our LwF-type loss, we can improve the generalization of robust models. Our results are summarized in Table
'We do since in section 3 we found the source of robustness to be the feature extractors and this obser- vation was later reinforced due to the empirical results in section
"The analysis is in the supplementary. Source code for LwF-d experiments: http://gttiikooomssaaiiik/hittth

---

Published as a conference paper at ICLR 2020
Table 5:
Decreasing generalization gap by transferring with LwF. For reference, last row shows
results from adversarial training CIFAR-100. The '+' sign refers to using augmentation. (c = 8)
Source - Target Dataset
Source Model
Ad
val.
PGD-20
CW-20
le-5
61.53%
21.83%
21.97%
5e-5
61.71%
23.44%
22.94%
le-4
61.38%
23.95%
23.40%
CIFAR-100+ CIFAR-100
robust
0.001
60.17%
24.31%
23.17%
0.01
59.87%
24.10%
22.96%
CIFAR-100+
robust
NA
59.87%
22.76%
23.16%
CONCLUSION
We identified the feature extractors of adversarially trained models as source of robustness, and use this observation transfer robustness new problems domains without adversarial training. While transferring from natural model can achieve higher validation accuracy in comparison to transferring from robust model, we can close the gap and maintain the initial transferred robustness by borrowing ideas from the lifelong learning literature. The success of this methods suggests that robust feature extractor effectively filter that sifts out relevant components of an image that are needed to assign class labels. We hope that the insights from this study enable practitioners to build robust models in situations with limited labeled training data, or when the cost and complexity of adversarial training from scratch is untenable.
Goldstein and his students were supported by the DARPA QED for RMI. program, the DARPA GARD program, and the National Science Foundation.
Acknowledgements:
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv: :xXi::880.000200, 2018.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndié, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposiun Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representation, 2015. 2015.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. arXiv preprint arXiv: 1901009960, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.022 02531, 2015.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03100 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint 161 /.0 1.11666. 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):227)2222222 1998.
10

---

Published as a conference paper at ICLR 2020
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(11):2935-233) 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint trXiv:/706.0600 2017.
Seyed-Mohsee Moosavi-Dezfooll Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2574-2582, 2016.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions knowledge and data engineering, 22(10):1345-1355 2009.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11(3))211-252, 2015.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires data. In Advances in Neural Information Processing Systems, pp. 5014-5026, 2018.
Ali Shafahi, W Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? ICLR, 2019a.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! arXiv preprint arXiv: 12843, 2019b.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-199) 2014.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yu Chen, and Yupeng Gao. robustness the cost of accuracy?-a comprehensive study on the robustness of 18 deep image classification models. Proceedings of the European Conference on Computer Vision (ECCV), pp. 631-648, 2018.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6799 2013.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint 2017.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. stat, 1050:11, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320-3328, 2014.
Sergey Zagoruyko and Nikos Komodakis. drXiv:1605.07146. 2016.
Wide residual networks.
arXiv preprint
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint 1rX1v:1901.08573, 2019.
11

---

Published as a conference paper at ICLR 2020
A EXPERIMENT DETAILS
A.1 LWF-BASED EXPERIMENTS
In our LWF-based experiments, we use batch-size of 128, fixed learning-rate of le-2m, and fine-tune for an additional 20,000 iterations. The first 10,000 iterations are used for warm-start; during which we only update the final fully connected layer's weights. During the remaining 10,000 iterations, we update all of the weights but do not update the Batch-normalization parameters.
A.2 IMAGENET TO CIFAR EXPERIMENTS
When freezing the feature extractor and fine-tuning adversarial examples, we train the last fully connected layer's weights for 50 epochs using batch-size=128. We start with an initial learning rate of 0.01 and drop the learning rate to 0.001 at epoch 30. In the case of fine-tuning examples, we generate the adversarial examples using a step PGD attack with step-size and a perturbation bound 5.
adversarial
A.3 FREE TRAINING EXPERIMENTS
In all of our training experiments where we train the u-ResNet-50, we train for 90 epochs using a batch-size of 128. The initial learning rate used is 0.1 and we drop it by factor of 10 at epochs 30 and 60. We use a replay parameter m = and perturbation bound e 5.
B THE DISTANCE BETWEEN FEATURE REPRESENTATIONS OF NATURAL IMAGES AND AUGMENTED IMAGES
To speed up the LwF experiments, did not use data augmentation during training. Instead of computing the robust feature representations on the fly, before starting training on the new target task, we passed the entire training data of the target task through the robust network and stored the feature representation vector. If we were doing data augmentation, we would have to pass the entire augmented training data through the network, which would be slow and memory intensive. Alternatively, we could use the robust feature representation of the non-augmented images instead. The latter would have been feasible if the distance between the robust feature representations of the non-augmented and augmented images were very small. However, as shown in fig 7 this quantity not often negligible.
|+[s (7) - a(x,, Bflla ix CIFARIO
[|a(x 03) afs., M/s CIFARI00
trala
0.31
01375
6.15
0.150
0.00
0.100
41775
6/5
01225
6.00
2.
13.5
Icln. 953
(a) Histogram of [læ( 07), 2(2,, l22 for CIFAR-10 dataset, given 0; for CIFAR-100 dataset. The mean for 100 dataset, given 0; for CIFAR-100 dataset. The both training and test examples is = 5.53
(b) Histogram of [ 67), 2(2,, (22 for CIFAR-
mean for both training and test examples is = 5.57
Figure 7:
Figures 7 and 7b both show that the values of [1 07), 2(xa, 0;) are high most of the time, consequently, LwF is better done without data augmentation.
12

---

Published as a conference paper at ICLR 2020
Crnertenil permmnee Oiteed del reghas be CIPA311
Isterra a dedeo dats ont be 01P55-11
Aegintent peber vifleed Cllkke
(a) Clean validation
(b) Adversarial validation
(c) Average validation
Figure 8: When the number of training data per-class very limited (right bars), adversarially robust transfer learning [Transferred] is better overall. However, as the number of training data increases (left bars), fine-tuning with adversarial examples of the target domain [Fine-tuned with AT] results in an overall better performing model. Adversarially robust transfer learning is 3x faster than fine- tuning with adversarial examples of the target domain.
C LOW-DATA REGIME TRANSFER LEARNING FROM IMAGENET TO CIFAR-10
we illustrated that robust transfer learning is most beneficial where we have limited data (i.e., limited number of training data per class). We used CIFAR- 100 as an illustrative target dataset. However, the overall results are not data set dependent. When we have limited number of training data, robust transfer learning results in the highest overall performance (see Fig. g for transferring from a robust ImageNet model to smaller instances of CIFAR-10).
In section 4.2
D LWF-BASED ROBUST TRANSFER LEARNING FOR SIMILAR SOURCE AND TARGET DATASETS
In Table we conduct LwF experiments on the split CIFAR- 100 task which is more suited for transfer learning due to the similarities between the source and target datasets. In these situations, the LwF regularizer on the feature representations still works and can improve generalization without becoming vulnerable to adversarial examples. If we take the average performance of the robust classifiers on the split tasks (average of robust half CIFAR-100 100 and the LwF setting model for Aa = 0.01) we get (63.32 + 64.96) = 64.14% average validation accuracy and 20.42% average robustness which is comparable with the case that we had adversarially trained the entire CIFAR-100 dataset (Table TT.
'Tbble 6: Distilling robust features using LwF for the split CIFAR- 100 task. For reference, we have included the results from transfer learning by freezing the features at the bottom of the table.
Source + Target Dataset
Source Model
Ad
val.
PGD-20
0.001
73.30%
1.92%
0.005
66.96%
10.52%
CIFAR-100+ (1/2) - CIFAR-100 (other 1/2)
robust
0.01
63.32%
15.68% %
0.1
55.14%
17.26
natural
NA
71.44%
0.00%
CIFAR-100+ (1/2) + CIFAR-100+ (other 1/2)
robust
NA
58.48%
15.86%
E IMPROVING GENERALIZATION OF THE CIFAR- 10 ADVERSARIALLY TRAINED MODEL
Similar to the case of improving the generalization for CIFAR-100, we use our LwF-d loss func- tion to transfer from the robust CIFAR-I0 domain to the natural CIFAR-10 domain. We summarize the results Table
13

---

Published as a conference paper at ICLR 2020
Table 7:
Decreasing generalization gap by transferring with LwF. For reference, last row shows
results from adversarial training CIFAR-10. The '+' sign refers to using augmentation. (e 8)
Source -
Target Dataset
Source Model
Aa
val.
PGD-20
CW-20
Ie-5
88.
16.
45.31%
46. 15%
5e-5
88.08 %
46.24%
47.06% %
le-4
87.81% 8
46.54%
47.29%
CIFAR-I0+ CIFAR-10
robust
5e-4
87.44%
46.36%
47.16%
0.001
87.31%
46.27%
47.06%
0.01
87.27%
46.09%
46.92%
0.1
87.49%
46.11%
46.84%
CIFAR-10+
robust
NA
87.25%
45.84%
46.96%
14


Reviews:
[{"review_id": "r1xHX9DptH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies transfer learning from the point of view of adversarial robustness. The goal is, given a robust deep neural network classifier for a source domain, learn a robust classifier for a target domain as efficiently and with as few samples as possible. The authors empirically evaluate different strategies and compare with relevant baselines.\n\nAt a high level, the paper addresses an interesting problem. Robust models are quite computationally and sample intensive to train, so exploring pre-training is a reasonable way to deal with small datasets or computational constraints.\n\nThe authors perform a diverse set of experiments from which I identified the following individual contributions:\n\na) Retraining the last layer of the model on natural examples preserves robustness. Robustness degrades smoothly when pre-training progressively more layers.\nThis is an interesting contribution providing evidence that robust models do learn in fact robust input representations/features.\n\nb) Transferring a learned robust representation from a source to a target domain preserves its robustness (training a linear layer on top of it leads to a robust classifier).\nThis provides further evidence that robust models learn _general purpose_ robust features of the input, while establishing robust pre-training as a valid strategy for cheaper robust models. The baselines considered are: 1) adversarial training on target domain which always underperforms the proposed method, 2) fine-tuning on adversarial samples from the target domain which performs better when there are a lot of samples from that domain and worse when there are only a few (this method is also more computationally expensive than transfer learning).\n\nc) The \"perceptually-aligned\" saliency maps of Tsipras et al. 2018 are also a property of robust models obtained through transfer learning.\nThis illustrates that these saliency maps can also arise for out-of-distribution inputs and hence are likely to correspond to general, high-level features.\n\nd) Fine-tuning all layers of the model while ensuring that the representations stay close to the original ones for natural examples can lead to transfer with improved validation accuracy and robustness (even when the source and target domains are the same).\nThis is an interesting improvement over the simpler transfer methods producing competitive results.\n\nOverall, the paper contains an experimental study that, in my opinion, is thorough, presents interesting findings, and contains the necessary ablations. I believe that this paper would be of interest to the adversarial ML community and I hence recommend acceptance.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SyxAm1RaFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------\nThis paper addresses the problem of performing robust transfer learning. A first contribution of the paper is to robust and classic training with respect to usual validation accuracy and robustness to adversarial attacks on the CIFAR task. Then, the same comparison is made on a transfer learning task. The transfer learning setting is then completed by studying transfer from ImageNet-based models with a particular attention to low-data regime and training deeper networks on top of the feature extractor. An analysis of robust features is provided and finally the authors studies the interest of  Learning without Forgetting strategies to provide robust transfer. The tendency s to obtain the Best performance from robust-trained source models having a good validation accuracy.\n\n\nOverall\n------\nThe paper presents  a study of robust transfer learning that can be interesting for practitioners to know the type of results that can be obtained by robust transfer learning. However, I feel that the results obtained are rather expected and the paper does not provide some interesting methodological contribution that could help to develop robust transfer training. \n\nComments\n---------\n\nThe results obtained in Section 3, 4 and 6 are rather expected and similar.  I think that the paper could benefit by reducing these 3 sections in only one section where the results obtained can be summarized in one big table and two or three figures for example - the complete set of results can then be reported in the supplementary section.\n\nThen, if the contribution of the paper is to propose to focus on robust transfer learning including a Learning without Forgetting strategy, the authors should then focus more on this part and analyze better the behavior of learning.\nIn particular, the combination between distillation and robust training is certainly interesting, and trying to propose a methodological framework for doing robust training in this context would certainly result in a more significant contribution. How to constrain the feature extraction layers, how to make use of them with distillation and additionally what are the additional contraints/additions that can be made to learning problem (3) to improve robust transfer are some important questions. \n\nSo far, the contribution appears to me rather limited for ICLR. If we restrict to the part related to experimental comparisons made, they are restricted to particular trainings and datasets with specific PGD attacks. The contribution would have been stronger is different types of adversarial attacks with different parameters have been studied and analyzed. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gqPAioYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary: This paper explores the problem of robustly transfer learning using only standard training (as opposed to adversarial training (AT)) on the target domain. The authors start by highlighting that intermediate representations learned by adversarially trained networks are themselves fairly robust. Then they propose two strategies for robust transfer from a robust model trained on the source domain: (1) naturally fine-tuning the final linear layer on the target domain and (2) naturally fine-tuning all the layers using lifelong learning strategies. They study transfer between CIFAR10 and CIFAR100, as well as, from ImageNet to CIFAR10/100.\n\nHigh-level comments: Overall, I find the paper interesting and well-written. Prior work from Hendrycks et al. showed that AT on the source domain followed by *adversarial* fine-tuning on the target domain attains better performance as compared to just AT on the target domain. The main contribution of this paper is to show that using instead careful *natural* fine-tuning on the target domain is sufficient to recover a reasonable amount of this robustness. \n\nEven though the clean/robust accuracy of the proposed approach is lower than just doing adversarial training/prior work from Hendrycks et al., I feel this paper could be useful to the community for two main reasons:\n\n1. The authors perform a nice exploration of the thesis that robust models have robust representations, and how this connects to transfer learning. In particular, the experiments in Figure 1 (the effect of naturally re-training later layers of a robust network on its robustness) and Figure 5 (where the authors show that their naturally fine-tuned models have some of the unexpected benefits from Tsipras et al.) seem particularly interesting.\n\n2. Despite its lower accuracy, this approach could be useful in settings where data is scarce or compute is expensive, and hence adversarial training on the target domain is not successful. \n\nSpecific comments/questions:\n\ni. Could the authors clarify what they mean by point 3 (re: validation accuracy drop) below Table 3? As far as I can tell, the drop in clean validation accuracy between Table 1 and Table 2 are similar for both the naturally and adversarially pre-trained models.\n\nii. In Table 2, it would also be interesting to see the performance when the source domain is CIFAR10 and the target domain is CIFAR100.\n\niii. For Table 2 is the eps=8? This should be mentioned in the caption as it is important to highlight that Tables 2 and 3 are not directly comparable.\n\niv. Why isn’t experiment in Figure 3 should be repeated for CIFAR10 as well? The authors should add this result to the paper, even if in the appendix. It is important to verify that this trend is not specific to CIFAR100 and holds across datasets (even though CIFAR10/100 are not too different).\n\nv. The comment at the end of page 6 re: natural model is confusing (“Note, this seems to...perfectly”)---as far as I can tell, Figure 4 does not include the results of fine-tuning a naturally trained model.\n\nvi. General comment motivated by the comment (\"Note...perfectly\") mentioned 5 above: For all the adversarial evaluation in the paper, the authors should also try CW attacks/black-box attacks to get a more confident estimate of their model’s robustness.\n\nvii. The authors reference prior work on the tradeoff between robustness and accuracy and motivate Section 6 as an avenue to alleviate this trade-off for their model. However, I don’t see the lower performance of their model as an instance of this trade-off---the model in the paper performs worse in terms of both clean and adversarial accuracy. The approach proposed in Section 6 seems interesting, but more as an approach to improve the *overall* performance of the model. The authors mention this in retrospect, but I think the narrative of this section should be modified to make this clearer.\n\nviii. In Table 5, in the experiments corresponding to CIFAR100+ -> CIFAR100, is the dataset split into two halves (for the source and target domains) or is the fine-tuning performed on the same data. In general, I find it odd that natural fine-tuning on the *same data* can improve both the clean and adversarial accuracy of the model (compared to the CIFAR100+ robust baseline). Is the robust model trained long enough/with enough hyperparameter search?\n\nOverall, the exploration in the paper seems novel and could be useful to the community. Thus, I recommend acceptance.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an empirical study towards understanding the transferability of robustness (of a deep model against adversarial examples) in the process of transfer learning across different tasks.

The paper received divergent reviews, and an in-depth discussion was raised among the reviewers.

+ Reviewers generally agree that the paper makes an interesting study to the robust ML community. The paper provides a nice exploration of the hypothesis that robust models learn robust intermediate representations, and leverages this insight to help in transferring robustness without adversarial training on every new target domain. 

- Reviewers also have concerns that, as an experimental paper, it should perform a larger study on different datasets and transfer problems to eliminate the bias to specific tasks, and explore the behavior when the task relatedness increases or decreases.

AC agrees with the reviewers and encourages the authors to incorporate these constructive suggestions in the revision, in particular, explore more tasks with different task relatedness.

I recommend acceptance, assuming the comments will be fully addressed.
Decision: Accept (Poster)