Adversarial Feature Desensitization
Authors: Pouya Bashivan
Reza Bayat
Adam Ibrahim
Kartik Ahuja
Mojtaba Faramarzi
Touraj Laleh
Blake Aaron Richards
Irina Rish
Venue: NeurIPS 2021 Poster (2021)
Keywords: adversarial robustness
adversarial learning
adversarial examples
domain adaptation

Abstract: Neural networks are known to be vulnerable to adversarial attacks -- slight but carefully constructed perturbations of the inputs which can drastically impair the network's performance. Many defense methods have been proposed for improving robustness of  deep networks by training them on adversarially perturbed inputs. However, these models often remain vulnerable to new types of attacks not seen during training, and even to slightly stronger versions of previously seen  attacks. In this work, we propose a novel approach to  adversarial robustness, which builds upon the insights from the domain adaptation field. Our method, called Adversarial Feature Desensitization (AFD), aims at learning  features that are invariant towards adversarial perturbations of the inputs. This is achieved through a game where we learn features that are both predictive and robust (insensitive to adversarial attacks), i.e. cannot be used to discriminate between natural and adversarial data. Empirical results on several benchmarks  demonstrate the effectiveness of the proposed approach against a wide range of attack types and attack strengths. Our code is available at https://github.com/BashivanLab/afd.

Adversarial Feature Desensitization
Pouya Bashivan 1.2.
Reza Bayat
Adam Ibrahim
Kartik Ahuja²
Mojtaba Faramarzi
Touraj Laleh
Blake Richards
Irina R
McGill University, Montreal, Canada
2 MILA, Université de Montréal, Montreal, Canada *Correspondence to: (bashivap, irriaann rishhmmila quebec
Abstract
Neural networks are known to be vulnerable to adversarial attacks slight but carefully constructed perturbations of the inputs which can drastically impair the network's performance. Many defense methods have been proposed for improving robustness of deep networks by training them on adversarially perturbed inputs. However, these models often remain vulnerable to new types of attacks not seen during training, and even to slightly stronger versions of previously seen attacks. In this work, we propose novel approach to adversarial robustness, which builds upon the insights from the domain adaptation field. Our method, called Adversarial Feature Desensitization (AFD), aims at learning features that are invariant towards adversarial perturbations of the inputs. This achieved through a game where we learn features that are both predictive and robust (insensitive to adversarial attacks), i.e. cannot be used to discriminate between natural and adversarial data. Empirical results several benchmarks demonstrate the effectiveness of the proposed approach against a wide range of attack types and attack strengths. Our code is available at whttts::/// /i thub. com/ Bashi vanLab/afd
Introduction
When training classifier, common assume that the training and test samples are drawn from the same underlying distribution. In adversarial machine learning, however, this assumption is intentionally violated by using the classifier itself perturb the samples from the original (natural) data distribution towards new distribution over which the classifier's 's error rate is increased [52]. As expected, when tested on such adversarially generated input distribution, the classifier severely underperforms. To date, various methods have been proposed to defend the neural networks against adversarial attacks [34] 2. additive noise patterns and corruptions [24 25 [4), and transformations 17.. Among these methods, two of the most successful adversarial defense methods to date are adversarial training (34), which trains the neural network with examples that are perturbed to maximize the loss the target model, and TRADES [57], which regularizes the classifier to push the decision boundary away from the data. While past adversarial defence methods have successfully improved the neural network robustness against adversarial examples, it has also been shown that these robust networks remain susceptible to even slightly larger adversarial perturbations or other forms of attacks 19 46 48).
In this paper, we propose to view the problem of adversarial robustness through the lens of domain adaptation, and to consider distributions of natural and adversarial images as distinct input domains
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

that a classifier is expected to perform well on. We then focus our attention on learning features that are invariant under such domain shifts. Building upon domain adaptation literature [4] we use the classification-based H HAA--divergence to quantify the distance between the natural and adversarial domains. The theory of domain adaptation allows us to formulate a bound on the adversarial classification error (i.e. the error under the distribution of adversarial examples) in terms of the classification error on natural images and the divergence between the natural and adversarial features.
We further propose an algorithm for minimizing the adversarial error using this bound. For this, we train a classifier and a domain discriminator to respectively minimize their losses on the label classification and domain discrimination tasks. The feature extractor is trained to minimize the label classifier's loss and maximise the discriminator's loss. In this way, the feature extractor network is encouraged to learn features that are both predictive for the classification task and insensitive to the adversarial attacks. The proposed setup is conceptually similar to prior work in adversarial domain adaptation [18] [53, where domain-invarian features are learned through an adversarial game between the domain discriminator and feature extractor network.
This setup is similar to the adversarial learning paradigm widely used in image generation and transformation 20 28 60]. unsupervised and semi-supervised learning (39], video prediction 31
active learning 47. and continual learning [16]. Some prior work have also considered adversarial
learning to tackle the problem of adversarial examples [54 36 g These methods used generative
models to learn the distribution of the adversarial images [34] 33). or to learn the distribution of imput
gradients [9 8. Unlike our method which learns a discriminator function between distributions of adversarial and natural features and updates the feature extractor to reduce the discriminability of those distributions.
The main contributions of this work are as follows:
We apply domain adaaaiatn theory to the problem of adversarial robustness; this allows to bound the adversarial error in terms of the error the natural inputs and the divergence between the feature (representation) distributions adversarial and natural domains.
Aiming to minimize this bound, we propose a method which learns adversarially robust features that both predictive and insensitive to adversarial attacks, i.e. cannot be used to discriminate between natural and adversarial data.
We empirically demonstrate the effectiveness of the proposed method in learning robust models against wide range of attack types and attack strengths, and show that our proposed approach often significantly outperforms most previous defense methods.
Related Work
There is an extensive literature on mitigating susceptibility to adversarial perturbations 34 57
13 59, 3, (22 77). Adversarial training (34) is one of the carliest successful attempts to improve robustness of the learned representations to potential perturbations the input pattern by solving min-max optimization problem. TRADES (57) adds a regularization term to the cross-entropy loss which penalizes the network for assigning different labels to natural images and their corresponding perturbed images. [41] proposed an additional regularization term (local linearity regularizer) that encourages the classification loss to behave linearly around the training examples. [55 51 proposed to regularize the flatness of the loss to improve adversarial robustness.
Our work is closely related to the domain adaptation literature in which adversarial optimization has recently gained much attention [8] 32 53 From this viewpoint one could consider the clean and perturbed inputs as two distinct domains for which a network aims to learn an invariant feature set. Although in our setting, i) the perturbed domain continuously evolves while the parameters of the feature network are tuned; ii) unlike the usual setting in domain adaataaiont problems, here we have access to the labels associated with some samples from the perturbed (target) domain. Recent [or regularized the network to have similar logit values response clean and perturbed inputs and showed that this additional term leads to better robust generalization to unseen perturbations. Related to this, Adversarial Logit Pairing (27 increases robustness by directly matching the logits for clean and adversarial inputs. JARN 9 Another line of work is on developing certified defenses which consist of methods with provable bounds over which the network is certified to operate robustly [58 56 10 While these approaches provide sense of guarantee about the proposed defenses, they

---

caturbe
Fa
Ci
Clsot
Cland
Calogiory
D
toweds
avayy
clean Perturodd
Adversarial Training
TRADES
AFD
(a)
(b)
Figure 1: (a) Overview of the proposed AFD approach; (b) Visual comparison of adversarial robustness methods (Adversarial training 34, TRADES [57] and AFD), The dashed red and dotted black lines correspond to the decision boundary of classes and the domain discriminator respectively.
are usually prohibitively expensive to train, drastically reduce the performance of the network on natural images, and the empirical robustness gained against standard attacks is low.
3 Our approach
We will now make a connection between the domain adaptation and adversarial robustness, and build upon this connection to develop an approach for improving the network's robustness against adversarial attacks.
3.1 Preliminaries
Let Fo(x) X' - 2, where X' C R", 2 C Rmm be feature extractor (e.g. a neural network with parameters 0) mapping the input
X into the feature vector (representation) Co 2 - V, where )' (l...., ......,KK K} are the class labels, be a classifier, with parameters ø (e.g., the last linear layer of neural network plus the softmax function, on top of the extracted features).
2, and let
Adversarial attack: Let क (xx denote a perturbation function (an adversarial attack) which, for given (z, y) E X x y. generates perturbed sample x' E B(x, e) within the e-neighborhood of z, B(x, <) = (x E X: : || æ|| < c). by solving the following maximization problem
max tEB(x,e)
((C((Fo
where C is the task classification loss function. In practice, however, the perturbed sample a' found by an attacker is typically an approximate rather than the exact solution to this maximization problem.
In order to characterize the distance between the natural and adversarial data distributions, the following notion distance between two probability distributions, defined in [4 [8], will be used later to make connection with domain adaptation theory.
Let H be set of binary classifiers (hypotheses), called hypothesis space: then the symmetric difference hypothesis space HAH defines the set of hypotheses that capture the disagreements between two hypotheses in H., as in [4]:
HAH-distance:
g E HAH = g(x) = h(x) @ N'(x)) for some h, N E H,
where @ denotes the XOR function. Then the HAH-distance 4 48 between two data distributions (domains) S and T, with respect the hypothesis space H. is defined as:
dHAn(S,T)= =
sup Pa-sshhas = 1 - Pa-T [b(x) = 1]1-
(3)
HEHAH
This equation turns into an inequation when the supremum is taken over the hypothesis space H. instead of HAH (18.
3.2 A Domain Adaptation View of Adversarial Robustness
domain is defined as a data distribution on the set of inputs X' [5]. In the adversarial robustness setting, we consider two domains the natural and the adversarial domains, corresponding respectively
3

---

to the source and target domains in domain adaptation. We denote by Dx and D'y the natural and adversarial distributions of input instances respectively and by Dz and D'z their corresponding induced distributions over the feature space 2. As in domain adaptation, we assume that f X - y a labeling function common to both domains. The expected classification error ez of the classifier Co over Dz is defined as the probability that the classifier Co disagrees with the function f:
ez(Co) = DI (y + Ca(z)],
(4)
where f: - y/ is a mapping from the features to the class label such that f(x) f (()())) We similarly define <zz as the expected error of Cs over Dz'. Using theorem 2 from 4 that relates the source and the target domain errors, we get an upper bound on the expected adversarial error c'2 as:
(2) S cz (h) +
2 ddHAH(D,, D'2) +c,
(5)
where e is a constant term w.r.t. h. Eq. 5 @esenttially gives a bound on the adversarial error e'z in terms of the natural error eg and divergence dHAh between the natural and adversarial domains with respect to their induced representation distributions D2 and D'z. In the next section, we will describe an algorithm for improving adversarial robustness of model by iteratively estimating and minimizing these two components of the error bound.
3.3 Adversarial Feature Desensitization
Based on Eq. [] the expected adversarial error could be reduced by jointly minimizing the nat- ural error and the divergence between the distributions of natural and adversarial representations dHAH( (Dz, D'z). While minimizing the natural error ex is straightforward, minimizing the cross- domain divergence requires us to estimate d D'z). domain discriminator De is closely related to estimating the dHan (Dz, 2. D' ). The domain discrimi- nator is a classifier trained to assign a label of to samples from D2, and -1 to samples from D'z- Namely, it is shown 188 that
As was shown before [88. training a
dHAH((Z, (Dz, S 2 supl NEH
(h) 1/,
(6)
where O'22,D's (h) PZ-D2 [x= 1] + + [M(x) = -1] combines the true positives and true negatives, and is thus maximized by the optimal domain discriminator h = De. Note that, if the domain distributions Dz and D'z are the same, then even the best choice of domain discriminator De will achieve chance-level accuracy, corresponding to O'Dz.D', (Dw) 1. Our approach will aim at minimizing this estimated distance dHAx(DZ D'z) ) by tuning the feature extractor network parameters 0 in the direction that pushes the distributions Dzz and D'z closer logether. In parallel, we train the domain discriminator to estimate and guide the progress of the feature extractor's tuning.
We now describe the proposed approach (see Algorithm which essentially involves simultancous training of the feature extractor Fg. the task classifier Co and the domain discriminator Di (see Figure la One iteration of the training procedure consists of the following three steps.
First, parameters of the feature extractor Fe and classifier Co are updated aiming to minimize the natural error ex using the cross-entropy loss on natural inputs:
Lo= = - 1 Σ Js log (softmax(C,(Fo(x)))))
where ū; is a one-hot encoding of the true label of the i-th sample 24-
Next, steps two and three essentially implement two-player minimax game similar that in Generative Adversarial Networks (GAN) (20. carried out between the feature extractor network Fa and the domain discriminator Des, with value function
e)][] + Eg(u) (SD) v)]]]. (8) Note that we will somewhat abuse the notation, assuming that Co and Dr below correspond to the logits (last-layer output) of the corresponding networks. Also, we will use class-conditionss discriminators. DeoFo (z. v)), ie. train different domain discriminator for different label values y.
(iee, D+) = Ep(x)

---

Algorithm 1: AFD training procedure
Input: Adversarial perturbation function (attack) и, feature extractor Fo; task classifier Cø. domain discriminator Dæ,, learning rates ot f3, and 7y.
repeat
input next mini-batch {1r U1), (Imm Vm)]
for m: x < T E)
Compute Le according to Eq.
Compute LD according to Eq.
Compute Lp according to Eq.
I0
(0, Φ) f (0. Φ)
aVe. PCC % update feature extractor and task classifier
sb + e BWLLD
% update domain discriminator
- VoLr until convergence;
% update feature extractor
where is the softplus function. In particular, parameters of the domain discriminator D3 updated to minimize the cross-entropy loss associated with discriminating natural and adversarial inputs, maximizing ahh) in Eq. 6
CD Σ ((-Dv(Fe(a)) w)) + S(D(((F(()) u))]]
(9)
while the parameters the feature extractor function Fg are adversarially updated to maximize domain discriminator's loss from Eq. 9
m
CF
m Σ S( (Fo 9:))
(10)
In Figure Db. we visually compare the learning dynamics in adversarial training, TRADES and AFD. Essentially, the adversarial training solves the classification problem by pushing the representation of adversarial examples from different classes away. TRADES regularizes the normal classification loss on the natural inputs with an additional term that encourages the representation of adversarial and natural images to match. Similar to TRADES, in AFD, the regular classification loss natural inputs is augmented but with an adversarial game which consists of training the domain discriminator that distinguishes between the adversarial and natural inputs for each class followed by updates to the feature extractor to make the representations for natural and adversarial examples to become indistinguishable from each other. Notably, because the parameter update for the feature extractor network is done to maximize the domain discriminator loss and not to decrease the loss for particular adversarial examples (as is done adversarial training or TRADES), potentially increases the network robustness against any perturbation that could be correctly classified using the same domain discriminator. This could potentially lead to broader form of generalization learned by the network.
Relation to Adversarial Training. Adversarial training minimizes the expected error on adversarial examples (the perturbed versions of the natural samples), generated by attacker in order to maximize the classification loss. The adversarial training procedure involves minimax optimization problem consisting of an inner maximization to find adversarial examples that maxi- mize the classification loss and an outer minimization to find model parameters that minimize the adversarial loss. From the domain adaptation point of view, the inner optimization of adversarial training is equal to a sampling procedure that generates samples from the target domain. Intuitively, direct training of the classifier on samples from the target domain would be the best way improve the accuracy in that domain (i.e. adversarial classification accuracy). However, it's important to note that the adversarial examples found through the inner optimization only approximately maximize the classification loss, and therefore the adversarial eтror associated with these samples only act as a lower bound on the adversarial error and therefore the outer loop of the adversarial training method essentially minimizes lower bound on the adversarial classification error. In contrast to this setup, our proposed method minimizes conservative upper bound on the adversarial error and therefore is more likely to generalize to a larger set of unseen attacks, and to stronger versions of previously seen attacks (i.e. ones that generate higher-loss samples the inner optimization loop).
Discussion:
5

---

4 Experiments
4.1 Experimental setup
Datasets. We validated our proposed method on several common datasets including MNIST (0) CIFARIO, CIFAR [29], and tiny-Imagene, (26). The inputs for all datasets were used in their original resolution except for tiny-Imagenet where the inputs were resized to 32 x 32 to allow the experiments to finish within reasonable time on two GPUs.
Adversarial attacks. To fairly assess the generalization ability of each defense method across attack types, we tested each network on 9 well-known adversarial attacks from the literature, using existing implementations from the Foolbox 42] and Advertorch [12] Python packages. Namely, we tested the models against different variations of the Projected Gradient Descent (PGD) [34] (Log+ Lg. L1), Fast Gradient Sign Method (FGSM) (2). Momentum Iterative Method (MIM) 14. Decoupled Direction
and Norm (DDN) [4], Deepfool 40 C&W (6, and Autokt [T] attacks. Also to assess the generalization in robustness across stronger adversarial attacks, for each attack we also varied the value across wide range and validated different models on each. Specific hyperparameters used for
each attack are listed in Table-A2
Feature extractor network F' and classifier Cg. We used the same network architecture, ResNet18 [23 for the feature extractor and classifier networks in experiments on all datasets and only increased the number of features for more challenging datasets. The number of base filters in the ResNet architecture was set to 16 for MNIST and 64 for other datasets. We used the activations before the last linear layer as the the output of the feature extractor network (Z) and the last linear layer as the classifier network Cg- We added an activation normalization layer to the output of feature extractor network to provide normalized features to both Co and D& networks.
We compared several variations of the domain discriminator architecture and evaluated its effect on robust classification on MNIST dataset (Table A5 Overall, we found that using deeper networks for domain discriminator and adding projection discriminator layer improves the robust classification accuracy. The number of hidden units in all layers of D were equal (64 for MNIST and 512 for other datasets). Following the common design principles in Generative Adversarial Networks literature, we used the spectral normalization (37] on all layers of De. In all experiments, the domain discriminator (Dv) consisted of three fully connected layers with Leaky ReI.U nonlinearity followed by a projection discriminator layer that incorporated the labels into the adversarial discriminator through dot product operation 38 Further details of training for each experiment are listed in TAb
Domain discriminator network De
Training parameters and baselines. All networks including baselines were trained on an adaptive version of PGD attack (II) that adaptively tunes the step size during the attack with virtually no computational overhead compared to standard PGD attack. We used e = 0.3, 0.031, and 0.016 for MNIST, CIFAR, and finy-Imagemt randomly split the CIFAR 10 train set into a train and validation sets (45000 and 5000 images in train and validation sets respectively). We then carried out a grid-search using the validatii sets and picked the learning rates with highest validation performance. Based on this analysis, we selected the learning rate 7 = 0.5 for tuning the feature extractor Fo, and a = B = 0.1 for tuning the parameters in domain discriminator De, and the task classifier Co-
datasets respectively. To find the best learning rates, we
In all experiments we trained two versions of the AFD model, one with losses Lp and LF according 10 to Eq. 9 I0 which we call AFD-DCGAN and another version where we substitute the losses with
those from the Wasserstein GAN (I dubbed AFD-WGAN (see Eq. and 12 in the Appendix). We mainly compared the performance of our proposed method with two prominent defense methods, adversarial training and TRADES. We used re-implementation of adversarial training (AT) method 34] and the official code for TRADES? ? (57) and denoted these results with i in the tables. All experiments were run on NVIDIA V100 GPUs. We used one GPU for experiments on MNIST and GPUs for other datasets.
4.2 Robust classification against nominal attacks
We first evaluated our method against adversarial attacks under similar settings to those used during training (e = 0.3. 0.031, and 0.015 for MNIST, CIFAR, and Tiny-Imagent datasets respectively).

---

Table I: Comparison of adversarial accuracy against various attacks on different datasets. For PGD attack we used c = 0.3, 0.031, and 0.015 for MNIST, CIFAR CIFARROOCIFARIOM, and Tiny-Imagenet datasets respectively and for C attack we used = 1 for all datasets. i indicates replicated results. NT: natural training; AT: adversarial training: AFD: adversarial feature desensitization; WB: white-box attack; BB: black-box attack where the adversarial examples were produced by running attack on the NT R model. Numbers reported with μu + a denote mean and std values over three independent runs with different random initialization. RST additionally uses 500K
unlabeled images during training.
Method
Dataset
Network
Clean
PGD (WB)
C&W, (WB)
AA.,, (WB)
PGD,, (BB)
C&W, (BB)
NTİ AT 14
RN18
98.84
62.43
50.82
96.48
RN18
99.35
95.66
92.78
89.99
98.92
98.95
MNIST
TRADES
99.14
98
98.57
AFD-DCGAN
RNI8
99.24
95.72
93.78
88,79
98.65
98.49
AFD-WGAN
RN18
99.14
97.68
97.68
90.12
98.59
98.71
AT TRADES 57
RN18
87.3
45.8
86.0
RN18
84.92
56.61
87.60
RI
WRN-32
82.72
75.75
RST 75
WRN-34-10
89.82
64, 86
WRN-28-8
86.83
JARN 9
CIFARIO
WRN-34-10
84.8
46,7
NTI
RN18
94,89
0.55
17.93
AT
RN18
85.92
40.07
40,27
36.14
85.14
85.84
TRADES
RNI8
81.94
53.3
40.24
43.48
80.82
81.74
AFD-DCGAN
RN18
86.82
44.35
50,93
34.46
85.73
86.68
AFD-WGAN
RN18
85.95
$9.38
62.43
37,33
84,74
85.79
NT!
RNI8
76.76
0.01
0.52
RN18
56.49
18.54
17.71
18.30
56.07
56.42
TRADES[S7]
CIFARIOD
RN18
60.32
25,11
20.52
21.10
59.62
60.29
AFD.DCGAN AFD-WGAN
RN18
60.95
18.06
21.47
16.31
60.31
60.86
RN18
58.87
22.35
25.33
18.00
58. 15
58,75
NTI A TRADES 57 AFD-WGAN
Tiny-IN
RN18
43,80
12.62
4.90
9.48
41.87
42,86
37.70
13.26
36.26
RNI8
47.70
11.49
5.90
9.45
43,5
44.69
Table [alleeecosrrsss the robust classification performance of AFD and several other defense methods against L.D C& L2 and AutoAttack white-box black-box attacks. The black-box attacks were carried out by constructing the adversarial examples using ResNet18 architecture trained on the natural inputs I ~ xx- Overall both versions of AFD (AFD-DCGAN and AFD-WGAN) were highly robust against all five tested attacks while maintaining higher "Clean" accuracy (on natural data) compared to strong baseline models like TRADES and Adversarial Training. AFD-WGAN was consistently at the top MNIST and CIFAR10 datasets. On CIFARI 100 and Tiny-Imagennd AFD performed better than or similar to Adversarial Training on all the attacks and performed better than TRADES most of the attacks, although it occasionally behind TRADES (on PGD-L and AA white-bor box attacks). Analysis of feature sensitivity showed that on MNIST and CIFAR1O datasets on which AFD outperformed the other baselines by larger margin, the features more insensitive to adversarial perturbations and over a larger range of attack strengths (Figure-AA) In addition these tests, also evaluated the AFD model against transfer black-box attacks from Adversarial Training and TRADES models which further demonstrated AFD's 's higher robustness to those attacks too AT3
significantly
4.3 Robust classification against stronger and unseen attacks
To evaluate how each network generalizes unseen domains of adversarial inputs (i.e. adversarial attacks generated with unseen forms of adversarial attacks), we additionally validated the classification robustness against a range of possible values for several widely used attacks that were not used during training. To fairly compare different models while considering both attack types and values, we computed the grea-under-the-oun 2). Table-2] summarizes the AUC values for all 9 attack methods on four tested datasets. Compared with baselines, we found that, AFD-trained networks consistently performed better on various datasets and on almost all the tested attacks even for substantially larger values (Figure 2, also see Figures A A3 in the appendix). These results show that compared to other baselines, AFD- FFDttriind networks are robust against wider range of attacks and attack strengths (e). This further
(AUC) for accuracy vs. epsilon for each attack (similar to
7

---

Inlopd
Rogt
Opge
1gem
25-
Infrim
decsfool
Figure 2: Adversarial accuracy for different methods against white-box attacks on CIFARIO dataset with ResNet18 architecture.
Table 2: AUC measures for different perturbations and methods on MNIST, CIFARIO, CIFARIO0, and tiny-imagene datasets. AUC values are normalized to have maximum allowable value of Evaluations on AT and TRADES were made on networks trained using reimplemented or official
code.
Dataset
Model
PGD,..
PGD
PGD
FGSM
MIM
DDN
DeepFool
C&W
AA
NT
0.16
0.06
0.07
0.3
0.19
0.09
0.21
0.57
0.28
AT
0.74
0.29
0.19
0.83
0.95
0.49
0.55
0.87
0.89
MNIST
TRADES
0.71
0.26
0.15
0.47
0.86
AFD-DCGAN
0.77
0.33
0.3
0.78
0.91
0.51
0.49
0.9
0.88
AFD-WGAN
0.92
0.54
0.55
0.9
0.98
0.68
0.63
0.94
0.90
NT
0.05
0.1
0.17
0.19
0.05
0.1
0.16
0.1
0.12
AT
0.28
0,2
0.44
0.33
0.31
0.26
0.29
0.31
CIFARIO
TRADES
0.32
0.22
0.5
0.32
0.33
0.18
0.28
0.25
AFD-DCGAN
0.34
0.54
0.43
0.4
0.31
0.4
0.43
0.47
0.22
AFD-WGAN
0.56
0.54
0.66
0.59
0.56
0.4
0.52
0.62
0,24
NT
0.03
0.08
0.1
0.07
0.03
0.08
0.06
0.08
0.09
0.13
0.1
0.24
0.14
0.12
0.15
CIFARIO0
TRADES
0.16
0.13
0.31
0.12
0.17
0.18
0.1
0.16
0.15
AFD-DCGAN
0.14
0.12
0.27
0.17
0.16
0.15
0.16
0.18
0.13
AFD-WGAN
0.18
0.16
0.31
0.22
0.19
0.16
0.19
0.23
0.13
NT
0.04
0.03
0.08
0.05
0.04
0.06
0.07
0.07
0.07
Tiny-IN
0.10
0.03
0.16
0.15
0.09
0.14
0.13
0.11
TRADES
0.10
0.03
0,16
0.07
0.09
0.15
0.11
0.09
0.16
AFD-WGAN
0.10
0.04
0.19
0.12
0.09
0.15
0.16
0.12
0.15
suggests that the features learned through AFD generalize better across various forms of attacks and sustain larger perturbations.
also observed that the AFD-WGAN performs better than AFD-DCGAN under most tested conditions. This is potentially due to: 1) WGAN's ability to avoid vanishing gradients when the discriminator becomes too good compared to the generator (the feature extractor function in our case) [5]; 2) WGAN's ability to avoid mode-collapses during training. In training GANs, mode collapses lead to the generator network to only output a limited set of patterns instead learning to produce a diverse set of natural-looking images that fool the discriminator. Under our setting, WGAN potentially leads to learning a feature extractor that can produce a more diverse set of features for perturbed inputs, instead of focusing subset of latent dimensions. This suggests that applying more advanced GAN training algorithms could potentially further improve the robust performance in AFD-type models.
4.4 Estimated HAH--distance and adversarial-vs-natuee generalization gap
As stated in Eq. the upper bound on the adversarial error can be stated in terms of the natural error, the divergence between the two domains, and constant term. In practice, this means that the smaller the divergence term dRAN is, the smaller the gap between the adversarial and natural errors (e'z - <x) can be. We empirically tested this prediction using the domain discriminator trained on CIFARIO 10 dataset using the PGD-T attack. Figure Bp shows that the estimated dæAH using the domain discriminator (i.e., using the corresponding empirical value of a Eq. (6) trained on PGD Les with 0.031 is closely related to the adversarial-vs vestriirl--nnuuraal generalization gap over

---

0.9
0.6
1.4
linfpgd
11pgd
0.8
0.5
1.2
12pgd
fgsm
dHANDD (7,)
0.7
0.4
1.0
deepfoo!
0.8
CW
0.6
ddn
0.3 IN
W 0.6
0.5
0.2
0.4
0.4
0.1
0.2
0.3
0.01
0.02
0.03
0.06
0.13
0.0
0.2
0.4
0.6
0.8
1.0 1.2 1.4
Epsilon
duBAAD D2)
(b) Figure 3: (a) Estimated dHAH distance (based on empirical value of Eq. 6) and generalization gap in adversarial and natural error é'z scatter plot of the estimated dRaw distance using the domain discriminator and the gap adversarial and natural eπrоr across different attack types and magnitudes (i.e. e). Colors correspond to different attack types, Each dot corresponds to one attack evaluated a particular epsilon value. Estimations of the HHΛh distance for all attacks and epsilons are made with the domain discriminator trained on PGD- with 0.031.
(a)
as function of epsilon for PGD Loo attack; (b)
different e values predicted by Eq. 5 Moreover, estimations from the same domain discriminator also predicts the gap in generalization error attained for other forms of attacks (even ones not seen during AFD training) and values with high accuracy (Figure-3p). This further supports the proposal that minimizing the estimated distance between the natural and adversarial representations can be an efficient way to improve the model robustness against various adversarial attacks.
4.5 Learning a sparse representation
Because the AFD method aims learn representation that is insensitive adversarial attacks, expected the learned representational space to potentially of lower dimensionality (i.e. less number of orthogonal features). To test this, compared the dimensionality of the learned representation using two measures. i) number of non-zero features over the test set within each dataset and ii) number of Principal Component Analysis (PCA) dimensions that explains more than 99% of the variance the representation computed over the test-set of each dataset. We found that the same network architecture (i.e. ResNet18), when trained with AFD method learns much sparser and lower dimensional representational space (Table A4) compared to the naturally trained, adversarial training and TRADES models. The representational spaces learned with AFD on MNIST, CIFAR1O, and CIFARION 100 datasets had only 6, 9, and 76 principal components respectively.
4.6
Adversarial and norm-based desensitization
To investigate whether the same level of robustness could achieved by encouraging the network to produce similar representations in response to natural and adversarial inputs, we ran an additional experiment on the MNIST dataset in which we added a regularization term to the classification loss to directly minimize the representation sensitivity S, "E F(z'))i, [[. during training. We observed that although this augmented loss led to learning robustness representations, it achieved modest levels of robustness (~ 80%) and showed only weak generalization to stronger and other unseen attacks (FigureA.)). This result suggests that more direct forms of enforcing representational similarity may not lead to the same form of robustness with generalization properties similar that achieved using an adversarial training with domain discriminator (e.g. as in AFD).
Conclusion and limitations
Decreasing the input-sensitivity features has long been desired in training neural networks [15] and has been suggested as way improve adversarial robustness 44 61]. In this work proposed an

---

algorithm to decrease the sensitivity of neural network representations using an adversarial learning paradigm that involves joint training of domain discriminator, a feature extractor, and a task classifier. Essentially, our proposed algorithm iteratively estimates a bound on the adversarial enror in terms of the natural error and classification-bann and adversarial features and then minimizes the adversarial error by concurrently reducing the natural error as well as the distance between the two feature distributions.
measure of distance between the distributions of natural
Limitations. The empirical results presented here suggest that AFD-trained models are robust against wide range of adversarial attacks (distributions) even compared to strong baselines like Adversarial Training and TRADES. However, it is not guaranteed that the model would remain robust against any unseen attacks that we have not tested or may be invented in the future- as is the case in domain adaptation literature and the lack of theoretical guarantees for cross cross-domain generalization. With regards to the computational cost, when measuring the average per-epoch training time on the CIFAR 10 dataset (using 2 NVIDIA V100 GPUs), we found that the AFD training time is 31 % longer than adversarial training and only 4% longer than TRADES. This shows that while AFD requires three SGD updates per batch, the additional computational cost is not significantly higher than many prior methods when considering that most of the computational cost is associated with generating the adversarial examples during training.
6
Acknowledgements
We would like to thank Isabela Albuquerque, Joao Monteiro, and Alexia Jolicoeur-Martinge Martineau for their valuable comments on the manuscript. Pouya Bashivan was partially supported by the Unifying AI and Neuroscience Québec (UNIQUE) Postdoctoral fellowship and NSERC Discovery grant RGPIN-2021 -03005.. Irina Rish acknowledges the support from Canada CIFAR AI Chair Program and from the Canada Excellence Research Chairs Program.
References
[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. ICML, 1:298-322 00-,,,1,22 2017.
[2] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. 35th International Conference Machine Learning, ICML 2018, 1:436-448, 2018.
[3] Yang Bai, Yuyuan Zeng, Yong Jiang, Shu-Tao Xia, Xingjun Ma, and Yisen Wang. Improving Adversarial Robustness via Channel-wise Activation Suppressing. In ICLR, pages 2021.
[4] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 19(1-2):151-175, 2010.
[5] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems, pages 137-144, 2007.
[6] Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. Proceedings IEEE Symposium on Security and Privacy, pages 39-57, 2017.
[7] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, pages 11 191190111 1201, 2019.
[8] Alvin Chan, Yi Tay, and Yew-Soon Ong. What it thinks is important is important: Robustness transfers through input gradients. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 332-341, 2020.
[9] Alvin Chan, Yi Tay, Yew Soon Ong, and Jie Fu. Jacobian Adversarially Regularized Networks for Robustness. ICLR, 2020.
10

---

[10] Jeremy Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via ran- domized smoothing. 36th International Conference on Machine Learning. ICML 2019, 2019- June:2323-2356 2019.
[11] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fre free attacks. arXiv preprint arXi::2003.01690, 2020.
[12] Gavin Weiguang Ding, Luyu Wang, and Xiaomeng Jin. AdverTorch v0.1: An adversarial robustness toolbox based on pytorch. arXiv preprint arXiv:1902.07623, 2019.
[13] Yinpeng Dong. Zhijie Deng, Tianyu Pang, Hang Su, and Jun Zhu. Adversarial Distributional Training for Robust Deep Learning, In Neural Information Processing Systems (NIPS), number NeurIPS, pages 1-14, 2020.
[14] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting Adversarial Attacks with Momentum. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 9185-9193, 2018.
[15] Harris Drucker and Yann Le Cun. Improving generalization performance using double back- propagation. IEEE Transactions on Neural Networks, 3(6):991-99)) 1992.
[16] Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. arXiv preprint arX1vv2003.09553, 2020.
[17] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. 36th International Conference on Machine Learning, ICML 2019, Mo-Ju:n:n21-22222 2019.
[18] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. 32nd International Conference Machine Learning, ICML 2015, 20):1180-189, 2015.
[19] Ian Goodfellow, Dan Boneh, and Patrick Medaniel. Ensemble Adversarial Training: Attacks and Defenses. In ICLR, pages 1-20, 2018.
[20] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-26888 2014.
[21] Ian J Goodffllow,, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6577 2014.
[22] Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against com-boonded adversarial examples. arXiv, (2017):1-
30, 2020.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paitern recognition, pages 770-778, 2016.
[24] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, pages 1-16, 2019.
[25] Dan Hendrycks, Mantas Mazeika, Sauray Kadavath, and Dawn Song. Using pre-training can improve model robustness and uncertainty. NeurlPS, pages 4815-4826. 2019.
[26] Qiuyuan Huang, Paul Smolensky, Xiaodong He, Li Deng, and Dapeng Wu. Tensor product generation networks for deep NLP modeling. arXiv, pages 1263-1273, 2017.
[27] Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing, arXiv preprint 1833v:1803.06333, 2018.
[28] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.

---

[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[30] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227--2322 1998.
[31] Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint 180400000 2018.
[32] Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable Adversarial Training: A General Approach to Adapting Deep Classifiers. International Conference on Machine Learning, pages 4013-4022, 2019.
[33] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Noov: 2008.
[34] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0660 2017.
[35] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprini 15X 2015.
[36] Alexander Matyasko and Lap-Pui Chau. Improved network robustness with adversary critic. In Advances in Neural Information Processing Systems, pages 10578-10587, 2018.
[37] Takeru Miyato, Toshiki Kataoka, Koyama Masanori, and Yoshida Yuichi. Spectral normalization for generative adversarial networks. ICLR, 2018.
[38] Takeru Miyato and Masanori Koyama. egans with projection discriminator. arXiv preprint arXiv: 1802.05677 2018.
[39] Takeru Miyato, Shin Ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Sune Sumi-Super Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-16, 2018
[40] Seyed Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. DeepFool: Simple and Accurate Method to Fool Deep Neural Networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem:2574-2555 2016.
[41] Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhus- sein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial Robustness Through Local Linearization. (NeurIPS):1-10 2020.
[42] Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox benchmark the robustness of machine learning models. In Reliable Machine Learning in the Wild Workshop. 34th International Conference on Machine Learning. 2017.
[43] Jerome Rony, Luiz G. Hafemann, Luiz S. Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and norm for efficient gradient-based 12 adversarial attacks and defenses. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019 9019-June:4317-4335 2019.
[44] Andrew Slavin Ros and Finale Doshi- Velez. Improving the adversarial robustness and inter- pretability of deep neural networks by regularizing their input gradients. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pages 1660-1669, 2018.
[45] Evgenia Rusak, Lukas Schott, Roland S. Zimmermann. Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. Increasing the robustness of DNNs against image corruptions by playing the Game of Noise. 2020.
[46] Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversar- ially robust neural network model on mnist. arXiv preprint arXiv:1805.09) 2018
12

---

[47] Samrath Sinha. Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning.
Proceedings of the IEEE International Conference on Computer Vision, 00119-ctob::971-5900 2019.
[48] Chawin Sitawarin, Supriyo Chakraborty, and David Wagner. Improving adversarial robustness through progressive hardening. arXiv preprint arXiv:2003.093347, 2020.
[49] Chuanbiao Song, Kun He, Liwei Wang, and John Hoperoft. Improving the generalization of adversarial training with domain adaptation. In ICLR, pages 1-14, 2019.
[50]
Chuanbiao Song, He Kun, Lin Jiadong, John E Hopcroft, and Liwei Wang. Robust local features for improving the generalization of adversarial training. In ICLR, pages 1-12, 2020.
[51] Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, and R. Venkatesh Babu. Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses. Neural Information Processing Systems (NIPS), number NeurIPS, 2020.
[52] Christian Szegedy, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Joan Bruna, Rob Fergus, and Dumitru Erhan. Intriguing properties neural networks. pages 1-10, 2013.
[53] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. Proceedings 30th IEEE Conference on Computer Vision and Pattern Recognition, 2017.
CVPR 2017, 201 2917
[54] Huaxia Wang and Chun-Nam Yu. A direct approach to robust deep learning using ad adversarial networks. arXiv preprint arXiv: 2019.
[55] Dongxian Wu, Yisen Wang, and Xia Shu-Tao. Revisiting Loss Landscape for Adversarial Robustness. ICML, 2019.
[56] Runtian Zhai, Chen Dan, Di He, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsich, and Liwei Wang. Macer: Attack-free free and scalable robust training via maximizing certified radius. arXiv preprint arXiv:2001.02336 2020.
[57] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv: 1901008701 2019.
[58] Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards Stable and Efficient Training of Verifiably Robust Neural Networks. pages 1-25, 2019.
[59] Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighter adversarial training. In ICLR, pages 1-28, 2021.
[60] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference computer vision, pages 2223-2232, 2017.
[61] Sicheng Zhu, Xiao Zhang, and David Evans. Learning adversarially robust representations via worst-case mutual information maximization. ICML, 2020.
13


Reviews:
[{"review_id": "shKUwAqHiSl", "reviewer": "Reviewer_h6S6", "summary": "From the idea of domain-adaptation, the authors design a defence method to learn robust DNNs. The effectiveness is verified by numerical experiments. ", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The basic idea of this paper that is to minimize the divergence between the feature (representation) distributions of adversarial and natural domains is good. But although there is modification from the previous versions (submitted to ICLR 2021), I agree with those reviewers in some aspects. \n\n1. The explanation about the advantage over adversarial training is not very convincing in my point of view.  The upper bound on the adversarial error is not tight and I am not sure why minimizing it results in more attacks. Even in the case the claim is correct, it cannot explain why the proposed method is better for e.g., white-box and known attacks. \n\n2. In numerical experiments, the improvement is only significant for unseen attacks. But for Table 2, I have some worry about other methods, e.g., TRADES. In many literature, TRADES still can get good defence performance for unseen attacks. At least, the drop from using PGD with different norms was not so large. Another drawback is there is still significant accuracy drop from NT for natural images.  \n\n3. The motivation for domain-adaption is not very clear. In fact, in their previous version, domain-adaption is not their motivation. I hope the authors could tell the readers the real motivation/thought. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "no", "review_text": "", "extra_scores": {"ethical_concerns": "no ", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2 days"}, "extra_text": {}}, {"review_id": "ItNCOhu5PlH", "reviewer": "Reviewer_WEvD", "summary": "This paper introduced using domain adaptation for adversarial defence and following a similar setup as GAN. The proposed Adversarial Feature Desensitization (AFD) shows promising results on a wide range of datasets. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "---\nStrength:\n- Domain adaptation between clean and adversarial as two different domains is a technically sound idea.\n- Experiments considered a wide range of datasets that are commonly used to evaluated adversarial defence. The results are promising. Robustness against stronger shows strong results aginst larger $\\epsilon$ attacks than adversarial training (AT) models.\n- AFD learning a sparse representation is consistent with AT which also encouraged to learn such representation. \n\n---\nConcerns and Questions:\n- Evaluation against [AutoAttack](https://github.com/fra31/auto-attack) [11]. This paper claimed that this method could generalize to unseen attacks. But based on the evaluation, I do not see such a claim as valid. AutoAttack has evaluated several existing models and found several overestimated their results; for example, the feature-scattering (Zhang & Wang, 2019) uses an idea very similar to this submission. Though it is not guaranteed to be robust to future attacks, one should report the performance of existing strongest attacks to make such a claim.\n- What is the overhead of training in addition to generating the adversarial example?\n- Discussion on Adaptive Attack (Tramer et al., 2020), in white-box setting, if adversaries have access to the discriminator $D_\\psi$, could they generate adversarial examples that are not on the domain as a possible adaptive attack to circumvent the proposed method?\n\n\nZhang, Haichao, and Jianyu Wang. \"Defense against adversarial attacks using feature scattering-based adversarial training.\" NeuIPS (2019).\n\nTramer, Florian, et al. \"On adaptive attacks to adversarial example defenses.\" NeurIPS 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have adequately addressed the limitations and potential negative societal impact ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "AtFuc1hajfs", "reviewer": "Reviewer_xEbL", "summary": "The paper proposes Adversarial Feature Desensitization (AFD) to train classifiers robust to adversarial attacks. In particular, AFD trains jointly a feature extractor, a classifier for the original task and a discriminator which distinguishes between natural and adversarial inputs, in order to find features which are effective for classification but not sensitive to adversarial attacks. In the experimental evaluation on four datasets, AFD outperforms standard methods in most of the cases.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Using an approach based on domain adaptation to get robust features seems novel.\n\n- Four datasets are used for the experimental evaluations.\n\nWeaknesses\n- The main concern is about the experimental evaluations, since the attacks used seem not sufficient and there are hints suggesting the presence of some form of gradient obfuscation. In Figure 2, many attacks do not reduce the robustness of AFD significantly even for very large thresholds, e.g. against PGD with $\\epsilon_\\infty=0.125 \\approx 32/255$ AFD still has around 50% of robustness, which is around what one would expect for ResNet-18 at $\\epsilon_\\infty = 8/255$. Additionally, in the same setup on MNIST (Figure A1), AFD has high robustness at $\\epsilon_\\infty=0.5$, for which any starting image can be turned into uniform grey. This is a hint of gradient obfuscation [A].\nStronger methods like AutoAttack [B] (which supports different norms) and MultiTargeted Attack [C] could be used for evaluation.\n\nOverall, the experimental evaluation is not convincing.\n\n[A] https://arxiv.org/abs/1902.06705  \n[B] https://arxiv.org/abs/2003.01690  \n[C] https://arxiv.org/abs/1910.09338", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The broader impact is not discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "AUexzB1T2VZ", "reviewer": "Reviewer_SkVe", "summary": "This work propose a new formulation for training adversarially robust deep networks, which utilizes the knowledge from the domain-adaptation field. The proposed method of AFD is evaluated on benchmark datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) This paper is well-written. The writing is clear and straightforward. The experimental setup is aligned with other works in this field.\n2) This method seems novel to me. I have not read works that proposes a similar understandings or methods.\n\nHowever, I have four major questions:\n\n1) After reading the paper, I understand the training procedure of AFD, but I am not quite certain about the testing procedure. What exactly will happen during testing? A : Will the GAN part be eliminated during testing while the discriminator of F proceeding predictions? B : Or will the GAN part also participate in the predictions? If the GAN is kept (situation A), AFD clearly bring extra computation expanse, and the author should present the amount of the increased computation complexity. By the way, in algorithm, how exactly does AFD generate xi' from xi? with PGD?\n\n2) Why only present your results on the relatively small model of ResNet18. Also, in Table 1, considering both TRADES and AT present their results on the stronger WRN-34-10/WRN-28-10 but the authors of this paper did not present these results, I suspect the effectiveness of this method may come from the increased GAN part (if the answer of Q1 is situation A). BTW, the attacks evaluated in this work is somewhat out of date and weak. I hope to see evaluations on more powerful attacks like AutoAttack.\n\n3) The major benefit of AFD, as claimed by the authors for several times. is its capability to generalize to unseen attacks during testing. And they try to prove this theory with Table 2. If the theory is correct, a reasonable results should show that AFD exhibit similar performance against the attacks during training (seen attacks) with other methods like AT, and perform much better on unseen attacks. However, from Table 2, it seems that AFD perform better on all kinds of attacks, including the PGD attack used by training. This fact again make me suspect that the presented superiority of AFD may come from the extra computation of GAN. At least, I am not very convinced about the claim that \"AFD  can better generalize to unseen attacks\"\n\n4) In the paper, the author says:\n> Line 182 : In contrast to this setup, our proposed method minimizes a conservative upper bound on the adversarial error and therefore is more likely to generalize to a larger set of unseen attacks, and to stronger versions of previously seen attacks (i.e. ones that generate higher-loss samples in the inner optimization loop).\n\nI do not understand this argument. Please explain more clearly.\n\nI may lower my score if the author can not address all my four concerns.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to NeurIPS.  The reviewers and I are in agreement that there are some interesting aspects to the proposed approach.  However, I share the concerns echoed by the reviewers on the "too good" results as presented in the text.  Specifically, as highlighted e.g. by Reviewer xEbL, the presented results simply don't add up in most cases, showing accuracy that doesn't match up with what is possible for deep networks.  While the reviewers attempted to address these concerns, ultimately these weren't that convincing, and I am in agreement with the reviews that the work needs much more review before publication at NeurIPS.
Decision: Accept (Poster)