Adversarial Feature Desensitization
Authors: Pouya Bashivan
Mojtaba Faramarzi
Touraj Laleh
Blake Aaron Richards
Irina Rish
Venue:  (2021)
Keywords: adversarial robustness
adversarial learning
convolutional neural networks

Abstract: Deep neural networks can now perform many tasks that were once thought to be only feasible for humans. While reaching impressive performance under standard settings, such networks are known to be  susceptible to  adversarial attacks -- slight but carefully constructed perturbations of the inputs which drastically decrease the network performance. Here we propose a new way to improve the network robustness against adversarial attacks by   focusing on robust representation learning based on  adversarial training procedure, called here Adversarial Feature Desensitization (AFD). AFD desensitizes the representation via an adversarial game between the embedding network and an adversarial discriminator introduced on top of  the standard predictive model, which is trained to distinguish between the clean and perturbed inputs from their high-level representations. Our method substantially improves the state-of-the-art in robust classification on MNIST, CIFAR10, and CIFAR100 datasets. More importantly, we demonstrate that AFD has better generalization ability than previous methods, as the learned features maintain their robustness across a wide range of perturbations, including perturbations not seen during training. These results indicate that reducing feature sensitivity is a promising approach for ameliorating the problem of adversarial attacks in deep neural networks. 

Under review as a conference paper at ICLR 2021
ADVERSARIAL FEATURE DESENSITIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep neural networks can now perform many tasks that were once thought to be only feasible for humans. While reaching this impressive performance under standard settings, such networks are known to be vulnerable to adversarial attacks -lightt but carefully constructed perturbations of the inputs which drastically de- crease the network performance. Here we propose a new way to improve the network robustness against adversarial attacks by focusing on robust representation learning based on the adversarial learning paradigm, called here Adversarial Fea- ture Desensitization (AFD). AFD desensitizes the representation via an adversarial game between the embedding network and an additional ad adversarial discriminator, which is trained to distinguish between the clean and perturbed inputs from their high-level representations. Our method substantially improves the state-of-the-art in robust classification on MNIST, CIFARIO, 10, and CIFAR100 datasets. More im- portantly, we demonstrate that AFD has better generalization ability than previous methods, as the learned features maintain their robustness across a wide range of perturbations, including perturbations not seen during training. These results indicate that reducing feature sensitivity is a promising approach for ameliorating the problem of adversarial attacks in deep neural networks.
INTRODUCTION
Despite remarkable recent progress in deep learning that allowed neural networks to achieve near human- performance across range of complex tasks (He et al. 2016 Mnnh et al.) [015]] Silver al., [017 Vinyaas et al. 2019). number of important open challenges remain. For example, deep networks are know to be highly vulnerability to adversarial attacks (Szegedy et al. 2013). i.e. small but precise perturbations of the inputs that result in high-confidence predictions which are critically divergent from human judgement.
Many prior works on adversarial robustness have tackled the robust classification problem by forcing the classifier to output the correct label for the perturbed inputs (Madry al. 2017 Kannan et al. 2018 Zhang et al., 2019b). These approaches essentially push the representations of samples from different categories away from the decision boundary. For example, the Adversarial Training procedure (Madry et al. 2017). trains a network to minimize the classification loss on the distribution of perturbed input samples. Another recent approach (Zhang al. 2019b) augments the regular classification loss with an auxiliary term that encourages the network to match the assigned labels to clean and perturbed inputs (Figure [p]. More recently, several other works have tried to improve the classification robustness by enhancing the smoothness of the classification loss (Wu et al. 2019 Qin al. 2020), or the saliency of the Jacobian matrix (Chan et al. 2020b). These methods has been shown to further improve the robust performance compared to prior approaches that do not consider the gradient landscape of the network. However, despite all these efforts, most of these defenses remain vulnerable against other forms of attacks that were not used during training or even slightly stronger perturbations of the same kind (Schott et al., 2018 Sitawarin et al.) 2020).
One reason for the above could be an insufficient focus on the robustness of representations learned by the model. has been shown that many adversarial perturbations that are often small in magnitude lead to large deviations in the high-level features of deep neural networks (Liao et al., 2018 Yoon et al. 2019). In addition, previous work [yyas et al., 2019) demonstrated that adversarial patterns often rely on specific learned features which generalize even large datasets such as ImageNet (Deng et al., 2209). However, these features are highly sensitive to input changes, yielding a potential vulnerability that can be exploited by adversarial attacks. While humans can also experience altered

---

Under review as a conference paper at ICLR 2021
533
zeruzbe
Ee
Cless
Cles?
1wward kway
Do
Carmpory
Adversarial Training
TRADES
AFD
Crean
Perturbed
(a)
(b)
Figure 1: Overview of the proposed AFD approach: (a) visual comparison of several adversarial robustness methods (Adversarial training (Madry et al.] 2017). TRADES (Zhang et al. 2019b), and AFD). The dotted black line corresponds to the decision boundary of the adversarial discriminator; (b) schematic of the proposed AFD paradigm.
perception in response to particular visual patterns (e.g., visual illusions) they are seemingly insensitive to this particular class of perturbations, and often unaware of the subtle image changes resulting from adversarial attacks. This in turn suggests that current deep neural networks may rely on features that are still considerably different from those giving rise to perception in primates (and, particularly, in humans) even despite many recent studies highlighting their remarkable similarities (Yamins al., 2014, Khaligh-Razay & Kriegeskorte 2014 Bashivan al. 2019). It is therefore reasonable to hypothesize that a deep network may become more robust to such adversarial attacks if the corresponding higher-level representations are more robust input perturbations, similar to those used by our brains. One way to approach the issue of robust classification is to consider the classifier as a relatively simple mapping (e.g. a linear transformation) that produces predictions based on a learned representation. In this case, if the learned representation is robust then the predictions from the simple classifier would consequently be robust too (Garg et al. 2018 Zhu et al. 2020).
Here, instead of focusing on robust classification, we turned our attention to robustness of learned features from which the categories are inferred (e.g. using a simple linear classifier). Our goal is to learn representations that remain stable in the presence of adversarial attacks. We propose learn robust representations via an adversarial game between two agents: i) an attacker that searches for aefformmnncedegrrddiin perturbations given the embedding function and ii) discriminator function that distinguishes between the clean and perturbed inputs from their high-level representations. The parameters of the embedding and the adversarial discriminator functions are then tuned via an adversarial game between the two (Figure [p). This setup is similar to the adversarial learning paradigm widely used in image generation and transformation (Goodfellow et al. [011aa] Karras et al. 2019; Zhu et al. 2017), unsupervised and semi-supervisse
learning (Miyato et al. 2018b). video prediction (Mathieu al. 2015: Lee et al.) 2018). domain adaptation (Ganin & Lempitsky 2015
Tzeng et 2017), active learning (Sinha et al. 2019). and continual learning (Ebrahimi et al. 2020).
While some prior work have also considered adversarial learning to tackle the problem adversarial examples, they have often been used to learn the distribution of the adversarial images (Wang & Yu 2019 Matyasko & Chau 2018), or the input gradients( et al.) 2020b a).
The main contributions of this work are:
We propose novel method to learn adversarially robust representations through an adversar- ial game between the embedding function and an adversarial discriminator that distinguishes between the natural and perturbed representations.
We theoretically show that our proposed adversarial approach leads to flat loss function in the vicinity of the training samples, thereby making the overall representation more stable against adversarial attacks.
We perform extensive empirical evaluations against many prior art methods, on three datasets, eight types of attacks, with wide range of attack strength, and show that our proposed approach performs similar or better (often, significantly better) than most previous defense methods under most tested circumstances.
https://mi chae bbcch /e/ot/
2

---

Under review as a conference paper at ICLR 2021
2 METHODS
Let Ex(a): : X > H. where X C RR,, H S NR, be an embedding function (e.g. neural network with parameters 0) of the input E X into representation h E H. and let Des H + V. where y S RRN,, be linear decoding function, with parameters & (e.g., the last linear layer of neural network before applying softmax). The likelihood of each class i from a set of Ne classes, C = given the input æ; is computed as follows: (1(x) sof softmat tmaa (Co(Eg())) o E C. Let ((,,e) denote a perturbation function (an adversarial attack) which computes the perturbed input x' within the e-neighborhood of input æ:
Vx E X : T c) = x' E B(x, <); B(z, <) fa' E x ||
æ1 < <(),
such that argmax 1x(x)) A argmax 1xxx), i.e. the attack changes the class label of a sample x. iEC
iEC
It has been shown that adversarial examples are attributed to the presence of non-robust features which are predictive of the categories but are not shared with the human perception (Ilyas al. 2019). Naturally, reducing the sensitivity of the learned features could potentially enhance the network classification robustness against adversarial attacks. Given the perturbation vector 8 R.N; |1 S e. we could simply define the sensitivity of representation as an empirical average (over input samples) of the maximum norm change in the representation due to input perturbation (attack): S. = -. maxx [E(r) - E(x + 6)]l, and formulate the robust representation learning problem as an optimization problem which aims at minimizing the representation sensitivity S.. However, such an approach may negatively affect the empirical risk objective, i.e. the classification accuracy (as we will later see in the empirical section). Thus, we desire a more precise formulation which would be less disruptive to the classification objective of the network.
2.1 ADVERSARIAL FEATURE DESENSITIZATION
Instead of minimizing the empirical average of representation sensitivity across all samples in the dataset (as formulated in the previous section), we focus on minimizing the representation sensitivity at the level of distributions which we expect to be less disruptive to the classification objective. For this, we propose an adversarial learning procedure similar to Generative Adversarial Networks (GAN) (Goodfellow et al. 2014a), in which the generator network is replaced by an embedding network Eg that learns to map the clean and perturbed inputs into representations that are indistinguishable from each other. Similar to the original GAN setup, a discriminator network Da is trained to distinguish between representations of clean and perturbed inputs. The training procedure involves three loss functions that are optimized sequentially. First, parameters of the embedding function Fgg and decoder Deg are tuned to minimize the classification softmax entropy loss (on clean inputs). Second, parameters ti of the adversarial discriminator Da. are tuned to minimize the cross-entropy loss associated with discriminating natural and perturbed inputs conditioned on the natural labels. Lastly, parameters of the embedding function Eg are adversarially tuned to maximize the cross-entrops from the second step. Algorithm I summarizes the proposed approach (also, see Figure 1b). The adversarial training framework involves a two-player minimax game (Chrysos et al. 2019) between Eg and Daw, with value function ((Ea, Dag):
V(Eo, Day) = Pp [[pptl (D5(-ass(FF)) (Fo(z), mill] + Beycol [Eyestu (S( (Da (Fo(z), vi)]], (2)
where p and correspond to natural and perturbed distributions, and S denotes the softplus function. Chrysos al. 2211)) proves that the global minimum of the adversarial training criterion V VEE, Day) is achieved if and only if = qi in our setting, = (x), y) and q P(Eo(x'), y). i.c. achieving the global minimum in eq. 2 would imply that the representations of natural and perturbed images conditioned on the class label would belong to the same probability distribution. In that case, a Bayes optimal classifier would achieve the same error rate on the perturbed inputs as would on the natural inputs. We use this fact below to prove that, when ((E,, Du.)) is at its global minimum, the gradient of the likelihood function becomes equal to zero; ie. the adversarial attack will fail to change the class likelihoods.
Let Txx e) be a policy which computes the perturbed input x' within the neighborhood of input I: x(x,c) c) = x Dr = I' E B(x, <) where denotes the target (ground truth) class index; and S(Day) be a discriminator function H - {0, 1} that distinguishes between natural and perturbed representations; where S is the softplus function. The following theorem clarifies this property of our approach, which was not previously taken care of, at least not explicitly, by alternative methods.
81,
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1: AFD training procedure
Input:
Attack policy æ, mini-batch of size m, encoding network Eg. adversarial discriminator
network Da decoder network Dep, softplus function S, and learning rates o, 8, and 7.
Read mini-batch B {(x1, :/.):... (Fmm ym)) repeat
t- TTx c)
LEDe 1 Ein log (o0(ssottng softmaa(-Des S(-Day (Eo( 1:)) + (Du (x()) m)]
m
Da m Ei=1
LE 1 5m
S Es y:))
(0, Φ) (0, Ø)
V EDC
s + 10 VD
0- VoLE
until training converged;
If the udversarial optimization of embedding and discriminator converges to the global minimum ,0","") of the training objective in equation the true class (t) likelihood with respect to the input x is zero at any x E X. i.e. 01 = O. Bi
Theorem 1.
Eg and Du
2 then the gradient of
See appendix 6.2) for proof.
While the assumption of convergence to global optimum is a strong assumption, in practice it is possible derive bound on the classifier's 's robust error in terms of its error clean inputs and divergence measure between the clean and perturbed representations (see 6.4 in the appendix).
3 EXPERIMENTS
3.1 ADVERSARIAL ATTACKS
We used range of adversarial attacks in our experiments, using existing implementations in the Foolbox (Rauber al. 2017) and Advertorch (Ding et al. 2019) packages. We validated the models against different variations of the Projected Gradient Descent (PGD) (Madry et al.) 2017) (Trgo, L2 I.1), Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014b), Momentum Iterative Method (MIM) (Dong al. 2018 Decoupled Direction and Norm (DDN) (Rony et al. 2019), Deepfool (Moosavi-Dezfooll et al. 2016). and C&W (Carlini Wagner 2017) attacks. For each attack, we swept the value across wide range and validated different models each. Specific settings used for each perturbation are listed in TA2
3.2 ADVERSARIAL ROBUSTNESS
We validated our proposed approach on learning robust representations the MNIST (LeCun et al. 1998). CIFARIO. 10, and CIFAR100 (Krzhevskyy et al. 2009) datasets. We used the PGD-Log attack to perturb the inputs during training. respectively. We used the activations before the last linear layer as the high-level representations (H) of the network. In all experiments, the adversarial discriminator network (Da.)) consisted of three fully connected layers with Leaky ReLU nonlinearity followed by a projection discriminator layer that incorporated the labels into the adversarial discriminator through a dot product operation (Miyato) & Koyama| 2018). We compared several variations of the adversarial discriminator architecture and evaluated its effecc on robust classification on MNIST dataset (Table A6). Inereasing the depth of the adversarial discriminator and adding the projection discriminator layer drastically improved the robust classification accuracy. We verified that the adversarial discriminator could successfully discriminate between the clean and perturbed embeddings initially and that this performance was reduced during training (Figure A5). The number of hidden units in all layers of Dag were equal (64 for MNIST and 512 for CIFAR). We used spectral normalization (Miyato et al. 2018a) all layers of Day. Further details of training for each experiment are listed in Table A1 We used three separate learning rates for tuning the embedding Eg. adversarial discriminator Da and decoder Deg parameters. To find the best learning rates, we randomly split the CIFAR1O train set into a train and validation sets (45000 and 5000 images in train and validation sets respecively). We then carried
set to 0.3 and 0.031 for MNIST and CIFAR datasets
4

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison of robust accuracy against various attacks different datasets. For all 8 attacks we used = 0.3 and 255 for MNIST and indicates replicated results. NT: natural training; AT: adversarial training; AFD: adversarial feature desensitization; WB: white-box attack; BB: black-box attack where the adversarial examples were produced by running the attack on the NT ResNet18 model. Numbers reported with μ + d denote mean and std values over three independent runs with different random initialization. RSTTCar
datasets respectively.
et al. 2019) additionally uses 500K unlabeled images during training.
Method
Dalasel
Network
Clean
PGD,.. (WB)
FGSM (WB)
PGD:.. (BB)
FGSM (BB)
ATiMadry 2017
LoNet
96.0
TRADES Zianee
LeNet
99.48
96.07
ATES Blwwn d. A0A
LeNet
99:11
94.04
Defense-Ga
ABS semmn v vir 2orn
2018
MNIST
ComwNet
LeNet
99.20
99.0
13
34
93.23
RNI8
98.80+0011
0.0+0.0
2:00-0.07
11,25+3.60
20-9--291
Mrday al. TRADES Zhing et the 201
99.13+0.11
12
98.17=#0.18
RNI8
98.8420.118 18
80 50.69:9944 2444
91,75.22.7
97.1 97.11
86.4-88.22
AFD
RNIS
99.1510:11
96.131.0.5
97,70.10.34
98.3110.188
97.92_10.31
AT Madry 2017
RNIS
87.3
45.8
56.1
86.0
85.6
TRADES
RNI8
84.92
ATES RLFAT Soog e 2010
3RN-34-10
55.06
WRN-32-10
82.72
58.75
RST+ + Wu et al 2019 Co Ceell 2019
WRN-34-10
89.82
61.86
69.60
88.77
87.61
YOPO
LER Min
CIFARIO
WRN
RNI8
83.99
86.83
52,99
20X00
46.7
65.7
59.3
70.3
NET
RNI8
95.40
0.12
47.79
12.00
$4.65
AT Maday al. 2017
RNIS
83.58
41.05
50.12
83.20
82.3 88
TRADES
20199
RNIS
82.32
$2,30
58.16
80.36
79.69
Fenture
Zrang
WRN-28-10
90.00
70.5
78.4
89.38:2271
77.72:110.78
85,34:0.03
86.33:2.211
84.70=1.40
N MTyMady el al. 2017
RN18
76.12
0.01
9.67
1.55
15.43
RNIS
55.78
20.39
25. 25.09
53.83
53.25
TRADES
CIFARI00
RN18
55.18
27.36
30/16
$4,13
53.16
RLFAT
WRN-32-10
Fentue-acaterrigg Zising& Wang 2019
WRN-28-10
AFD
RN18
62.3515.70
44.88.18.30
49.52.16.73
63.63.1.122
54.90_...42
out a grid- using the validaaaiootion sets and picked the learning rates with highest validation performance. As baseline, we used re- re-implemennattion of adversarial training (AT) method (Madry al. (017)) and the official code for TRADES (Zhang al. 2019b) and denoted these results with in the tables.
CIFAR1O
CIFAR100
100
754
NT
Accuracy
50-
50-
AFD
25-
0.4
0.00
0.05
0.00
0.05
0.10
Figure 2: Robust accuracy for different strengths of PGD-Lxo attack on different datasets.
Adversarial robustness against the observed attack We first evaluated our approach against the same class and strength of attack that was used during training (PGD-Line with 0.3 and 0.031 for MNIST and CIFAR datasets respectively). Table compares the robust classification performance of our proposed approach against PGD-Loc (with similar setting as was used during training) and FGSM attacks. Training LeNet with AFD was unstable leading to frequent crashing of adversarial diseriminator accuracy despite our extensive hyperparameter search. For this reason, we conducted our MNIST experiments also using the ResNet18 architecture (He et al.] 2016). On all datasets, AFD-trained network performed much better than alternative methods against both white-box and black-box attacks. The relative improvement was largest on CIFAR 10 and CIFAR 100 datasets. We also observed a relatively high variance in robust accuracy of AFD-trained networks on CIFAR datasets when trained from different random initializations (standard deviation of 10.78 and 8.30 for CIFARIO 10 and CIFAR100) 100). We suspect this large variance to be due to the additional randomness in AFD training due to the adversarial game between the embedding and the adversarial discriminator networks. Across the three runs, the best trained models performed 83.72% and 54.95% against
5

---

Under review as a conference paper at ICLR 2021
Table 2: AUC measures for different perturbations and methods MNIST, CIFARIO, and CIFAR 100 datasets. AUC values are normalized to have maximum allowable value of 1. Evaluations on AT and TRADES were made on networks trained using reimplemented or official code.
Dataset
Model
PGD...
PGD.,
PGD..
FGSM
MIM
DDN
DeepFool
C&W
NT
0.16
0.12
0.14
0.29
0.16
0.18
0.23
0.75
AT
0.67
0.50
0.44
0.76
0.84
0.72
0.74
0.96
MNIST
TRADES
0.62
0.47
0.41
0.77
0.96
AFD
0.83
0.84
0.70
0.84
0.84
0.85
0.83
0.96
NT
0.04
0.06
0.35
0.06
0.13
CTFARIO
TRADES
AT
0.34
0.27
0.06
0.05
0.14
0.16
0.39
0.46
0.36
0.28
0.06
0.06
0.33
0.41
0.41
0.47
AFD
0.73
0.43
0.82
0.85
0.84
0.38
0.33
0.37
NT
0.03
0.02
0.03
0.09
0.02
0.05
0.03
0.10
CIFARI0O
AT
0.15
0.03
0.08
0.19
0.15
0.04
0.16
0.23
TRADES
0.19
0.04
0.09
0.23
0.20
0.04
0.19
0.26
AFD
0.43
0.19
0.47
0.50
0,49
0.15
0.19
the white-box PGD-Liaf attack (c 0.03) on CIFARIO and CIFARIOO respectively. Furthermore, AFD retained most of its robustness against large set of attacks while improving robustness against C& W and DeepFool attacks when using particular weaker attacks (e.g. PGD-Loo with e = 253 and iterations) during training (Figure-(A6) AA)) In addition, we also evaluated the AFD model on transfer black-box attacks from AT and TRADES models which further showed higher robustness to those attacks too (Table A4
Infood
Uppd
12ppd
fasm
Acceesyy
75-
50-
50-
50-
0-
0.00
0.05
0.10
0
30
100
1.00
200
5
30
3.5
0.00
0.55
0.10
300
Infmmi
ddh
deeptool
50-
50-
0.5
1.0
2.0
Ppolies
Pyolas
Figure 3: Comparison of robust accuracy of different methods against white-box attacks on CIFARIO
dataset with ResNet18 8 architecture.
Robust classification against stronger and unseen attacks We also validated the robustness clas- sification against higher degrees of the same attack used during training as well as suite of other attacks that were not observed during training. We found that, compared to alternative defense methods, the AFD-trained networks continued to perform well against white-box attacks even for very large perturbations while performance of other methods went down to zero relatively quickly (Figures A] The AFD-trained network also performed remarkably well against most other attacks that were not observed during training (8/8 on MNIST and 6/8 on CIFAR datasets). To compare different models considering both attack types and perturbation strength, we computed the crea-under-the-cure summarizes these values for our approach and two alternative approaches (adversarial training and TRADES). Our results showed that compared other baseline methods, AFD-trained networks are robust to wide range of attacks and strengths. As discussed in the Methods section, unlike most previous defense methods that focus on minimizing the robust classification error, AFD minimizes the representation sensitivity and consequently. the learned representation remains stable for large range of attack strengths compared to other methods (Figure left).
(AUC) for a range of epsilons for each attack and each approach. Table
Despite the large gain in robustness against most of the attacks, AFD-trained networks slightly under- performed against two of the attacks (Deepfool and C&W) when tested on CIFARIO and CIFAR 100 datasets. Our posthoc analyses showed that the direction of perturbations in the representational space in response to Deepfool and C&W attacks were more misaligned with the PGD- PGD-Laut attack compared to other attacks such as DDN which was comparatively less successful (Table A7). Moreover, it has been shown that most adversarial defenses are not guaranteed to transfer to unseen attacks (Maini et al. 2020 Pinot al. 2020) and that different adversarial training methods might even overfit to

---

Under review as a conference paper at ICLR 2021
TRADES
AFD
AT ATb
0/
04
0123456789
CIFARIDO CIFARIO
0.00
0:0b 0.00
Figure 4: (left) Comparison of normalized representation sensitivity test-set of MNIST (top), CIFARIO (middle), CIFARION (bottom) datasets under PGD-Loo attack. Plots show the median (Estd) sensitivity over test-set for each dataset. * denotes statistically significant difference between sensitivity distributions for AFD and TRADES. (right) Logarithm of the average gradient magnitudes of class likelihoods with respect to input, evaluated samples within the test-set of each dataset (4 (log (Eg-.x oz )). For each matrix, rows correspond to ground truth (target) labels and columns correspond to non-target labels.
the training set (Rice et al. 2020). While we did not observe any sign of overfitting for the PGD-Linf during training, the robustness against Deepfool and C& attacks decreased during the later stages of training and in a way, the network might have overfit to the PGD-Lin! attack during training (Figure A4).
Representation sensitivity We compared the robustness of the learned representation derived from training the same architecture using different methods. For that measured the normalized
sensitivity of the representations each network as [[(r)-((x')le For all three datasets we found [[ that the AFD-trained networks learn high-level representations that were more robust against input perturbations as measured by the normalized L2 distance between clean and perturbed representations
Gradient landscape To empirically validate the prediction from Theorem-1, we computed the average gradient of class likelihoods with respect to the input across samples within the test set of each dataset 44l,, i I.....N).. N.). We found that, on all datasets, the magnitude of gradients in the direction of most non-target classes were much smaller for AFD-trained network compared to other tested methods (Figure This empirically confirms that AFD stabilizes the representation in way that significantly reduces the gradients towards most non-target classes. Moreover, the output gradients of the AFD-trained network were highly salient and interpretable (Figure
Learning a sparse representation As we discussed in the Methods section, we expected the AFD method to find and remove the non-robust features from the learned representation. Thus, we expected the learned representational space to potentially be of lower dimensionality. To test this, we compared the dimensionality of the learned representation using two measures. i) number of non-zero features over the test set within each dataset and ii) number of PCA dimensions that explains more than 99% of the variance in the representation computed over the test-set of each dataset. We found that the same network architecture when trained with AFD method gave rise to much sparser and lower dimensional representational space (Table A5 The representational spaces learned with AFD on MNIST, CIFAR10, and CIFAR 100 datasets had only 6, 9, and 76 principal components respectively. Adversarial vs. 1.2 optimization We also ran an additional experiment on the MNIST dataset in which we added regularization term to the classification loss to directly minimize the representation sensitivity S, = 1 2., a [xx) E(x'))l [|, during training. We observed that although this augmented loss led to learning robustness representations, it only achieved modest levels of robustness (~ 80%) and showed only weak generalization to stronger and other unseen attacks (Figure A3 This result suggests that enforcing distributional form of feature desensitization (e.g. AFD) may lead robust behavior over larger range of perturbations compared to the case where feature stability is directly enforced through an L, norm measure.
7

---

Under review as a conference paper at ICLR 2021
Non-obfuscated gradients Recent literature have pointed out that many defense methods against adversarial perturbations could drive the network into a regime called obfuscated gradients in which the network appears to be robust against common iterative adversarial attacks but could easily be broken using -lxck-box or alternative attacks that do not rely on exact gradients (Papernot et al. 2017: Athalye al. 2018 Carlini et al., (2019). We believe that our results are not due to obfuscated gradients for several reasons. i) For most perturbations, the model performance continues to decrease with increased epsilon (Figures Al )2; ii) The iterative perturbations were consistently more successful than single-step ones (Table ; iii) Black-box attacks were significantly less successful than white-box attacks (T: iv) The AFD-trained model performed similar or better than alternate methods against the Boundary attack (Brendel et al., 2018 an attack which does not rely on the network gradients (Table-A.) ( In addition to these tests, we also evaluated the AFD performance on B&B (Brendel et al. (018)) and AutoAttack (Croce & Hein 2020). On these attacks, AFD was consistently better than or equal to the baseline models on MNIST and CIFARIO datasets but was less robust on the CIFAR 100 dataset A3))
4 RELATED WORK
There is an extensive literature on mitigating susceptibility to adversarial perturbations. Adversarial training (Madry et al. 2017) is one of the earliest successful attempts to improve robustness of the learned representations to potential perturbations
the input pattern by solving a "saddle point" problem composed of an inner and outer adversarial optimization. A number of other works suggest additional losses instead of direct training on the perturbed imputs. TRADES (Zhang al.) (019bb adds regularization term to the cross-entropy loss which penalizes the network for assigning different labels to natural images and their corresponding perturbed images. (Qin et al.) 2020) proposed an additional regularization term (local linearity regularizer) that encourages the classification loss to behave linearly around the training examples. (Wu et al. 2019) proposed to regularize the flatness of the loss to improve adversarial robustness.
Our work is closely related to the domain adaptation literature in which adversarial optimization has recently gained much attention (Ganin & Lempitsky/ 2015 Liu et al. [019 T Tzeng et al. (2017). From this viewpoint one could consider the clean and perturbed inputs as two distinct domains for which network aims to learn invariant feature set. Although in our setting, i) the perturbed domain continuously evolves while the parameters of the embedding network are tuned; ii) unlike the usual setting in domain-adaptation problems, here we have access to the labels associated with samples from the perturbed (target) domain. Despite this, (Song et al. 2019) regularized the network to have similar logit values in response to clean and perturbed inputs and showed that this additional term leads to better robust generalization to unseen perturbations. Related to this, Adversarial Logit Pairing (Kannan et al. 2018) increases robustness by directly matching the logits for clean and adversarial inputs. Another line of work is on developing certified defenses which consist of methods with provable bounds over which the network is certifted to operate robustly (Zhang et al., 2019c Zhai et al.) 2020 Cohen al. 2019). While these approaches provide sense of guarantee about the proposed defenses, they are usually prohibitively expensive to train, drastically reduce the performance of the network natural images, and the empirical robustness gained against standard attacks are low.
5 DISCUSSION
We proposed a method to decrease the sensitivity of learned representations in neural networks using adversarial optimization. Decreasing the input-sensitivity of features has long been desired in training neural networks (Drucker & Le Cun 1992) and has been suggested as a way to improve adversarial robustness (Ros & Doshi Velez, 2018 Zhu et al., 2020). Our results show that AFD can be used to effectively reduce the input- seniitvvnn of network features with minimal interference with the classification objective and to improve robustness against a family of adversarial attacks. Successful feature desensitization was dependent on having strong adversarial discriminator and maintaining a balance between the embedding and discriminator networks throughout training. With regards to the computational cost, while AFD requires three SGD updates per batch, the additional computational cost is not significantly higher than many prior methods when considering that most of the computational cost is associated with generating the adversarial examples during training.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. 35th International Conference on Machine Learning, ICML 2018, 1:436 448, 2018.
Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural Population Control via Deep Image Synthesis. Science, 364(6439), 2019. ISSN 1095-9203. doi: 10.32477/ccn.2018..222
Shai Senn-avid.. John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. theory of learning from different domains. Machine Learning, 79(1-2):151-11 2010. ISSN 15730565. doi: 10.1007/$ (0077//19944000-551
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision- based adversarial attacks: Reliable attacks against black-box machine learning models. ICLR, pp. 1-12, 2018.
Wieland Brendel, Jonas Rauber, Matthias Kümmerer, Ivan Ustyuzhaninov, and Matthias Bethge. Accurate, reliable and fast robustness evaluation. In Advances in Neural Information Processing Systems, pp. 1-287 2019.
Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. Proceedings IEEE Symposium on Security and Privacy, pp. 39-57, 2017. ISSN 10816011. doi: 10.1 10.1109/SP:20174999
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, lan Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint UrXiv:1902.0670) 2019.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy Liang. Unlabeled data improves adversarial robustness. In Advances Neural Information Processing Systems, pp. 11190-11200 1201, 2019.
Alvin Chan, Yi Tay, and Yew-Soon Ong. What it thinks is important is important: Robustness transfers through input gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 332-341, 2020a.
Alvin Chan, Yi Tay, Yew Soon Ong, and Jie Fu. Jacobian Adversarially Regularized Networks for Robustness. ICLR, 2020b.
Grigorios G Chrysos, Jean Kossaifi, and Stefanos Zafeiriou. Robust conditional generative adversarial networks. ICLR, pp. 1-27, 2019.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. 36th International Conference on Machine Learning, (CMI.2019, 2019-June:2323- 2356, 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. arXiv preprint -XXX::2003001690 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-258 Icee, 2009.
Gavin Weiguang Ding, Luyu Wang, and Xiaomeng Jin. AdverTorch v0.1: :: An adversarial robustness toolbox based on pytorch. arXiv preprint arXiv:190..07623 2019.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting Adversarial Attacks with Momentum. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 9185-9193, 2018. ISSN 10636919, doi: 10. 10.1109/CVP..201..000
Harris Drucker and Yann Le Cun. Improving generalization performance using double backpropaga- tion. IEEE Transactions Neural Networks, 3-6)99119997, 1992.

---

Under review as a conference paper at ICLR 2021
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning. arXiv preprint arXiv:2003.09553, 2020.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. 32nd International Conference on Machine Learning, ICML 2015, 2():1180-1189) 2015.
Shivam Garg, Vatsal Sharan, Brian Zhang, and Gregory Valiant. A spectral view of adversarially robust features. In Advances Neural Information Processing Systems, pp. 10138-10148, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farke Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-136, 2019.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing, arXiv preprint arXiv:1803.003373 2018
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Seyed Mahdi Khaligh-Razav and Nikolaus Kriegeskorte. Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation. PLoS Computational Biology, 10(11), 2014. ISSN 15537358. doi: 10. 10.1 pcb../.09....ii.00019911.1.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-233) 1998.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523, 2018.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang. Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1778-1787, 2018.
Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable Adversarial Training: A General Approach to Adapting Deep Classifiers. International Conference on Machine Learning, pp. 4013 4022, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):257--2605 2008.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 170.006083 2017.
Pratyush Maini, Eric Wong, and Zico Kolter. Adversarial robustness against the union of multiple perturbation models. ICML, 2020.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error, arXiv preprint 151 105440, 2015.
10

---

Under review as a conference paper at ICLR 2021
Alexander Matyasko and Lap-Pui Chau. Improved network robustness with adversary critic. In Advances in Neural Information Processing Systems. pp. 10578-10587, 2018.
Takeru Miyato and Masanori Koyama. egans with projection discriminator. arXiv preprint mrXiv::802.05637,, 2018.
Takeru Miyato, Toshiki Kataoka, Koyama Masanori, and Yoshida Yuichi. Spectral normalization for generative adversarial networks. ICLR, 8a.
Takeru Miyato, Shin Ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual Adversarial Training: Regularization Method for Supervised and Semi- ceni-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-16, 2018b. ISSN 19393539. doi: 10.1 109/
TPAMI.201 2,258821.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529553) 2015.
Seyed Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 10636919. doi: 10.1 11L.1////....1.......
2016. ISSN
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. ASIA CCS 2017 Proceedings of the 2017 ACM Asia Conference on Computer and Communications Security, pp. 506-519, 2017. doi: 10.1145/3052973.. 3.,5509.9
Rafael Pinot, Raphael Ettedgui, Geovani Rizk, Yann Chevaleyre, and Jamal Atif. Randomization matters. how to defend against strong adversarial attacks. JCML, 2020.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial Robustness Through Local Linearization. (NeurIPS):1-10, 2020.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox benchmark the robustness of machine learning models. In Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning, 2017.
Leslie Rice, Eric Wong, and J Zico Kolter. Overfitting in adversarially robust deep learning. ICML, 2020.
Jerome Rony, Luiz G. Hafemann, Luiz S. Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and norm for efficient gradient-based 12 adversarial attacks and defenses. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 4019-June:4311-4322 2019. ISSN 10636919. doi: P..100//CVP..2019.000
Andrew Slavin Ros and Finale Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp. 1660-1669, 2018.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gaa gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv: 1805. 06600, 2018.
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on mnist. arXiv preprint urXiv:1805.09199 2018.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-33 2017.
Samrath Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. Proceed- ings of the IEEE International Conference on Computer Vision, 00019-0ctob:5971--500 2019. ISSN 15505499. doi: 10. 0001000/0.22229900000.

---

Under review as a conference paper at ICLR 2021
Chawin Sitawarin, Supriyo Chakraborty, and David Wagner. Improving adversarial robustness through progressive hardeming. arXiv preprint arXiv:2003. 2020.
Chuanbiao Song, Kun He, Liwei Wang, and John E Hopcroft. Improving the generalization of adversarial training with domain adaptation. In ICLR, pp. 1-14, 2019.
Chuanbiao Song, He Kun, Lin Jiadong, John E Hoperoft, and Liwei Wang. Robust local features for improving the generalization of adversarial training. In ICLR, pp. 1-12, 2020.
Christian Szegedy, Joan Bruna, Dumitru Erhan. lan Goodfellow, Joan Bruna, Rob Fergus, and Dumitru Erhan. Intriguing properties of neural networks. pp. 1-10, 2013.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. Proceedings 30th IEEE Conference Computer Vision and Puttern Recognition, CVPR 2017, 2017 2017-January:2962 292.au22rryy22229979., 2017. doi: 10.1 10.1109/VVP.2201.....66
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(77822:350--34 2019.
Huaxia Wang and Chun-Nam Yu. A direct approach to robust deep learning using adversarial networks. arXiv preprint 1rXiv:1905.090 2019.
Dongxian Wu, Yisen Wang, and Xia Shu-Tao. Revisiting Loss Landscape for Adversarial Robustness. ICML, 2019.
Daniel I. K Yamins, Ha Hong, Charles Cadieu, Ethan A Solomon, Darren Seibert, and James hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences of the United States of America, 111(23);
DiCarlo. Performance-optimizn
8619-24, 2014. ISSN 1091-6490, doi: 10. 10.1007/pnas. 1aeis..10111111 12111.
Jihyeun Yoon, Kyungyul Kim, and Jongseong Jang. Propagated perturbation of adversarial attack for well-known CNNs: Empirical study and its explanation. Proceedings 2019 International Conference on Computer Vision Workshop, ICCVW 2019, pp. 4226 4234, 2019. doi: 10. 1199/ ICCVW.2019.00520.
Runtian Zhai, Chen Dan, Di He, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsich, and Liwei Wang. Macer: Attack-free and scalable robust training via maximizing certified radius. arXiv preprint arXiv:2001.02378. 2020.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You Only Propagate Once: Accelcrating Adversarial Training via Maximal Principle. (NeurIPS 2019), 2019a.
Haichao Zhang and Jianyu Wang. Defense against adversarial attacks using feature scattering-bas adversarial training. In Advances Neural Information Processing Systems, pp. 1831-1841, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv: /00i:/900.000 090730 2019b.
Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards Stable and Efficient Training of Verifiably Robust Neural Networks. pp. 1-25, 2019c.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
Sicheng Zhu, Xiao Zhang, and David Evans. Learning adversarially robust representations via worst-case mutual information maximization. ICML, 2020.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
6.1 NETWORK ARCHITECTURES
For all experiments, we trained the ResNet18 architecture (He al. 2016) using SGD optimizer with 0.9 momentum and learning rates as indicated in A1 weight decay of 10 batch size of 128. All learning rates were reduced by factor of 10 after scheduled epochs.
Table A1: Training hyperparameters for each dataset and network.
Dataset
Model
LRg
LRDe
LRgpe
weight decay
batch
Num. Epochs Scheduled Epochs
MNIST
100
[50, 80]
CIFAR-10
ResNet18
10
128
900
[400, 800]
CIFAR-100
900
[400. 800]
6.2 PROOF OF THEOREM 1
Theorem 1. If the adversarial optimization of embedding and discriminalor functions, Eg and Da converges to the global minimum (0","") of the training objective in equation 2 then the gradient of the true class (t) likelihood with respect to the input x is zero at any E X, i.e. 84, O.
Proof. Assume De, i E set of differentiable functions that implement the Bayes optimal classifier from the E(x) representation (note that we will drop the subscripts in Eg and Da notation for simplicity), i.e.
i = argmax,,4 and 4= P() soflmar (c(E(x)))))))))))))) EC
(3)
Assuming that for the perturbed inputs x' - (xre) the adversarial training of E and Da converges to its global minimum, from Proposition of (Chrysos et al. 2019) we have:
Vo E K.yEY. P(E(s), v) P(E(T(H, v).
Following from Bayes rule we have:
P(y = = = P(u. 1t 5)) - 8)), 5 Ø4,
(5)
dr
From equation 4 the marginal distributions P(E(x)) and P(E(r - 6)) should be equal which leads to:
P(u = (()))) = P(y = 1 F(r - 8)),
(6)
which can only be true if a4 dz
0.
L
6.3 ADVERSARIAL ATTACKS
We used a range of adversarial attacks our experiments. Hyperparameters associated with each attack are listed in the table below. Implementation of these attacks were adopted from Foolbox (Rauber et al. 2017), AdverTorch (Ding et al. 2019) packages.
6.4 BOUND ON CLASSIFIER'S ' ROBUST ERROR
Considering the representation distributions in response to clean and perturbed inputs (of a particular class) two distinct domains of inputs, is straight forward to the math from domain adaptation literature to derive bound on the classifier's robust error (i.c. under the perturbed scenario). this case, directly adapt Theorem 2 in (Bvid et al. 2010) to derive this bound.
De and Dp are distributions of representations in response to clean and perturbed inputs a particular class y respectively. Let U. and Up be samples of size each, drawn from D. and Dp- Let H. hypothesis space VC dimension d, then for any 8 e (0, 1), with probability at least 1-8 (over the choice of the samples), for every h H:
13

---

Under review as a conference paper at ICLR 2021
Table A2: Attack hyperparameters for each dataset and attack.
Attack
Dataset
Steps
More
Toolbox
MNIST
[0, 0.35
0.4, 0.45,
0.5]
FGSM
Foolbox
CIFAR
[0.
2
1
16
32
64
255
255
255
255
255
255
MNIST
PGD-L1
50
[[O, 10, 50, 100, 200]]
step=0.025
Foolbox
CIFAR
MNIST
PGD-L2
50
[0. 2, 5, 10]
step=0.025
Foolbox
CIFAR
MNIST
40
[0, 0.1,1. 0.3, 0.35, 0.4, 0.45, 0.5]
step=0.033
PGD-Loo
Foolbox
CIFAR
20
2 [0.
16
32
step=
255
255
255
MNIST
[0. 0. 0.3, 0.5, 0.8,
MIM
40
AdverTorch
CIFAR
10,
2
16
32
255
255
255
255
MNIST
[0, 1, 2, 5]
DDN
100
Foolbox
CIFAR
[0, 2, 5, 10, 15)
MNIST
[0, 0.1, 0.3, 0.35, 0.4, 0.45, 0.5]
Deepfool
50
Foolbox
CIFAR
[0. 255 255
2
16
32
64
255
255
235
255
MNIST
C&W
100
[0, 0.5, 1, 1.5, 21
stepsize=0.05
Foolbox
CIFAR
2dlog(2m) + log(3) (3)
(h) (h) + (H::((4c +AV
m
where & and Sp are the errors clean and perturbed inputs, dHAM is the empirical H-divergence (Ben-David et al.) 2010), and is the the combined error the ideal hypothesis h": Eo(h") + (h²).
Table A3: Comparison of robust accuracy against Auto/ttack (Croce & Hein 2020). Boundary attack
(Brendel et al. 2018) with 5000 steps and the robust performance of each model 100 random samples from each dataset's test-set.
2, and B&B attack (Brendel et al. 2019). We tested
Dataset
Model
Method
AutoAttack
Boundary Attack
B&B
NT
25
3
AT TRADES
88
63
92
MNIST
RN18
48
17
AFD
92
78
96
NT
0.0
0
0
AT
34
51
36
CIFARIO
RN18
TRADES
49
58
54
AFD
25
68
41
NT
0
2
0
AT
24
32
27
CIFAR100
RN18
TRADES
30
35
30
AFD
15
32
10
14

---

Under review as a conference paper at ICLR 2021
Table A4: Transfer black-box attack from ResNet18 network trained with adversarially training (AT) and TRADES on different datasets.
Dataset
Method
AT Transfer
TRADES Transfer
NT
73.46
62.09
AT
97.11
97.23
MNIST
TRADES
93.58
AFD
97.48
97.63
NT
94.11
76.09
AT
82.32
62.54
CIFARIO
TRADES
80.78
AFD
88.56
65.13
NT
56.84
51.95
AT
36.6
CIFARI00
TRADES
40.1
AFD
42.72
40.29
Table A5: Dimensionality of the learned representation space on various datasets using different methods and measures. Units: number of non-zero feature dimensions over the test-set within each dataset. Dims: number of PCA dimensions that account for 99% of the variance across all images within the test-set of each dataset.
Dataset
MNIST
CIFAR10
CIFAR100
RNI8
RN18
RNI8
Network
Units
Dims
Units
Dims
Units
Dims
NT
64
24
512
97
512
429
AT
64
43
512
455
512
481
TRADES
64
40
512
349
512
461
AFD
18
6
417
9
500
76
Table A6:
Comparison of robust accuracy against PGD-Lxc with 0.3 using different architectures for the adversarial discriminator, tested on MNIST dataset.
Dataset
Model
Da Architecture
Robust Acc.
FC1-PD
85,96
MNIST
RN18
FC3
90.73
FC3-PD
97.03
Table A7: Comparison of representation perturbations in response to different attacks. We computed the cosine angle between representation perturbations due to each attack to those from PGD-LLnf- Values are reported in radians.
Dataset
Model
Attack
Median Angle (rad)
DDN C&W
0.25
MNIST
RN18
0.40
Deepfool
0.38
DDN
1.03
CIFAR10
RN18
C&W
1.13
Deepfool
1.12
DDN
1.15
CIFAR100
RN18
C&W
1.34
Deepfool
1.35
15

---

Under review as a conference paper at ICLR 2021
100
100
102-
100
Arcunay
mures
50-
52-
53-
50
100
150
201
0.0
2.5
5.0
75
15.0
0.
02
2.4
+ps
Linfmim
100
1.00
100
Accuracy
50-
Arcunnry
50-
50-
02-
0.5
0.2
0.
0.5
1.0
155
Figure A1: Comparison of robust accuracy of different methods against white-box attacks on MNIST dataset with R8 architecture.
lnffpdd
12pgd
75-
75-
75-1
Acourcy
Accuracy
50-
Accuracy
Accurazy
23-
25-
01
0.50
0.25
0.10
50
100
150
200
10
20
0.00
0.55
0.30
aps
ops
linfmim
decofool
75-
75-
Accurecy
Accuracy
52-
Accunccy
25
25-
0.05
Figure A2: Comparison of robust accuracy of different methods against white-box attacks on CIFAR 100 dataset with ResNet18 architecture.
Infood
100-
deepfoo
102-
100-4
Sensitivit--2
Accum.y
Accuracy
AFD
2
50-
20
120
170
202
0.0
64
2.0
0.0
0.2
Figure A3; Comparison of robust accuracy of AFD and representation matching against white-box attacks on MNIST dataset with ResNet18 8 architecture.
16

---

Under review as a conference paper at ICLR 2021
linfpgd
deepfool
chkpt- 300
75
chkpt-
CIFARIO
Accuracy
Accuracy
75
chipt- 500
50
Accuracy
<hkpt-
chkot 900
25
25+
0.10
0.00
0.05
0.10
eps
eps
0p5
linfpgd
deepfool
60
CW
chkpt-300
CIFAR100
Accuracy
Accuracy
60
c
chipt= 400
40-
40
Accuracy
40
<hkpt-600
chkpt 900
20-
20
0-
0.00
0.05
0,10
0.00
0.05
0.10
ops
Figure A4: Robust accuracy against various attacks different training stages. First and second correspond to models trained on CIFAR10 and CIFAR 100 respectively.
MNIST
CIFAR10
CIFAR100
1.04
0.8
0.6
Clean Adversarial
Acc. (%6)
0.6
0.6
Acc.
Acc
0.4
0.5
0.4
0.2
0.4
0.2
0.3
20
50
60
90
300
400
500
600
750
899
300
400
500
600
750
899
Epoch Num.
Epoch Num.
Epoch Num.
Figure A5: Classification accuracy of the adversarial discriminator Da different training stages on different datasets.
linfpgd
deepfool
Accuracy
80
Accuracy
Accuracy
75
4/255, 10it
50
4/255,
50-
8/255, 10it
25-
25
8/255,
40-
0-
8/255, 20it
0.00
0.05
0.10
0.0
0.1
0.2
Figure A6: Robust accuracy of AFD-trained models CIFAR10 dataset against various attacks when using different levels of attack strength during training.
17

---

Under review as a conference paper at ICLR 2021
Same
Different
deer
frog
horse deer
dog
frog
horse
truck deer
ship
ship
ship
bird
frog
bird
bird
frog
car
truck
frog
Figure A7: Feature visualization. Pixel values were changed in the direction the gradients that would maximize either the ground truth class (left column) or a randomly selected class (right column). For each image, the original image (left), transformed image (middle) and gradient map (right) are shown.
18

---

Under review as a conference paper at ICLR 2021
n
Figure A8: Scatter plot of 2-dimensional t-SNE projection (Maaten & Hinton 2008 of the rep- resentations derived from training the ResNet18 architecture MNIST dataset. (top row) t-SNE projection representations of clean images for networks trained with different methods. Each point corresponds to the representation of of the images from the MNIST test-set. (rows 2 to 5) t-SNE projection of the representation of the clean and perturbed MNIST test-set images. Columns are sorted from left right with the strength of the perturbation (left-most column corresponds clean images and right-most column with highest tested perturbation). Perturbations are generated using PGD-L attack. NT: naturally trained: AT: adversarially (raiyy al., 2017); TRADES: (Zhang et al. 2019b); AFD: adversarial feature desensitization.
19

---

Under review as a conference paper at ICLR 2021
Figure A9: Scatter plot of 2-dimensional t-SNE projection (Maaten & Hinton 2008 of the repre- sentations derived from training the ResNet5 architecture on CIFARIO dataset. (top row) t-SNE projection of representations of clean images for networks trained with different methods. Each point corresponds to the embedding one the images from the CIFARIO test-set. (rows 2 to 5) t-SNE projection of the embedding of the clean and perturbed CIFARIO 10 test-set images. Columns are sorted from left to right with the strength of the perturbation (left-most column corresponds to clean images and right-most column with highest tested perturbation). NT: naturally trained; AT: adversarially trained trained[Madry Madry et al. 2017); TRADES: (Zhang et al. AFF:: adversarial feature desensitization.
20

---

Under review as a conference paper at ICLR 2021
Figure A10: Scatter plot of 2-dimensional t-SNE projection (Maaten & Hinton 2008) of the rep- resentation derived from training the ResNet5 architecture on CIFAR 100 dataset. (top row) t-SNE projection of representations of clean images for networks trained with different methods. Each point corresponds to the representation of one of the images from the CIFAR 100 test-set. (rows to 5) t-SNE projection of the representation of the clean and perturbed CIFAR 100 test-set images. Columns are sorted from left to right with the strength of the perturbation (left-most column corresponds to clean images and right-most column with highest tested perturbation). NT: naturally trained; AT: adversarially trained (Madry et al. 2017); TRADES (Zhang et al.) 2019b); AFD: adversarial feature desensitization.
21


Reviews:
[{"review_id": "gzin1KIZA8V", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is about leveraging the GAN idea to get robust features that are insensitive to adversarial attacks. Specifically, with a shared encoder/embedding, the feature of the adversarial attack are considered as the fake sample, while the feature from the real image is the real sample; by forming an adversarial game in the feature space and by jointly considering the classification loss, the AFD method is proposed. \n\nThe main idea is reasonable. But the current manuscript may not be ready for publication.\n\nThe structure of the current manuscript should be revised substantially. For example, too many materials (like experimental results) are given in Appendix, which makes the main manuscript less self-contained; I would suggest rearranging the contents and put the important ones in the main manuscript. \n\nFigures and their captions should be revised, especially Figures 3 and 4.\n\n\nIn the last paragraph of Page 2, the authors stated that “the proposed method outperforms existing methods by a large margin. However, this might not be true, by considering the performance shown in Tables 1 and 2 and the high variance observed in Table 1 and Figure 4.\n\nIn the 5th line of the first paragraph of Section 2, the definition of l_i (x) is different from the one in Algorithm 1. \n\nIn the last paragraph of Page 3, the definition of pi(x, epsilon) is not consistent. Epsilon is missing.\n\nIn the last paragraph of Page 3, the output of Da_psi should be real and the ideal discriminator output is {0,1}.\n\nIn Algorithm 1, why the policy pi is related to Dc_phi.  \n\nTheorem 1 might not be right. A GAN matches two distributions. Besides, if E() is a linear function, delta could be a vector in the null space of the weight.\n\nMany typos exist. For example, PGD-L_{inf} in Page 5.\n\nIn the last paragraph of Page 7, the representation sensitivity Sc is not defined.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_fo4PIapVIv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to leverage an additional adversarial discriminator to distinguish between the clean and perturbed inputs from the representation level.\n\nThe empirical evaluation shows promising results in terms of generalizing to unforeseen attacks. \nHowever, the baselines TRADES is evaluated on ImageNet which is a more challenging task and it would be good to evaluate the performance on large scale dataset for the proposed method as well.\nIt would be important to compare with related work on improving robustness by learning robust representation [1][2].\nIn addition, from figure 4, the sensitivity comparison is actually not very clear and it would be good to compute the significance level to show how sensitive the AFD learned representation with quantitative results.\n\nFrom the methodology perspective, it would be good to analyze the difference between the distributions of benign and adversary representation. Several previous work shows that with only activation patterns or logits it is not enough to distinguish the benign and adversarial instances. It would be important to evaluate and confirm that the learned representation of the benign and adversarial instances can indeed be learned and separated by a trained discriminator, which may lead to further interesting analysis and findings.\n\n\n[1] Liao, Fangzhou, et al. \"Defense against adversarial attacks using high-level representation guided denoiser.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n[2] Samangouei, Pouya, Maya Kabkab, and Rama Chellappa. \"Defense-gan: Protecting classifiers against adversarial attacks using generative models.\" ICLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1oi1FDz7HoE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an adversarial defense method that builds a robust classifier by using a min-max optimization in the feature extractor. \n\nPros:\n1. The proposed idea is interesting and reasonable.\n2. The experiment results look good.\n\nCons:\n1. The proposed idea is not novelty enough. There is a similar work [1] talking about the feature scattering to make the model robustness.  The only difference is the paper further uses a discriminator which involves a different network in the training process.\n2. The results might not be reliable. while [1] uses a similar idea, it later shows a significant performance drop by using some later proposed adversarial attack method like Autoattack[2]. Specifically, it reduces the performance to from claimed 60.6to 36.64. I suggest the author should try more blackbox based attack or the ensemble attack in Autoattack. \n3. The paper quality needs to be improved. For example, Figures 3  doesn't have x-axis label. I personally find it very confusing on the different x scale used in the figure. \n4. The methods mentioned in non-obfuscated gradients are not sufficient for me to believe there is no such case. (ii) (iii) is quite unrelated to the obfuscated gradients problem. Also, the boundary attack in the appendix looks very confusing. I can't understand what the number represents in the table. By the way, the boundary attack is not a good way to attack the adversarial defended method. Attacks like B&B or FAB are more suitable.\n\n\n\n\n\n[1] Zhang, Haichao, and Jianyu Wang. \"Defense against adversarial attacks using feature scattering-based adversarial training.\" Advances in Neural Information Processing Systems. 2019.\n\n[2] https://github.com/fra31/auto-attack", "extra_scores": {}, "extra_text": {}}, {"review_id": "15s1UaZWMwZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes Adversarial Feature Desensitization (AFD) as a defense against adversarial examples. AFD employs a min-max adversarial learning framework where the classifier learns to encode features of both clean and adversarial images as the same distribution, thereby desensitizing adversarial features. With the aim of fooling a separate discriminator model into categorizing the classifier’s adversarial features as from clean images, the classifier is trained with the standard cross-entropy loss and adversarial loss terms. The authors showed through experiments on MNIST, CIFAR10 and CIFAR100 datasets that AFD mostly outperform previous defenses across different adversarial attacks under white- and black-box conditions.\n\nPros:\n+Strong defense performance\n+Novel idea\n\nCons:\n-No discussion on the scalability of AFD defense or results on larger dataset such as imagenet\n\nRecommendation:\nThe idea is interesting and is backed by strong empirical results. Generally, the paper is well-written and easy to follow. Given AFD employs both a GAN training component and adversarial example generation, I can imagine it to be computationally more expensive than most of the existing defenses. It would be more convincing to show experiments and discussions addressing AFD’s scalability. Apart from this point, there is no major flaw in the paper and I am inclined towards acceptance.\n\n\nOther questions and comments:\nFor the black-box attack assessment, are the adversarial examples generated from models that are trained on their respective defense, initialized with different random seeds? It would be good to mention how exactly the black-box attacks are conducted.\n \nSome prior defenses have been shown to be prone to transfer black-box attacks, e.g. adversarial examples from a TRADES model, transferred to the AFD defense. It would be more convincing that AFD is not relying on obfuscated gradients with results on this form of black-box attacks.\n \nHow much is AFD’s performance dependent on the strength of adversarial examples using during training? How many iterations were used for the L-inf attack policy used to perturb the training inputs?\n \nI believe the sentence in the Introduction is untrue: “adversarial learning has not yet been successfully applied to the problem of adversarial robustness.” Several studies using adversarial learning for adversarial robustness exist:\n\n“Improved Network Robustness with Adversary Critic” NeurIPS 2018\n\n\"A Direct Approach to Robust Deep Learning Using Adversarial Networks\" ICLR2019\n\n“What it Thinks is Important is Important: Robustness Transfers through Input Gradients” CVPR 2020”\n\n“GanDef: A GAN based Adversarial Training Defense for Neural Network Classifier” arXiv:1903.02585\n\n--Update after rebuttal--\n\nThe reviewer thanks the authors for addressing the key questions and concerns and have updated the confidence score accordingly.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Adversarial Feature Desensitization (AFD) as a defense against adversarial examples. Specifically, following the spirit of GAN and Adversarial Domain Adaptation, an adversarial discriminator is introduced to distinguish clean and perturbed inputs at the representational level. 

This paper receives 3 reject and 1 accept recommendations. On one hand, though the proposed method shares some similarity with the Feature Scattering method at a high level, most of the reviewers still find the proposed method is interesting. The AC also agrees that the paper's organization and typos does not warrant a rejection. 

On the other hand, the reviewers have also raised a few concerns. (i) A more careful discussion on the scalability of the proposed method is needed. (ii) Experiments are mostly focused on small datasets, while results on ImageNet is lacking, which makes the paper less convincing. The authors claim that they are trying to at least run Tiny-ImageNet experiments; however, this set of results are not provided by the end. (iii) A more detailed analysis and visualization on the learned difference between the distributions of benign and adversary representation is needed, since a discriminator is learned here. 

The rebuttal unfortunately did not fully address the reviewers' main concerns. On balance, the AC regrets that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere. 
Decision: Reject