Jacobian Adversarially Regularized Networks for Robustness
Authors: Alvin Chan
Yi Tay
Yew Soon Ong
Jie Fu
Venue:  (2020)
Keywords: adversarial examples
robust machine learning
deep learning

Abstract: Adversarial examples are crafted with imperceptible perturbations with the intent to fool neural networks. Against such attacks, adversarial training and its variants stand as the strongest defense to date. Previous studies have pointed out that robust models that have undergone adversarial training tend to produce more salient and interpretable Jacobian matrices than their non-robust counterparts. A natural question is whether a model trained with an objective to produce salient Jacobian can result in better robustness. This paper answers this question with affirmative empirical results. We propose Jacobian Adversarially Regularized Networks (JARN) as a method to optimize the saliency of a classifier's Jacobian by adversarially regularizing the model's Jacobian to resemble natural training images. Image classifiers trained with JARN show improved robust accuracy compared to standard models on the MNIST, SVHN and CIFAR-10 datasets, uncovering a new angle to boost robustness without using adversarial training.

Published as a conference paper at ICLR 2020
JACOBIAN ADVERSARIALLY REGULARIZED NETWORKS FOR ROBUSTNESS
Alvin Chan'; Yi Tay', Yew-Soon Ong'. Jie Fu²
Nanyang Technological University, ?Mila, Polytechnique Montreal
ABSTRACT
Adversarial examples are crafted with imperceptible perturbations with the intent fool neural networks. Against such attacks, adversarial training and its variants stand as the strongest defense to date. Previous studies have pointed out that ro- bust models that have undergone adversarial training tend to produce more salient and interpretable Jacobian matrices than their non-robust counterparts. A natural question is whether a model trained with an objective to produce salient Jacobian can result in better robustness. This paper answers this question with affirma- tive empirical results. We propose Jacobian Adversarially Regularized Networks (JARN) method to optimize the saliency of a classifier's Jacobian by adver- sarially regularizing the model's Jacobian to resemble natural training images]] Image classifiers trained with JARN show improved robust accuracy compared standard models on the MNIST, SVHN and CIFAR-10 datasets, uncovering a new angle to boost robustness without using adversarial training examples.
INTRODUCTION
Deep learning models have shown impressive performance in a myriad of classification tasks (Le- Cun et al.) 2015). Despite their success, deep neural image classifiers are found be easily fooled by visually imperceptible adversarial perturbations (Szegedy et al. 2013).. These perturbations can be crafted reduce accuracy during test time or veer predictions towards a target class. This vulnera- bility not only poses a security risk in using neural networks in critical applications like autonomous driving (Bojarski et al. 2016) but also presents an interesting research problem about how these models work.
Many adversarial attacks have come into the scene (Carlini & Wagner 2017 Papernot et 2018 Croce & Hein 2019), not without defenses proposed to counter them (Gowal et al.) [018]] Zhang jct al. 2019). Among them, the best defenses are based on adversarial training (AT) where models trained on adversarial examples to better classify adversarial examples during test time (Madry
ct al. 2017). While several effective defenses that employ adversarial examples have emerged (Qin
al. 2019 Shafahi et al. 2019), generating strong adversarial training examples adds non-trivial computational burden on the training process (Kannan et al. 2018: Xie et al. (019)..
Adversarially trained models gain robustness and are also observed to produce more salient Jaco- bian matrices (Jacobians) at the input layer as a side effect (Tsipras et al.) 2018). These Jacobians visually resemble their corresponding images for robust models but look much noisier for standard non-robust models. It is shown in theory that the saliency in Jacobian is result of robustness (Et- mann et al. 2019). A natural question to ask is this: can an improvement in Jacobian saliency induce robustness in models? In other words, could this side effect be a new avenue to boost model robustness? To the best of our knowledge, this paper is the first show affirmative findings for this question.
To enhance the saliency of Jacobians, we draw inspirations from neural generative networks (Choi et al. 20188] Dai & Wipi 2019). More specifically, in generative adversarial networks (GANs) [Goodfellow et al., 20014). a generator network learns to generate natural-looking images with a training objective to fool discriminator network. In our proposed approach, Jacobian Adversarially
"Corresponding author: guowela
Sourre code available https: / //g /////tt/ thub. com/a a vi nchhngg/JJAN ICIR2200

---

Published as a conference paper at ICLR 2020
Regularized Networks (JARN), the classifier learns to produce salient Jacobians with regulariza- tion objective to fool discriminator network into classifying them as input images. This method offers new way look improving robustness without relying adversarial examples during training. With JARN, we show that directly training for salient Jacobians can advance model ro- bustness against adversarial examples in the MNIST, SVHN and CIFAR-10 image dataset. When augmented with adversarial training, JARN can provide additive robustness to models thus attaining competitive results. All in all, the prime contributions of this paper are follows:
We show that directly improving the saliency of classifiers' input Jacobian matrices increase its adversarial robustness.
To achieve this, we propose Jacobian adversarially regularized networks (JARN) as method to train classifiers to produce salient Jacobians that resemble input images.
Through experiments in MNIST, SVHN and CIFAR-1O, find that JARN boosts ad- versarial robustness image classifiers and provides additive robustness to adversarial training.
BACKGROUND AND RELATED WORK
Given input classifier [(x; 0) +> R maps it to output probabilities for i classes in C, where 0 is the classifier's parameters and y E TR the one-hot label for the input. With a training dataset D, the standard method train a classifier ming LE (xxx)) y), where [(x, y) is the standard cross-entropy loss function defined as
empirical risk minimization (ERM), through
y(x,y) B(xx)~D [ log J (x)]
While ERM trains neural networks that perform well on holdout test data, their accuracy drops drastically in the face of adversarial examples. With an adversarial perturbation of magnitude at input x, a model is robust against this attack if
arg max fx(x;0) arg max fi(x +0;0), Ag E B,(e) = ō: 0::6 S IEC
REC
We focus on p = oo in this paper.
Adversarial Training To improve models' robustness, adversarial training (AT) (Goodfellow al., 2016) seek to match the training data distribution with the adversarial test distribution by training classifiers on adversarial examples. Specifically, AT minimizes the loss function:
L(x,y) E (xy))~
L(x +8.) (EB(e)
where the inner maximization, maXgeB(e) C(x vd, is usually performed with an iterative gradient-based optimization. Projected gradient descent (PGD) is one such strong defense which performs the following gradient step iteratively:
8 Proj 56 sign (xx((x+8y))
where Proj(x) = arg minçe x <ll The computational cost of solving Equation 3 is domi- nated by the inner maximization problem of generating adversarial training examples. A naive way to mitigate the computational cost involved to reduce the number gradient descent iterations but that would result in weaker adversarial training examples. A consequence of this is that the models are unable to resist stronger adversarial examples that are generated with more gradient steps, due to phenomenon called obfuscated gradients (Carlini & Wagner 2017 Uesato et al. 2018).
Since the introduction of AT, a line of work has emerged that also boosts robustness with adversarial training examples. Capturing the trade-off between natural and adversarial errors, TRADES (Zhang ct al. 2019) encourages the decision boundary to be smooth by adding a regularization term to re- duce the difference between the prediction of natural and adversarial examples. Qin al. (209
2

---

Published as a conference paper at ICLR 2020
seeks to smoothen the loss landscape through local linearization by minimizing the difference be- tween the real and linearly estimated loss value of adversarial examples. To improve adversarial training. Zhang & Wang (2019) generates adversarial examples by feature scattering, ie., maximiz- ing feature matching distance between the examples and clean samples.
Tsipras et al. (2018) observes that adversarially trained models display an interesting phenomenon: they produce salient Jacobian matrices (VxL) that loosely resemble input images while less robust standard models have noisier Jacobian. Etmann al. (2019) explains that linearized robustness (distance from samples to decision boundary) increases as the alignment between the Jacobian and input image grows. They show that this connection strictly true for linear models but weakens for non- lon-linear neural networks. While these two papers show that robustly trained models result in salient Jacobian matrices, our paper aims to investigate whether directly training to generate salient Jacobian matrices can result in robust models.
Non-Adversarial Training Regularization Provable defenses are first proposed to bound mini- mum adversarial perturbation for certain types of neural networks (Hein & Andriushchenko 2017] Weng et al.) 2018; Raghunathan et al. 2018). One of the most advanced defense from this class of work (Wong et al., 2018) uses a dual network to bound the adversarial perturbation with linear programming. The authors then optimize this bound during training boost adversarial robustness. Apart from this category, closer to our work, several works have studied a regularization term on top of the standard training objective reduce the Jacobian's Frobenius norm. This term aims to reduce the effect input perturbations have on model predictions. Drucker & Le Cun (191)) first proposed this to improve model generalization on natural test samples and called it "double backpropagation'. Two subsequent studies found this to also increases robustness against adversarial examples Ross & Doshi-Veler (0118; Jakubovitz & Giryes 2018). Recently, |Hoflman al. (2019) proposed an efii cient method to approximate the input-class probability output Jacobians of a classifier to minimize the norms of these Jacobians with a much lower computational cost. [Simon- Gabriel al. (2019) proved that double backpropagation is equivalent to adversarial training with l2 examples. Etmann eet al.] (219)) trained robust models using double backpropagation to study the link between robust- ness and alignment in non- linear models but did not propose new defense in their paper. While the double backpropagation term improves robustness by reducing the effect that perturbations in individual pixel have on the classifiers prediction through the Jacobians norm, it does not have the aim to optimize Jacobians to explicitly resemble their corresponding images semantically. Different from these prior work, train the classifier with an adversarial loss term with the aim to make the Jacobian resemble input images more closely and show in our experiments that this approach confers more robustness.
3 JACOBIAN ADVERSARIALLY REGULARIZED NETWORKS (JARN)
Motivation Robustly trained models are observed to produce salient Jacobian matrices that semble the input images. This begs a question in the reverse direction: will an objective function that encourages Jacobian more closely resemble input images, will standard networks become robust? To study this, we look at neural generative networks where models are trained to produce natural-looking images. We draw inspiration from generative adversarial networks (GANs) where generator network is trained to progressively generate more natural images that fool discriminator model. in a min-mi max optimization scenario (Goodfellow al. 2014). More specifically, we frame a classifier as the generator model in the GAN framework so that its Jacobians can progressively fool a discriminator model to interpret them as input images.
Another motivation lies in the high computational cost of the strongest defense to date, adversarial training. The cost on top of standard training is proportional to the number of steps its adversarial examples take to be crafted, requiring an additional backpropagation for each iteration. Especially with larger datasets, there is a need for less eesource-intense (JARN), there is only one additional backpropagation through the classifier and the discriminator model on top of standard training. We share JARN in the following paragraphs and offer some theoretical analysis in 3.1
defense. In our proposed method
3

---

Published as a conference paper at ICLR 2020
Jacobian Adversarially Regularized Networks Denoting input RAwe for h w-size images with c channels, one-hot label vector of classes as y E RR, we express fas(x) E IR as the prediction of the classifier (few), parameterized by 0. The standard cross-entropy loss is
Lets (x,) [-y log fex(x)
(5)
With gradient backpropagation to the input layer, through fas with respect to Leks we can get the Jacobian matrix J E hhwe
Ohch 0x1
J(x) = VxLes= =
(6)
0x.
where hwc. The next part of JARN entails adversarial regularization of Jacobian matrices to induce resemblance with input images. Though the Jacobians of robust models are empirically observed be similar images, their distributions of pixel values do not visually match (Etmann et al., (219). The discriminator model may easily distinguish between the Jacobian and natural images through this difference, resulting in the vanishing gradient (Arjovsky et al.) 2017) for the classifier train on. To address this, an adaptor network (Sapt) is introduced to map the Jacobian into the domain of input images. In our experiments, we use a single 1xl convolutional layer with tanh activation function to model fapt. expressing its model parameters v.. With the J as the input of Sap,, we get the adapted Jacobian matrix J' E Riwe,
='= (J)
We can frame the classifier and adaptor networks as generator G(x.y) y)
(x y) = fapi( (x y))
(8)
learning to model distribution of p. that resembles px-
We now denote a discriminator network, parameterized by o. as faise that outputs a single scalar. Jdise (x) represents the probability that came from training images px rather than p. To train
Go to produce J' that faise perceive as natural images, we employ the following adversarial loss:
Ladv Ex [log faise (x) Ey [log(1 (J')(
= Ex [log fasee (x) + Exx,y) [log(1 faiso (Go,s (x)))
(log faiee [x) | + [xxy)) [log(1 - faise( Sape( VxLes(x,y))))
Combining this regularization loss with the classification loss function Les in Equation (5), optimize through stochastic gradient descent to approximate the optimal parameters for the classifier fas as follows,
0- arg CLats + Ladv)
(10)
where Aado control how much Jacobian ad adversarial regularization term dominates the training.
Since the adaptor network (Smp)) is part of the generator G, its optimal parameters v0" can be found with minimization of the adversarial loss,
arg min Ladv
(11)
On the other hand, the discriminator (faise) is optimized to maximize the adversarial loss term to distinguish Jacobian from input images correctly,
= arg max Lado
(12)
Analogous to how generator from GANs learn to generate images from noise, we add [-c, =]] uniformly distributed noise input image pixels during JARN training phase. Figure T shows
4

---

Published as a conference paper at ICLR 2020
summary of JARN training phase while Algorithm [ details the corresponding pseudo-codes. In our experiments, we find that using JARN framework only on the last few epoch (25%) to train the classifier confers similar adversarial robustness compared to training with JARN for the whole duration. This practice saves compute time and is used for the results reported in this paper.
apt
Image
VaCaa.
cls
disc
Lels
Ladv
Figure 1: Training architecture of JARN.
Algorithm 1: Jacobian Adversarially Regularized Network
Training data Dinin Learning rates for classifier Sas, adaptor fax and discriminator Sane: (ax, B, for each training iteration do Sample (x. y) ~ Duain
Input:
X + x xtC, &, ~ uni
Les + -y" log fxx
(1) Compute classification cross-entropy loss
+ VxLal
D (2) Compute Jacobian matrix (3) Adapt Jacobian to image domain (4) Compute adversarial loss
J' fan(J)
Law log Save (x) + logg(1 - Sane(J'))
0 0 Vo Andt Ladu)
(5a) Update the classifier fes to minimize Ceta and Ludt >> (5b) Update the adaptor fag minimize Lade b- (5c) Update the discriminator fase to maximize Lado
10
10- - Lode
11
+ σ + VoLado
3.1 THEORETICAL ANALYSIS
Here, we study the link between JARN's adversarial regularization term with the notion of linearized robustness. Assuming a non- non-parameteric setting where the models have infinite capacity, we have the following theorem while optimizing G with the adversarial loss Ladv-
Theorem 3.1. The global minimum of Lady is achieved when G(x) maps x to itself. i.e., G(x)
Its proof is deferred to & A If we assume Jacobian J of our classifier fel to be the direct output of G, then J G(x) x the global minimum of the adversarial objective.
In Etmann et al. (2019), it is shown that the linearized robustness of a model is loosely upper- bounded by the alignment between the Jacobian and the input image. More concretely, denoting T as the logits value of class i in a classifier F, its linearized robustness can be expressed as
q (x)
p(x) = minj TV [*]] Here we quote the theorem from Etmann et al. 2019:
Theorem 3.2 (Linearized Robustness Bound). (Emmann et al. 2019) Defining i' arg max, V and j arg m V as top two prediction, we let the Jacobian with respect to the difference in top two logits be g == <x(V" -w0' (x). Expressing alignment between the Jacobian with the input a(x)
1(.9)1 then [gll
C
p(x) < a(x) + |Ill|
(13)
where C. is a positive constant.
5

---

Published as a conference paper at ICLR 2020
Combining with what have in Theorem 3.1 assuming to be close g in fixed constant term, the alignment term (x) in Equation 13 is maximum when Cadv reaches its global minimum. Though this is not strict upper bound and, to facilitate the training in JARN in practice,
an adaptor network to transform the Jacobian, i.e., J' = [a]( our experiments show that model robustness can be improved with this adversarial regularization.
EXPERIMENTS
We conduct experiments three image datasets, MNIST, SVHN and CIFAR-10 evaluate the adversarial robustness of models trained by JARN.
4.1 MNIST
Setup MNIST consists of 60k training and 10k test binary-colored images. We train a CNN, sequentially composed of 3 convolutional layers and I final softmax layer. All 3 convolutional layers have a stride of while each layer has an increasing number output channels (64-128-2566 For JARN, use Aadv 1, discriminator network of 2 CNN layers (64-128 output channels) and update it for every 10 fas training iterations. We evaluate trained models against adversarial examples with Lo perturbation 0.3, crafted from FGSM and PGD (5 & 40 iterations). FGSM generates weaker adversarial examples with only one gradient step and weaker than the iterative PGD method.
The CNN trained with JARN shows improved adversarial robustness from standard model across the three types of adversarial examples (Table I In the MNIST experiments, we find that data augmentation with uniform noise to pixels alone provides no benefit in robustness from the baseline.
Results
Table 1: MNIST accuracy (%) on adversarial and clean test samples.
Model
FGSM
PGD5
PGD40
Clean
Standard
76.5
0
0
98.7
Uniform Noise
77.5
0
0.02
98.7
JARN
98.4
98.1
98.1
98.8
4.2 SVHN
Setup SVHN is 10-class house number image classification dataset with 73257 training and 26032 test images, each size 32 x 32 x 3. train the Wide-Resnet model following hyper- parameters from (Madry al. 2017)'s setup for their CIFAR-10 experiments. For JARN, Aud = 5, a discriminator network of 5 CNN layers (16-32-64- 044-122-256 output channels) and update it for every 20 Jas training iterations. We evaluate trained models against adversarial examples with (s 8/255), crafted from FGSM and 5, 10, 20-iteration PGD attack.
Results Similar to the findings in $ 4.1 JARN advances the adversarial robustness of the classifier from the standard baseline against all four types of attacks. Interestingly, uniform noise image aug- mentation increases adversarial robustness from the baseline the SVHN experiments, concurring with previous work that shows noise augmentation improves robustness (Ford et al. 2019).
Table 2: SVHN accuracy (%) adversarial and clean test samples.
Model
FGSM
PGD5
PGDTO
PGD20
Clean
Standard
64.4
26.0
5.47
1.96
94.7
Uniform Noise
65.0
42.6
18.4
9.21
95.3
JARN
67.2
57.5
37.7
26.79
94.9

---

Published as a conference paper at ICLR 2020
4.3 CIFAR-10
Setup 10 contains 32 x 32 x 3 colored images labeled as 10 classes, with 50k training and 10k test images. We train the Wide-Resnet model using similar hyperparameters to (Madry et al. (017) for our experiments. Following the settings from Madry et al. (2017), we compare with a strong adversarial training baseline (PGD-AT7) that involves training the model with adversarial examples generate with 7-iteration PGD attack. For JARN, we use 'ddv = 1, discriminator net- work of 5 CNN layers (32-64-12) 1---128-22 22--256-512 output channels) and update it for every 20 fas training iterations. We evaluate trained models against adversarial examples with (s 8/255), crafted from FGSM and PGD (5, 10 & 20 iterations). We also add in a fast gradient sign attack baseline (FGSM- ATI) that generates adversarial training examples with only 1 gradient step. Though FGSM-trained models are known to rely on obfuscated gradients to counter weak attacks, we augment it with JARN to study if there is additive robustness benefit against strong attacks. We also implemented double backpropagation (Drucker& & Le Cun 1991 Ross & Doshi- Velez 2018) to compare.
Results Similar to results from the previous two datasets, the JARN classifier performs better than the standard baseline for all four types of adversarial examples. Compared to the model trained with uniform-noise augmentation, JARN performs closely in the weaker FGSM attack while being more robust against the two stronger PGD attacks. JARN also outperforms the double backpropagation baseline, showing that regularizing for salient Jacobians confers more robustness than regularizing for smaller Jacobian Frobenius norm values. The strong PGD-AT7 baseline shows higher robustness against PGD attacks than the JARN model. When train JARN together with 1-step adversarial training (JARN-ATI), we find that the model's 's robustness exceeds that of strong PGD-AT7 baseline on all four adversarial attacks, suggesting JARN's gain in robustness is additive to that of AT.
Table 3: CIFAR-10 accuracy (%) on adversarial and clean test samples.
Model
FGSM
PGD5
PGDI0
PGD20
Clean
Standard Uniform Noise
13.4
0
0
0
95.0
67.4
44.6
19.7
7.48
94.0
FGSM-ATI
94.5
0.25
0.02
0.01
91.7
Double Backprop
28.3
0.05
0
95.7
JARN
67.2
50.0
27.6
15.5
93.9
PGD-AT7
56.2
55.5
47.3
45.9
87.3
JARN-ATI
65.7
60.1
51.8
46.7
84.8
4.3.1 GENERALIZATION OF ROBUSTNESS
Adversarial training (AT) based defenses generally train the model examples generated by pertur- bation of a fixed e. Unlike AT, JARN by itself does not have & as training parameter. To study how JARN-ATI robustness generalizes, we conduct PGD attacks of varying and strength (5, 10 and 20 iterations). We also include another PGD-AT7 baseline that was trained at higher & =
JARN-ATI shows higher robustness than the two PGD-AT7 baselines against attacks with higher E values (< 8/255) across the three PGD attacks, as shown in Figure 2 We also observe that the PGD-AT7 variants outperform each other on attacks with E values close to their training E, suggest- ing that their robustness is more adapted to resist adversarial examples that they are trained on. This relates to findings by Tramèr & Boneh (2019) which shows that robustness from adversarial training is highest against the perturbation type that models are trained on.
PGDS attack
PGD10 attack
PGD20 attack
Accuracy 1863
80
80
80-
60
60
PGD-ATT, (R/255)
40
PCDATIT (12255)
20-
20
20
5
10
15
10
15
20
0
5
10
15
20
(122581
(1/255)
(1.255)
Figure 2: Generalization of model robustness to PGD attacks of different E values.
7

---

We compute the classification loss value along the adversarial perturbation's direction and a random that the Published as a conference paper at ICLR 2020
4.3.2 LOss LANDSCAPE
orthogonal direction analyze the loss landscape of the models. From Figure 3 models trained by the standard and FGSM-AT method display loss surfaces that are jagged and non- linear. This explains why the FGSM-AT display modest accuracy at the weaker FGSM attacks but fail at attacks with more iterations, phenomenon called obfuscated gradients (Carlini & Wagner, 2017 Uesato al. 2018) where the initial gradient steps are still trapped within the locality of the input but eventually escape with more iterations. The JARN model displays a loss landscape that less steep compared to the standard and FGSM-AT models, marked by the much lower (1 order of magnitude) loss value in Figure 3c When JARN is combined with one iteration of adversarial training. the JARN-ATI model is observed to have much smoother loss landscapes, similar to that of the PGD-AT7 model, strong baseline previously observed to be free of obfuscated gradients. This suggests that JARN and AT have additive benefits and JARN-ATI adversarial robustness is not attributed obfuscated gradients.
possible explanation behind the improved robustness through increasing Jacobian saliency is that the space of Jacobian shrinks under this regularization, i.e., Jacobians have resemble non-noisy images. Intuitively, this means that there would be fewer paths for an adversarial example to reach an optimum in the loss landscape, improving the model's robustness.
35
28
1g 4
15
-15.10
15..10
15
(a) Standard
(b) FGSM-ATI
JARN
-3365 5-bit
15.10
-3.100
(d) JARN-ATI
(c) PGD-AT7
Figure Loss surfaces of models along the adversarial perturbation and random direction.
4.3.3 SALIRNCY OF JACOBIAN
The Jacobian matrices of JARN model and PGD-AT salient and visually resemble the images more than those from the standard model (Figure 4 Upon closer inspection, the Jacobian matrices of the PGD-AT model concentrate their values small regions around the object of interest whereas those of the JARN model cover a larger proportion of the images. One explanation that the JARN model is trained to fool the discriminator network and hence generates Jacobian that contains details of input images to more closely resemble them.
4.3.4 COMPUTE TIME
Training with JARN is computationally more efficient when compared to adversarial training (Ta- ble 4). Even when combined with FGSM adversarial training JARN, it takes less than half the time of 7-step PGD adversarial training while outperforming it in robustness.
8

---

Published as a conference paper at ICLR 2020
deer
Trog
horse
shlp
cat
dog
plana
bind
Image
JORWATT
PGD-AT7
Figure 4: Jacobian matrices of CIFAR-10 models.
Table Average wall-clock time per training epoch for CIFAR-IO adversarial defenses.
Model PGD-AI7
JARN-ATI
FGSM-ATI
JARN only
Time (scc)
704
294
267
217
4.3.5 SENSITIVITY TO HYPERPARAMETERS
The performance of GANs in image generation has been well-known to be sensitive to training hyperparameters. We JARN performance across range of Xadv. batch size and discriminator update intervals that are different from 4.3 and find that its performance is relatively stable across hyperparameter changes, as shown Appendix Figure a typical GAN framework, each training step involves real image sample and an image generated from noise that is decoupled from the real sample. In contrast, a Jacobian is conditioned on its original input image and both are used in the same training step of JARN. This training step resembles that of VAB-GAN (Larsen et al.) 2015) where pairs of real images and its reconstructed versions are used for training together, resulting in generally more stable gradients and convergence than GAN. We believe that this similarity favors JARN's stability over a wider range hyperparameters.
4.3.6 BLACK-BOX TRANSFER ATTACKS
Transfer attacks adversarial examples generated from alternative, substitute model and eval- uated on the defense to test for gradient masking (Papernot et al.) [016)] Carlini et al.) 2019). More specifically, defenses relying on gradient masking will display lower robustness towards transfer tacks than white-box attacks. When evaluated on such black-box attacks using adversarial examples generated from PGD-AT7 trained model and their differently initialized versions, both JARN and JARN-ATI display higher accuracy than when under white-box attacks (Table(5). This demonstrates that JARN's robustness does not rely on gradient masking. Rather unexpectedly, JARN performs better than JARN-AT1 under the PGD-AT7 transfer attacks, which we believe is attributed to its better performance on clean test samples.
Table 5: CIFAR-10 10 accuracy (%) on transfer attack where adversarial examples are generated from PGD-AT7 trained model.
PGD-AT7
Same Model
White-box
Model
Clean
FGSM
PGD20
FGSM
PGD20
FGSM
PGD20
JARN
79.6
76.7
73.6
17.4
67.2
15.5
93.9
JARN-ATI
66.4
63.0
70.3
59.3
65.7
46.7
84.8

---

Published as a conference paper at ICLR 2020
5 CONCLUSIONS
In this paper, we show that training classifiers to give more salient input Jacobian matrices that re- semble images advance their robustness against adversarial examples. We achieve this through an adversarial regularization framework (JARN) that train the model's Jacobians to fool a discrim- inator network into classifying them as images. Through our experiments in three image datasets, JARN boosts adversarial robustness of standard models and give competitive performance when added on to weak defenses like FGSM. Our findings open the viability of improving the saliency of Jacobian as new avenue to boost ad adversarial robustness.
ACKNOWLEDGMENTS
This work is funded by the National Research Foundation, Singapore under its AI Singapore pro- gramme [Award No.: AISG-RP-2018 and the Data Science and Artificial Intelligence Research Center (DSAIR) at Nanyang Technological University.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875. 2017.
Wasserstein gan.
arXiv preprint
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End end learning for self-driving cars. arXiv preprint arXiv:1604.0755:
2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, and Aleksander Madry. On evaluating adversarial robustness. arXiv preprint arXiv:1902.067000 2019.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Wea Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. arXiv preprint arXiv: arXiv:1907.0200 1907.02044, 2019.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv:1903.05799 2019.
Harris Drucker and Yann Le Cun. Double backpropagation increasing generalization performance. In JJCNN-91-Seatie International Joint Conference on Neural Networks, volume 2, pp. 145-150. IEEE, 1991.
Christian Etmann, Sebastian Lunz, Peter Maass, and Carola-Bibiane Schönlieb. On the nection between adversarial robustness and saliency map interpretability. arXiv preprint arXiv:1905.00 04172, 2019.
Nic Ford, Justin Gilmer, Nicolas Carlini, and Dogus Cubuk. Adversarial examples are natural consequence of test error in noise. arXiv preprint 100i::/901.10555 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Ue- sato, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv: 18/0. /2715, 2018
10

---

Published as a conference paper at ICLR 2020
Matthias Hein and Maksym Andriushchenko, Formal guarantees the robustness of classifier against adversarial manipulation. In Advances in Neural Information Processing Systems, pp- 2266-2276, 2017.
Judy Hoffman, Daniel A Roberts, and Sho Yaida. Robust learning with jacobian regularization. arXiv preprint arXiv: 100.:::990.0.29, 2019.
Daniel Jakubovitz and Raja Giryes. Improving dnn robustness to adversarial attacks using jacobian regularization. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 514-529, 2018.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprini arXiv: 1803,0633, 2018.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. arXiv preprint arXiv: 2015.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nalure, 521 2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. arXiv preprint 2017.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv: 1605.07277, 2016.
Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Ku- rakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy. Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, Yi-Lin Juang. Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, and Rujun Long. Technical report on the cleverhans v2.1.0 adversarial examples library. arXiv preprint arXiv:1610.000 00768, 2018.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Alhussein Fawzi, Soham De, Robert Stanforth, Pushmeet Kohli, al. Adversarial robustness through local linearization. arXiv preprint arXiv:1907.0220 2019.
Aditi Raghunathan, Jacob Steinhardt, and Percy S Liang. Semidefinite relaxations for certifying robustness to adversarial examples. In Advances in Neural Information Processing Systems, pp. 10877-10887, 2018.
Andrew Slavin Ross and Finale Doshi-Veles Improving the adversarial robustness and interpretabil- ity of deep neural networks by regularizing their input gradients. In Thirty-second AAAI confer- ence on artificial intelligence, 2018.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! arXiv preprint arXiv:1904. 12843. 2019.
Carl-Johann Simon Gabriel, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, and David Lopez- Paz. First-order adversarial vulnerability of neural networks and input dimension. In International Conference Machine Learning, pp. 5809-5817, 2019.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6199, 2013.
Florian Tramèr and Dan Bonch. Adversarial training and robustness for multiple perturbations. arXiv preprint arXiv: 1904. /3000, 2019.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. arXiv preprint /805. 12/52, 2018.
11

---

Published as a conference paper at ICLR 2020
Jonathan Uesato, Brendan O' O'Donoghue, Aaron van den Oord, and Pushmeet Kohli. Adversarial risk and the dangers of evaluating against weak attacks. arXiv preprint
2018.
Tsui- Weng, Pin- Yu Chen, Lam M Nguyen, Mark S Squillante, Ivan Oseledets, and Luca Daniel. Proven: Certifying robustness of neural networks with probabilistic approach. arXiv preprint arXiv:1812.08832. 2018.
Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and J Zico Kolter. Scaling provable adversarial defenses. In Advances Neural Information Processing Systems, pp. 8400-8409 2018
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 501-509, 2019.
Haichao Zhang and Jianyu Wang. Defense against adversarial attacks using feature scattering- adversarial training. arXiv preprint arXiv: 10764, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint (XKiv/9001.8800 2019.
12

---

Published as a conference paper at ICLR 2020
A PROOF OF THEOREM 3.1
Theorem A.1. The global minimum of Lude is achieved when G(x) maps x to itself, ie, G(x)
X.
Proof. From (Goodfellow et al. 2014), for fixed G, the optimal discriminator
Pdita (x)
dise (x)
(14)
Pdlata
(x) + pe (x)
We can include the optimal discriminator into Equation 9 to get
Ladv (G)
Ex-pan [log Jäse (x)] + Exe-pan [log(1 - Jaise (G(x)))]
Ex-pau [log fase (x)] + Exewra [log(1 fäise (x))
Petat (x)
px(x)
Ex-epa
log
+ Exxupa
log
Petata (x) + (x)
Paaa x) + pG (x)
Pdttaa (x))
po po( (x)
Ex-epau
log
Exmop
log [Pamau (x) + pc())))
2 log 2
(Pastul (x) + PG((x)
2
KL Pdata
Pdata + PG
+ KL
PG
Pdata
+ PG
log 4
2
2
(p (PG) - log
(15)
where KL and JS are the Kullback-Leibler and Jensen-Shanmen divergence respectively, Since the Shaen-Shnnnon divergence is always non-negative, L (G) reaches its global minimum value of llg 4 when (p PG)) = 0. When G(x) = x, we get Pdata = Pa and consequently
J (p |PG) O, thus completing the proof.
B SENSITIVITY TO HYPERPARAMETERS
100
100
100
80
80
80
Accuracy (%)
-B- Clean
60
60
60
FGSM
PGD5
PGD10
PGD20
20
6
20
10
10a
101
20
40
60
15
20
25
30
Awdr
Batch Size
Disc update interval
Figure 5: Accuracy of JARN with different hyperparameters on CIFAR- 10 test samples.
13


Reviews:
[{"review_id": "ryxg23o2tS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel regularization strategy for improving the robustness of networks to adversarial noise. A term is added to the standard supervised cross-entropy loss that encourages the Jacobian of the network to itself be interpreted as a valid image. This \"regularization\" term is constructed by running the input-output Jacobian of the classification network through an \"adapter network\" and then in turn interpreting its output as a \"generator\" in a GAN setup. A separate discriminator network is training to distinguish real input images from these adapter-processed input-output Jacobians. The overall regularization is the standard minimax GAN loss applied to this generator/discriminator setup. The impetus for this stems from a previous observation that salient or interpretable input-output Jacobians naturally arise for networks that have undergone adversarial training to increase robustness.\n\nAlthough this whole setup seems to be a little \"Rube-Goldberg\"-esque, I think there's some real sensible reasons for this sort of regularization to make intuitive sense. The input-output Jacobian characterizes how much the output (i.e. the logits) are affected by small changes to the input. The Jacobian, reinterpreted as living in the input image space (as the authors do), is a map of which input pixels have the strongest effect on the output of the network. If the Jacobian image looks like the underlying input image -- in particular, highlighting the labeled object -- this indicates that changing those pixels will result in the largest change on the network output. (This should be clear when looking at Figure 4 of the paper.) On the other hand, adversarial noise by definition leaves the underlying object alone (so that a human isn't aware of the perturbation) and modifies other pixels. Models that fall for such adversarial noise will not have salient Jacobians.\n\nThis is an amusing original idea, and I think this paper probably should be accepted to ICLR -- though I don't hold that position very strongly. However, I think the most interesting point is idea of Jacobian saliency, which is from prior work (Tsipras et al., 2018) that I haven't read. Therefore, I'm not sure how significant this paper is on it's own. Regardless, I would have liked to see more discussion in the paper of why Jacobian saliency should confer robustness (as I tried to do in the paragraph above), with perhaps some additional experiments designed around understanding whether this intuition (or something similar) is actually correct. There's some discussion of the theory behind the method in section 3.1, but it's not very intuitive to the situation at hand (non-linear neural networks), and I don't find it particularly informative.\n\nFinally, some effort is spent arguing that this method is more computational efficient than adversarial training -- I wonder if that's still true when the all the complexity of GAN training is taken into account or how to consider that point when part of the conclusion is that their method is best when it is also combined with some amount of adversarial training.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rklhsmOwFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think the main contribution of this paper is that it introduces a new way of robust training by encouraging Jacobian saliency. Previous research like Etmann et al 2019 and Tsipras et al 2018 showed that robustness leads to saliency. But surprisingly, this paper shows the other way, saliency map can also lead to robustness, which indicates a stronger connection between these two. In general, I like the intuition behind this paper, since it introduces a new perspective of robust training.\n\nThe training method proposed in this paper is still kind of preliminary, though. I suspect that training a GAN together with the classifier will cost even more time than min-max adversarial training or some certified robust training methods. It would be great if the authors can provide the training time comparison between JARN and some state-of-the-art robust training methods. Another concern is reproducibility since the training process of GAN is sensitive to hyperparameter selection. It would be better if the author can have some discussion on the training process to show that the reported performance of the defense is easy to reproduce instead of cherry-picking.  Also, there are too many works on robustness defense that have been proven ineffective (consider the works by Carlini). Since this is a completely new way of robust training and there is no certified guarantee, I suggest the authors refer [1] to evaluate the effectiveness of the defense more thoroughly to convince the readers that it really works. Especially, evaluation under adaptive attack is necessary.\n\nI think this is a very interesting work. But since this method is completely new, more detailed information is needed to convince me that it really works. If it does work, I believe there must exist better ways to encourage Jacobian saliency than using a GAN.\n\n[1] Carlini, Nicholas, et al. \"On evaluating adversarial robustness.\" arXiv preprint arXiv:1902.06705 (2019).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1xkPfF6tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIt was previously observed that models that were more robust to adversarial perturbation had more interpretable jacobian. The authors attempt to train for interpretable jacobian in order to improve the robustness of the model.\nThis is done by employing a GAN-like procedure where a discriminator attempts to distinguish between the transformed jacobian matrix (fake images, equivalent to generator) and real images.\n\nExperiments indicates that this improves robustness compared to unprotected models and approximately similarly to models trained with adversarial training.\n\nComments:\n* The motivation given for this line of research is the cost of adversarial training (2nd paragraph of Section 3)\nNo experimental comparison is given with regards to the time it takes to train a model with adversarial training, versus the time it takes to train a model with JARN. It is also important to note that this introduces additional complexity (needs to choose an architecture for the discriminator, tune proper learning rates, etc...), which is not mentionned.\n\n* Why not test simpler jacobian regularization method as proposed by other papers (see below). Proposition 3 of Simon-Gabriel et al. shows that results similar to adversarial training can be obtained, and they don't need several iterations like adversarial training, nor do they need to train an additional discriminator like your method.\n\nOpinion:\nThe paper provides an interesting proof of concept for a method, showing that it is feasible. It however doesn't make the the case for why it is a good idea. Discussion and comparison to very significant related work is missing and experimental measurement of any advantages of the proposed method vs. adversarial training is lacking. I think that these aspects should be improved before the paper is ready for publication.\n\nTypos:\nLine 11 in Algorithm 1 -> The label is wrong, i assume it's \"Update the discriminator f_disc to maximize L_adv\"\n\nRelated works that needs discussing:\n- Drucker, Lecun 91, \"Double backpropagation increasing generalization performance\" for other regularizer on the jacobian, discusses generalization rather than robustness.\n- Simon-Gabriel et al., \"First-order Adversarial Vulnerability of Neural Networks and Input Dimension\"\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends previous observations (Tsipars, Etmann etc) in relations between Jacobian and robustness and directly train a model that improves robustness using Jacobians that look like images. The questions regarding computation time (suggested by two reviewers, including one of the most negative reviewers) are appropriately addressed by the authors (added experiments). Reviewers agree that the idea is novel, and some conjectured why the paper’s idea is a very sensible one. We think this paper would be an interest for ICLR readers. Please address any remaining comments from the reviewers before the final copy.

Decision: Accept (Poster)