Title: JACOBIAN ADVERSARIALLY REGULARIZED NETWORKS FOR ROBUSTNESS

Abstract: Adversarial examples are crafted with imperceptible perturbations with the intent to fool neural networks. Against such attacks, adversarial training and its variants stand as the strongest defense to date. Previous studies have pointed out that robust models that have undergone adversarial training tend to produce more salient and interpretable Jacobian matrices than their non-robust counterparts. A natural question is whether a model trained with an objective to produce salient Jacobian can result in better robustness. This paper answers this question with affirmative empirical results. We propose Jacobian Adversarially Regularized Networks (JARN) as a method to optimize the saliency of a classifier's Jacobian by adversarially regularizing the model's Jacobian to resemble natural training images 1 . Image classifiers trained with JARN show improved robust accuracy compared to standard models on the MNIST, SVHN and CIFAR-10 datasets, uncovering a new angle to boost robustness without using adversarial training examples.

1. INTRODUCTION:

Deep learning models have shown impressive performance in a myriad of classification tasks (Le-Cun et al., 2015) . Despite their success, deep neural image classifiers are found to be easily fooled by visually imperceptible adversarial perturbations (Szegedy et al., 2013) . These perturbations can be crafted to reduce accuracy during test time or veer predictions towards a target class. This vulnerability not only poses a security risk in using neural networks in critical applications like autonomous driving (Bojarski et al., 2016) but also presents an interesting research problem about how these models work.

Many adversarial attacks have come into the scene (Carlini & Wagner, 2017; Papernot et al., 2018; Croce & Hein, 2019) , not without defenses proposed to counter them (Gowal et al., 2018; Zhang et al., 2019) . Among them, the best defenses are based on adversarial training (AT) where models are trained on adversarial examples to better classify adversarial examples during test time (Madry et al., 2017) . While several effective defenses that employ adversarial examples have emerged (Qin et al., 2019; Shafahi et al., 2019) , generating strong adversarial training examples adds non-trivial computational burden on the training process (Kannan et al., 2018; Xie et al., 2019) .

Adversarially trained models gain robustness and are also observed to produce more salient Jacobian matrices (Jacobians) at the input layer as a side effect (Tsipras et al., 2018) . These Jacobians visually resemble their corresponding images for robust models but look much noisier for standard non-robust models. It is shown in theory that the saliency in Jacobian is a result of robustness (Etmann et al., 2019) . A natural question to ask is this: can an improvement in Jacobian saliency induce robustness in models? In other words, could this side effect be a new avenue to boost model robustness? To the best of our knowledge, this paper is the first to show affirmative findings for this question.

To enhance the saliency of Jacobians, we draw inspirations from neural generative networks (Choi et al., 2018; Dai & Wipf, 2019) . More specifically, in generative adversarial networks (GANs) (Goodfellow et al., 2014) , a generator network learns to generate natural-looking images with a training objective to fool a discriminator network. In our proposed approach, Jacobian Adversarially Regularized Networks (JARN), the classifier learns to produce salient Jacobians with a regularization objective to fool a discriminator network into classifying them as input images. This method offers a new way to look at improving robustness without relying on adversarial examples during training. With JARN, we show that directly training for salient Jacobians can advance model robustness against adversarial examples in the MNIST, SVHN and CIFAR-10 image dataset. When augmented with adversarial training, JARN can provide additive robustness to models thus attaining competitive results. All in all, the prime contributions of this paper are as follows:

• We show that directly improving the saliency of classifiers' input Jacobian matrices can increase its adversarial robustness. • To achieve this, we propose Jacobian adversarially regularized networks (JARN) as a method to train classifiers to produce salient Jacobians that resemble input images. 

2. BACKGROUND AND RELATED WORK:

Given an input x, a classifier f (x; θ) : x → R k maps it to output probabilities for k classes in set C, where θ is the classifier's parameters and y ∈ R k is the one-hot label for the input. With a training dataset D, the standard method to train a classifier f is empirical risk minimization (ERM), through min θ E (x,y)∼D L(x, y), where L(x, y) is the standard cross-entropy loss function defined as

EQUATION

While ERM trains neural networks that perform well on holdout test data, their accuracy drops drastically in the face of adversarial test examples. With an adversarial perturbation of magnitude ε at input x, a model is robust against this attack if

arg max i∈C f i (x; θ) = arg max i∈C f i (x +δ; θ) , ∀δ ∈ B p (ε) = δ : δ p ≤ ε (2)

We focus on p = ∞ in this paper.

Adversarial Training To improve models' robustness, adversarial training (AT) (Goodfellow et al., 2016) seek to match the training data distribution with the adversarial test distribution by training classifiers on adversarial examples. Specifically, AT minimizes the loss function:

EQUATION

where the inner maximization, max δ∈B(ε) L(x +δ, y), is usually performed with an iterative gradient-based optimization. Projected gradient descent (PGD) is one such strong defense which performs the following gradient step iteratively:

EQUATION

where Proj(x) = arg min ζ∈B(ε) (Carlini & Wagner, 2017; Uesato et al., 2018) .

Since the introduction of AT, a line of work has emerged that also boosts robustness with adversarial training examples. Capturing the trade-off between natural and adversarial errors, TRADES (Zhang et al., 2019) encourages the decision boundary to be smooth by adding a regularization term to reduce the difference between the prediction of natural and adversarial examples. Qin et al. (2019) seeks to smoothen the loss landscape through local linearization by minimizing the difference between the real and linearly estimated loss value of adversarial examples. To improve adversarial training, Zhang & Wang (2019) generates adversarial examples by feature scattering, i.e., maximizing feature matching distance between the examples and clean samples. Tsipras et al. (2018) observes that adversarially trained models display an interesting phenomenon: they produce salient Jacobian matrices (∇ x L) that loosely resemble input images while less robust standard models have noisier Jacobian. Etmann et al. (2019) explains that linearized robustness (distance from samples to decision boundary) increases as the alignment between the Jacobian and input image grows. They show that this connection is strictly true for linear models but weakens for non-linear neural networks. While these two papers show that robustly trained models result in salient Jacobian matrices, our paper aims to investigate whether directly training to generate salient Jacobian matrices can result in robust models.

Non-Adversarial Training Regularization Provable defenses are first proposed to bound minimum adversarial perturbation for certain types of neural networks (Hein & Andriushchenko, 2017; Weng et al., 2018; Raghunathan et al., 2018) . One of the most advanced defense from this class of work (Wong et al., 2018) 2019) trained robust models using double backpropagation to study the link between robustness and alignment in non-linear models but did not propose a new defense in their paper. While the double backpropagation term improves robustness by reducing the effect that perturbations in individual pixel have on the classifiers prediction through the Jacobians norm, it does not have the aim to optimize Jacobians to explicitly resemble their corresponding images semantically. Different from these prior work, we train the classifier with an adversarial loss term with the aim to make the Jacobian resemble input images more closely and show in our experiments that this approach confers more robustness.

3. JACOBIAN ADVERSARIALLY REGULARIZED NETWORKS (JARN):

Motivation Robustly trained models are observed to produce salient Jacobian matrices that resemble the input images. This begs a question in the reverse direction: will an objective function that encourages Jacobian to more closely resemble input images, will standard networks become robust? To study this, we look at neural generative networks where models are trained to produce natural-looking images. We draw inspiration from generative adversarial networks (GANs) where a generator network is trained to progressively generate more natural images that fool a discriminator model, in a min-max optimization scenario (Goodfellow et al., 2014) . More specifically, we frame a classifier as the generator model in the GAN framework so that its Jacobians can progressively fool a discriminator model to interpret them as input images.

Another motivation lies in the high computational cost of the strongest defense to date, adversarial training. The cost on top of standard training is proportional to the number of steps its adversarial examples take to be crafted, requiring an additional backpropagation for each iteration. Especially with larger datasets, there is a need for less resource-intensive defense. In our proposed method (JARN), there is only one additional backpropagation through the classifier and the discriminator model on top of standard training. We share JARN in the following paragraphs and offer some theoretical analysis in § 3.1.

Jacobian Adversarially Regularized Networks Denoting input as x ∈ R hwc for h × w-size images with c channels, one-hot label vector of k classes as y ∈ R k , we express f cls (x) ∈ R k as the prediction of the classifier (f cls ), parameterized by θ. The standard cross-entropy loss is

EQUATION

With gradient backpropagation to the input layer, through f cls with respect to L cls , we can get the Jacobian matrix J ∈ R hwc as:

EQUATION

where d = hwc. The next part of JARN entails adversarial regularization of Jacobian matrices to induce resemblance with input images. Though the Jacobians of robust models are empirically observed to be similar to images, their distributions of pixel values do not visually match (Etmann et al., 2019) . The discriminator model may easily distinguish between the Jacobian and natural images through this difference, resulting in the vanishing gradient (Arjovsky et al., 2017) for the classifier train on. To address this, an adaptor network (f apt ) is introduced to map the Jacobian into the domain of input images. In our experiments, we use a single 1x1 convolutional layer with tanh activation function to model f apt , expressing its model parameters as ψ. With the J as the input of f apt , we get the adapted Jacobian matrix J ∈ R hwc ,

EQUATION

We can frame the classifier and adaptor networks as a generator G(x, y)

EQUATION

learning to model distribution of p J that resembles p x .

We now denote a discriminator network, parameterized by φ, as f disc that outputs a single scalar. f disc (x) represents the probability that x came from training images p x rather than p J . To train G θ,ψ to produce J that f disc perceive as natural images, we employ the following adversarial loss:

EQUATION

Combining this regularization loss with the classification loss function L cls in Equation ( 5), we can optimize through stochastic gradient descent to approximate the optimal parameters for the classifier f cls as follows,

EQUATION

where λ adv control how much Jacobian adversarial regularization term dominates the training.

Since the adaptor network (f apt ) is part of the generator G, its optimal parameters ψ * can be found with minimization of the adversarial loss,

EQUATION

On the other hand, the discriminator (f disc ) is optimized to maximize the adversarial loss term to distinguish Jacobian from input images correctly,

EQUATION

Analogous to how generator from GANs learn to generate images from noise, we add [-ε, -ε] uniformly distributed noise to input image pixels during JARN training phase. Figure 1 shows a summary of JARN training phase while Algorithm 1 details the corresponding pseudo-codes. In our experiments, we find that using JARN framework only on the last few epoch (25%) to train the classifier confers similar adversarial robustness compared to training with JARN for the whole duration. This practice saves compute time and is used for the results reported in this paper. Its proof is deferred to § A. If we assume Jacobian J of our classifier f cls to be the direct output of G, then J = G(x) = x at the global minimum of the adversarial objective.

In Etmann et al. (2019) , it is shown that the linearized robustness of a model is loosely upperbounded by the alignment between the Jacobian and the input image. More concretely, denoting Ψ i as the logits value of class i in a classifier F , its linearized robustness ρ can be expressed as

ρ(x) := min j =i * Ψ i * (x)-Ψ j (x) ∇xΨ i * (x)-∇xΨ j (x) .

Here we quote the theorem from Etmann et al. ( 2019): Theorem 3.2 (Linearized Robustness Bound). (Etmann et al., 2019) Defining i * = arg max i Ψ i and j * = arg max j =i * Ψ j as top two prediction, we let the Jacobian with respect to the difference in top two logits be g := ∇ x (Ψ i * -Ψ j * )(x). Expressing alignment between the Jacobian with the input as α

EQUATION

)

where C is a positive constant.

Combining with what we have in Theorem 3.1, assuming J to be close to g in a fixed constant term, the alignment term α(x) in Equation ( 13) is maximum when L adv reaches its global minimum. Though this is not a strict upper bound and, to facilitate the training in JARN in practice, we use an adaptor network to transform the Jacobian, i.e., J = f apt (J), our experiments show that model robustness can be improved with this adversarial regularization.

4. EXPERIMENTS:

We conduct experiments on three image datasets, MNIST, SVHN and CIFAR-10 to evaluate the adversarial robustness of models trained by JARN.

4.1. MNIST:

Setup MNIST consists of 60k training and 10k test binary-colored images. We train a CNN, sequentially composed of 3 convolutional layers and 1 final softmax layer. All 3 convolutional layers have a stride of 5 while each layer has an increasing number of output channels (64-128-256).

For JARN, we use λ adv = 1, a discriminator network of 2 CNN layers (64-128 output channels) and update it for every 10 f cls training iterations. We evaluate trained models against adversarial examples with l ∞ perturbation ε = 0.3, crafted from FGSM and PGD (5 & 40 iterations). FGSM generates weaker adversarial examples with only one gradient step and is weaker than the iterative PGD method.

Results:

The CNN trained with JARN shows improved adversarial robustness from a standard model across the three types of adversarial examples (Table 1 ). In the MNIST experiments, we find that data augmentation with uniform noise to pixels alone provides no benefit in robustness from the baseline. Results Similar to the findings in § 4.1, JARN advances the adversarial robustness of the classifier from the standard baseline against all four types of attacks. Interestingly, uniform noise image augmentation increases adversarial robustness from the baseline in the SVHN experiments, concurring with previous work that shows noise augmentation improves robustness (Ford et al., 2019) . FGSM and PGD (5, 10 & 20 iterations) . We also add in a fast gradient sign attack baseline (FGSM-AT1) that generates adversarial training examples with only 1 gradient step. Though FGSM-trained models are known to rely on obfuscated gradients to counter weak attacks, we augment it with JARN to study if there is additive robustness benefit against strong attacks. We also implemented double backpropagation (Drucker & Le Cun, 1991; Ross & Doshi-Velez, 2018) to compare.

Results Similar to results from the previous two datasets, the JARN classifier performs better than the standard baseline for all four types of adversarial examples. Compared to the model trained with uniform-noise augmentation, JARN performs closely in the weaker FGSM attack while being more robust against the two stronger PGD attacks. JARN also outperforms the double backpropagation baseline, showing that regularizing for salient Jacobians confers more robustness than regularizing for smaller Jacobian Frobenius norm values. The strong PGD-AT7 baseline shows higher robustness against PGD attacks than the JARN model. When we train JARN together with 1-step adversarial training (JARN-AT1), we find that the model's robustness exceeds that of strong PGD-AT7 baseline on all four adversarial attacks, suggesting JARN's gain in robustness is additive to that of AT. To study how JARN-AT1 robustness generalizes, we conduct PGD attacks of varying ε and strength (5, 10 and 20 iterations). We also include another PGD-AT7 baseline that was trained at a higher ε = (12/255). JARN-AT1 shows higher robustness than the two PGD-AT7 baselines against attacks with higher ε values (≤ 8/255) across the three PGD attacks, as shown in Figure 2 . We also observe that the PGD-AT7 variants outperform each other on attacks with ε values close to their training ε, suggesting that their robustness is more adapted to resist adversarial examples that they are trained on. This relates to findings by Tramèr & Boneh (2019) which shows that robustness from adversarial training is highest against the perturbation type that models are trained on. 

4.3.2. LOSS LANDSCAPE:

We compute the classification loss value along the adversarial perturbation's direction and a random orthogonal direction to analyze the loss landscape of the models. From Figure 3 , we see that the models trained by the standard and FGSM-AT method display loss surfaces that are jagged and nonlinear. This explains why the FGSM-AT display modest accuracy at the weaker FGSM attacks but fail at attacks with more iterations, a phenomenon called obfuscated gradients (Carlini & Wagner, 2017; Uesato et al., 2018) where the initial gradient steps are still trapped within the locality of the input but eventually escape with more iterations. The JARN model displays a loss landscape that is less steep compared to the standard and FGSM-AT models, marked by the much lower (1 order of magnitude) loss value in Figure 3c . When JARN is combined with one iteration of adversarial training, the JARN-AT1 model is observed to have much smoother loss landscapes, similar to that of the PGD-AT7 model, a strong baseline previously observed to be free of obfuscated gradients. This suggests that JARN and AT have additive benefits and JARN-AT1's adversarial robustness is not attributed to obfuscated gradients. A possible explanation behind the improved robustness through increasing Jacobian saliency is that the space of Jacobian shrinks under this regularization, i.e., Jacobians have to resemble non-noisy images. Intuitively, this means that there would be fewer paths for an adversarial example to reach an optimum in the loss landscape, improving the model's robustness. 

4.3.3. SALIENCY OF JACOBIAN:

The Jacobian matrices of JARN model and PGD-AT are salient and visually resemble the images more than those from the standard model (Figure 4 ). Upon closer inspection, the Jacobian matrices of the PGD-AT model concentrate their values at small regions around the object of interest whereas those of the JARN model cover a larger proportion of the images. One explanation is that the JARN model is trained to fool the discriminator network and hence generates Jacobian that contains details of input images to more closely resemble them.

4.3.4. COMPUTE TIME:

Training with JARN is computationally more efficient when compared to adversarial training (Table 4). Even when combined with FGSM adversarial training JARN, it takes less than half the time of 7-step PGD adversarial training while outperforming it in robustness. The performance of GANs in image generation has been well-known to be sensitive to training hyperparameters. We test JARN performance across a range of λ adv , batch size and discriminator update intervals that are different from § 4.3 and find that its performance is relatively stable across hyperparameter changes, as shown in Appendix Figure 5 . In a typical GAN framework, each training step involves a real image sample and an image generated from noise that is decoupled from the real sample. In contrast, a Jacobian is conditioned on its original input image and both are used in the same training step of JARN. This training step resembles that of VAE-GAN (Larsen et al., 2015) where pairs of real images and its reconstructed versions are used for training together, resulting in generally more stable gradients and convergence than GAN. We believe that this similarity favors JARN's stability over a wider range of hyperparameters.

4.3.6. BLACK-BOX TRANSFER ATTACKS:

Transfer attacks are adversarial examples generated from an alternative, substitute model and evaluated on the defense to test for gradient masking (Papernot et al., 2016; Carlini et al., 2019) . More specifically, defenses relying on gradient masking will display lower robustness towards transfer attacks than white-box attacks. When evaluated on such black-box attacks using adversarial examples generated from a PGD-AT7 trained model and their differently initialized versions, both JARN and JARN-AT1 display higher accuracy than when under white-box attacks (Table 5 ). This demonstrates that JARN's robustness does not rely on gradient masking. Rather unexpectedly, JARN performs better than JARN-AT1 under the PGD-AT7 transfer attacks, which we believe is attributed to its better performance on clean test samples. 

5. CONCLUSIONS:

In this paper, we show that training classifiers to give more salient input Jacobian matrices that resemble images can advance their robustness against adversarial examples. We achieve this through an adversarial regularization framework (JARN) that train the model's Jacobians to fool a discriminator network into classifying them as images. Through our experiments in three image datasets, JARN boosts adversarial robustness of standard models and give competitive performance when added on to weak defenses like FGSM. Our findings open the viability of improving the saliency of Jacobian as a new avenue to boost adversarial robustness.

A PROOF OF THEOREM 3.1

Theorem A.1. The global minimum of L adv is achieved when G(x) maps x to itself, i.e., G(x) = x.

Proof. From (Goodfellow et al., 2014) , for a fixed G, the optimal discriminator is

EQUATION

We can include the optimal discriminator into Equation ( 9) to get 

L adv (G) = E x∼pdata [log f * disc (x)] + E x∼pdata [log(1 -f * disc (G(x)))] = E x∼pdata [log f * disc (x)] + E x∼pG [log(1 -f * disc (x))] = E

:



Figures and Tables:

Figure 1: Training architecture of JARN.

Figure 2: Generalization of model robustness to PGD attacks of different ε values.

Figure 3: Loss surfaces of models along the adversarial perturbation and a random direction.

Figure 4: Jacobian matrices of CIFAR-10 models.

Figure 5: Accuracy of JARN with different hyperparameters on CIFAR-10 test samples.

• Through experiments in MNIST, SVHN and CIFAR-10, we find that JARN boosts adversarial robustness in image classifiers and provides additive robustness to adversarial training.

Table content: <table/>

uses a dual network to bound the adversarial perturbation with linear programming. The authors then optimize this bound during training to boost adversarial robustness.

Table content: <table/>

MNIST accuracy (%) on adversarial and clean test samples.Setup SVHN is a 10-class house number image classification dataset with 73257 training and 26032 test images, each of size 32 × 32 × 3. We train the Wide-Resnet model following hyperparameters from(Madry et al., 2017)'s setup for their CIFAR-10 experiments. For JARN, we use λ adv = 5, a discriminator network of 5 CNN layers (16-32-64-128-256 output channels) and update it for every 20 f cls training iterations. We evaluate trained models against adversarial examples with (ε = 8/255), crafted from FGSM and 5, 10, 20-iteration PGD attack.

Table content: <table><tr><td>Model</td><td colspan="4">FGSM PGD5 PGD40 Clean</td></tr><tr><td>Standard</td><td>76.5</td><td>0</td><td>0</td><td>98.7</td></tr><tr><td>Uniform Noise</td><td>77.5</td><td>0</td><td>0.02</td><td>98.7</td></tr><tr><td>JARN</td><td>98.4</td><td>98.1</td><td>98.1</td><td>98.8</td></tr><tr><td>4.2 SVHN</td><td/><td/><td/><td/></tr></table>

SVHN accuracy (%) on adversarial and clean test samples.Setup CIFAR-10 contains 32 × 32 × 3 colored images labeled as 10 classes, with 50k training and 10k test images. We train the Wide-Resnet model using similar hyperparameters to(Madry et al., 2017) for our experiments. Following the settings fromMadry et al. (2017), we compare with a strong adversarial training baseline (PGD-AT7) that involves training the model with adversarial examples generate with 7-iteration PGD attack. For JARN, we use λ adv = 1, a discriminator network of 5 CNN layers(32-64-128-256-512 output channels)  and update it for every 20 f cls training iterations. We evaluate trained models against adversarial examples with (ε = 8/255), crafted from

Table content: <table><tr><td>Model</td><td colspan="5">FGSM PGD5 PGD10 PGD20 Clean</td></tr><tr><td>Standard</td><td>64.4</td><td>26.0</td><td>5.47</td><td>1.96</td><td>94.7</td></tr><tr><td>Uniform Noise</td><td>65.0</td><td>42.6</td><td>18.4</td><td>9.21</td><td>95.3</td></tr><tr><td>JARN</td><td>67.2</td><td>57.5</td><td>37.7</td><td>26.79</td><td>94.9</td></tr></table>

CIFAR-10 accuracy (%) on adversarial and clean test samples.

Table content: <table><tr><td>Model</td><td colspan="5">FGSM PGD5 PGD10 PGD20 Clean</td></tr><tr><td>Standard</td><td>13.4</td><td>0</td><td>0</td><td>0</td><td>95.0</td></tr><tr><td>Uniform Noise</td><td>67.4</td><td>44.6</td><td>19.7</td><td>7.48</td><td>94.0</td></tr><tr><td>FGSM-AT1</td><td>94.5</td><td>0.25</td><td>0.02</td><td>0.01</td><td>91.7</td></tr><tr><td>Double Backprop</td><td>28.3</td><td>0.05</td><td>0</td><td>0</td><td>95.7</td></tr><tr><td>JARN</td><td>67.2</td><td>50.0</td><td>27.6</td><td>15.5</td><td>93.9</td></tr><tr><td>PGD-AT7</td><td>56.2</td><td>55.5</td><td>47.3</td><td>45.9</td><td>87.3</td></tr><tr><td>JARN-AT1</td><td>65.7</td><td>60.1</td><td>51.8</td><td>46.7</td><td>84.8</td></tr><tr><td>4.3.1 GENERALIZATION</td><td/><td/><td/><td/><td/></tr></table>

Average wall-clock time per training epoch for CIFAR-10 adversarial defenses.

Table content: <table><tr><td>Model</td><td colspan="4">PGD-AT7 JARN-AT1 FGSM-AT1 JARN only</td></tr><tr><td>Time (sec)</td><td>704</td><td>294</td><td>267</td><td>217</td></tr><tr><td colspan="3">4.3.5 SENSITIVITY TO HYPERPARAMETERS</td><td/><td/></tr></table>

CIFAR-10 accuracy (%) on transfer attack where adversarial examples are generated from a PGD-AT7 trained model.

Table content: <table><tr><td>Model</td><td colspan="6">PGD-AT7 FGSM PGD20 FGSM PGD20 FGSM PGD20 Same Model White-box</td><td>Clean</td></tr><tr><td>JARN</td><td>79.6</td><td>76.7</td><td>73.6</td><td>17.4</td><td>67.2</td><td>15.5</td><td>93.9</td></tr><tr><td>JARN-AT1</td><td>66.4</td><td>63.0</td><td>70.3</td><td>59.3</td><td>65.7</td><td>46.7</td><td>84.8</td></tr></table>

x∼pdata log p data (x) p data (x) + p G (x) + E x∼pG log p G (x) p data (x) + p G (x) = E x∼pdata log p data (x)

Table content: <table><tr><td/><td colspan="2">1 2 (p data (x) + p G (x))</td><td colspan="2">+ E x∼pG log</td><td>p G (x) 2 (p data (x) + p G (x)) 1</td><td>-2 log 2</td></tr><tr><td>= KL p data</td><td>p data + p G 2</td><td colspan="2">+ KL p G</td><td>p data + p G 2</td><td>-log 4</td></tr></table>

Peer review (iclr2020):
I think the main contribution of this paper is that it introduces a new way of robust training by encouraging Jacobian saliency. Previous research like Etmann et al 2019 and Tsipras et al 2018 showed that robustness leads to saliency. But surprisingly, this paper shows the other way, saliency map can also lead to robustness, which indicates a stronger connection between these two. In general, I like the intuition behind this paper, since it introduces a new perspective of robust training. The training method proposed in this paper is still kind of preliminary, though. I suspect that training a GAN together with the classifier will cost even more time than min-max adversarial training or some certified robust training methods. It would be great if the authors can provide the training time comparison between JARN and some state-of-the-art robust training methods. Another concern is reproducibility since the training process of GAN is sensitive to hyperparameter selection. It would be better if the author can have some discussion on the training process to show that the reported performance of the defense is easy to reproduce instead of cherry-picking. Also, there are too many works on robustness defense that have been proven ineffective (consider the works by Carlini). Since this is a completely new way of robust training and there is no certified guarantee, I suggest the authors refer [1] to evaluate the effectiveness of the defense more thoroughly to convince the readers that it really works. Especially, evaluation under adaptive attack is necessary. I think this is a very interesting work. But since this method is completely new, more detailed information is needed to convince me that it really works. If it does work, I believe there must exist better ways to encourage Jacobian saliency than using a GAN. [1] Carlini, Nicholas, et al. *On evaluating adversarial robustness.* arXiv preprint arXiv:1902.06705 (2019).