Title: ADVERSARIALLY ROBUST TRAINING THROUGH STRUCTURED GRADIENT REGULARIZATION

Abstract: We propose a novel data-dependent structured gradient regularizer to increase the robustness of neural networks vis-a-vis adversarial perturbations. Our regularizer can be derived as a controlled approximation from first principles, leveraging the fundamental link between training with noise and regularization. It adds very little computational overhead during learning and is simple to implement generically in standard deep learning frameworks. Our experiments provide strong evidence that structured gradient regularization can act as an effective first line of defense against attacks based on long-range correlated signal corruptions. 1

1. INTRODUCTION:

Deep neural networks, in particular convolutional neural networks (CNNs), have been used with great success for perceptual tasks such as image classification (Simonyan & Zisserman, 2014) or speech recognition (Hinton et al., 2012) . However, it has been shown that the accuracy of models obtained by standard training methods can dramatically deteriorate in the face of so-called adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014) , i.e. small perturbations in the input signal, typically imperceptible to humans, that are sufficient to induce large changes in the output. This apparent vulnerability is worrisome as CNNs start to proliferate in the real-world, including in safety-critical deployments.

Although the theoretical aspects of vulnerability to adversarial perturbations are not yet fully understood, a plethora of methods has been proposed to find adversarial examples. These often transfer or generalize across different architectures and datasets (Papernot et al., 2016a; Liu et al., 2016; Tramèr et al., 2017) , enabling black-box attacks even for inaccessible models. The most direct and commonly used strategy of protection is to use adversarial examples during training, see e.g. (Goodfellow et al., 2014) or (Miyato et al., 2015) . This raises the question, of whether one can generalize across such examples in order to become immune to a wider range of possible adversarial perturbations. Otherwise there appears to be a danger of overfitting to a specific attack, in particular if adversarial examples are sparsely generated from the training set.

In this paper, we pursue a different route, starting from the hypothesis that the weakness revealed by adversarial examples first and foremost hints at a data scarcity problem. Inspired by the classical work of Bishop (1995) and similar in spirit to Miyato et al. (2017) , we thus propose regularization as the preferred remedy to increase model robustness. Our regularization approach comes without rigid a priori assumptions on the model structure and filters (Bruna & Mallat, 2013) as well as strong (and often overly drastic) requirements such as layer-wise Lipschitz bounds (Cisse et al., 2017) or isotropic smoothness (Miyato et al., 2017) . Instead, we will rely on the ease of generating adversarial examples to learn an informative regularizer, focusing on the correlation structure of the adversarial noise. Our regularizer consistently outperforms structure-agnostic baselines on long-range correlated perturbations -achieving over 25% higher classification accuracy. We thus consider it an effective first line of defense on top of which other defense mechanisms may be built.

In summary, we make the following contributions:

• We provide evidence that current adversarial attacks act by perturbing the short-range covariance structure of signals.

• We propose Structured Gradient Regularization (SGR), a data-dependent regularizer informed by the covariance structure of adversarial perturbations. • We present an efficient SGR implementation with low memory footprint.

• We introduce a long-range correlated adversarial attack against which SGR-regularized classifiers are shown to be robust.

2. ADVERSARIAL ROBUST LEARNING:



2.1. VIRTUAL EXAMPLES:

Imagine we had access to a set of transformations τ ∈ T of inputs that leave outputs invariant. Then we could augment our training data by expanding each real example

(x i , y i ) into virtual examples (τ (x i ), y i ).

This approach is simple and widely-applicable, e.g. for exploiting known data invariances (Schölkopf et al., 1996) , and can greatly increase the statistical power of learning. A similar stochastic method has been suggested by Maaten et al. (2013) : a noisy channel or perturbation Q corrupts x → x with probability Q(x|x), leading to an altered (more precisely: convolved) joint distribution

P Q (x, y) = P (y) P (x|y)Q(x|x)dx.

Where could such a corruption model come from? We propose to use a generative mechanism to learn adversarial corruptions from examples. In fact, we will use a simple additive noise model in this paper,

EQUATION

Our focus will be on leveraging the correlation structure of the noise, captured by the matrix of raw second moments Σ, aggregated as a running exponentially weighted batch estimate computed from adversarial perturbations ξ, obtained by methods such as FGSM (Goodfellow et al., 2014) , PGD (Madry et al., 2017) or DeepFool (Moosavi Dezfooli et al., 2016) .

2.2. ROBUST MULTICLASS LEARNING OBJECTIVE:

Once we have an estimate of Q available, we propose to optimize any standard training criterion (e.g. a likelihood function) in expectation over a mixing distribution between the uncorrupted and the corrupted data. For the sake of concreteness, let us focus on the multiclass case, where y ∈ {1, . . . , K}. Define a probabilistic classifier

EQUATION

For a given loss function such as the negative log-loss, (y, π) = -log π y , our aim is to minimize

EQUATION

where P denotes the empirical distribution and PQ the noise corrupted distribution. The hyperparameter λ gives us a degree of freedom to pay more attention to the training data vs. our (imperfect) corruption model.

2.3. FROM VIRTUAL EXAMPLES TO REGULARIZATION:

Approaches relying on the explicit generation of adversarial examples face three statistical challenges: (i) How can enough diverse samples be generated to cover all attacks? (ii) How can we avoid a computational blow-up by adding too many virtual examples? (iii) How can we prevent overfitting to a specific attack? A remedy to these problems is through the use of regularization. The basic idea is simple: instead of sampling virtual examples, one tries to calculate the corresponding integrals in closed form, at least under reasonable approximations.

The general connection between regularization and training with noise-corrupted examples has been firmly established by Bishop (1995) . The benefits of regularization over adversarially augmented training in terms of generalization error have been hypothesized by Galloway et al. (2018) . In (Maaten et al., 2013) , properties of the loss function are exploited to achieve efficient regularization techniques for certain types of noise. Here, we follow the approach pursued in (Roth et al., 2017) in using an approximation that is accurate for small noise amplitudes.

3. STRUCTURED GRADIENT REGULARIZATION:



3.1. FROM CORRELATED NOISE TO STRUCTURED GRADIENT REGULARIZATION:

To approximate the expectation with regard to the noisy channel, we generalize a recent approach for gradient regularization of Generative Adversarial Networks (GANs) (Roth et al., 2017) , from binary classification to multiclass classification and from white noise to (adversarially) structured noise.

For the sake of simplicity of derivations we assume adversarial perturbations to be centered. Let us Taylor expand the log-component functions, making use of the shortcut ψ y := log φ y ,

ψ y (• + ξ) = ψ y + ∇ψ y ξ + 1 2 ξ [ψ y ]ξ + O( ξ 3 ) (4)

For any zero-mean distribution Q with second moments Σ,

EQUATION

)

The Hessian is calculated via the chain rule,

EQUATION

We will make a similar argument to (Roth et al., 2017) to show that it is reasonable to neglect the terms involving Hessians φ y . Let us therefore consider the Bayes-optimal classifier φ * y , for which up to a normalization constant φ * y (x) ∝ P (x|y)P (y) ,

such that

EQUATION

Exchanging summation and differentiation, we however have as a consequence of the normalization

EQUATION

Thus, under the assumption that φ ≈ φ * and of small perturbations (such that we can ignore higher order terms in Eq. ( 4)), we get

EQUATION

We can turn this into a regularizer by taking the leading terms (Roth et al., 2017) . Identifying P = P with the empirical distribution, we arrive at the following robust learning objective:

Structured Gradient Regularization (SGR) min φ → L SGR (φ) = E P [-log φ y (x)] + λΩ Σ (φ) (11) Ω Σ (φ) := 1 2 E P ∇ x log φ y (x) Σ ∇ x log φ y (x)

For uncentered adversarial perturbations E Q [ξ] = µ it is easy to see that the corresponding regularizer is given by

EQUATION

In practice, we however observed this correction to be small.

Algorithm 1 Adversarial SGR. Default values: λ ∈ [0, 1], β = 0.1

Input: Regularization strength λ, exponential decay rate β, batch size m Initial classifier parameters (weights and biases) ω 0 while ω t not converged do Sample minibatch of data {(x (1) , y (1) ), ..., (x (m) , y (m) )} ∼ P. Compute (normalized) adversarial perturbations ξ (1) , ..., ξ (m) Compute covariance matrix or covariance function Cov(ξ (1) , ...,

ξ (m) ) Update running average Σt ← (1-β) Σt-1 + β Cov(ξ (1) , ..., ξ (m) ) L(ω) = 1 m m i=1 K k=1 -y (i) k log φ k (x (i) ; ω) Ω Σt (ω) = 1 2m m i=1 K k=1 y (i) k ∇ x log φ k (x (i) ; ω) Σt ∇ x log φ k (x (i) ; ω) ω t ← ω t-1 -∇ ω L(ω) + λ Ω Σt (ω) ωt-1

return ω t The gradient-descent step can be performed with any learning rule. We used Adam in our experiments. See Sec. 3.3 for an efficient implementation of the regularizer.

3.2. PROPERTIES:

There are a few facts that are important to point out about the derived SGR regularizer.

(i) As the regularizer provides an efficiently computable approximation for an intractable expectation, it is clearly data-dependent. Ω Σ penalizes loss-gradients evaluated at training points. This is different from standard regularizers that penalize some norm of the parameter vector, such as L 2 -regularization. In the latter case, we would expect the regularizer to have the largest effect in the empty parts of the input space, where it should reduce the variability of the classifier outputs. On the contrary, Ω Σ has its main effect around the data manifold. In this sense, it is complementary to parameter regularization.

(ii) The SGR regularizer is intrinsic in the sense that it does not depend on the parameters of the classifier (for a good reason, we have not mentioned any parameterization), but instead directly acts on the function realized by the classifier φ (the relevant gradients measure the sensitivity of φ with regard to the input x and not with regard to parameters). Thus, it is parameterization invariant and can be naturally applied to any function space, whether it is finite-dimensional or not.

(iii) We can gain complementary insights into SGR by explicitly computing it in terms of classifier logits ϕ y (x). As outlined in Sec 7.1 in the Appendix, we obtain the following expression for the structured gradient regularizer

EQUATION

We can therefore see that SGR is penalizing large variations of the class-conditional logit-gradients ∇ϕ y around their data-dependent average ∇ϕ . For simple one-layer softmax classifiers we obtain

Ω Σ (φ) = 1/2 E P (ω y -ω ) Σ (ω y -ω ) .

This suggests an intriguing connection to variancebased regularization. Weight-decay regularization, on the other hand, simply penalizes large norms.

3.3. IMPLEMENTATION:

The implementation of SGR training is outlined in Algorithm 1. The matrix of raw second moments is aggregated through a running exponentially weighted average

EQUATION

with the decay rate β as a tuneable parameter trading off weighting between current (β → 1) and past (β → 0) batch averages. For low-dimensional data sets, we can directly compute the full matrix

EQUATION

For high-dimensional data sets, we have to resort to more memory-efficient representations leveraging the sparseness of the covariance matrix and covariance-gradient matrix-vector product. For image data sets, the covariance matrix can be estimated as a function of the displacement between pixels, as illustrated in Fig. 2 ,

EQUATION

)

where i = (i r , i c ) denotes the 2D pixel location and i is the covariance matrix index obtained by flattening the 2D pixel location into a 1D array such that i r and i c denote the row and column numbers of pixel i, and similar for j respectively. Rounding pixel displacements ||i -j|| 2 to the nearest integer, the covariance function simply becomes an array of real numbers storing the current estimate of the covariance between two pixels displaced by the difference in array indices.

In our experiments, we normalize the perturbations such that the average diagonal entry of the covariance matrix is one, thus allowing for more intuitive optimization over the regularization strength parameter λ. We note also that it is not necessary to update Σ at every training step and it is even possible to train with a fixed "expert"-designed covariance matrix.

The other crucial quantities needed to evaluate the regularizer are log φ k (•) for training points (x (i) , y (i) ). This is simply the per-sample cross-entropy loss, which is often available as a highly optimized callable operation in modern deep learning frameworks, reducing the implementation of the SGR regularizer to merely a few lines of code at very little computational overhead.

4. EXPERIMENTS:



4.1. EXPERIMENTAL SETUP:

Classifier architectures & data preprocessing. We trained Convolutional Neural Networks (CNNs) with seven hidden layers (and different numbers of filters for each data set), on CIFAR10 (Krizhevsky & Hinton, 2009) and MNIST (LeCun, 1998) . Our models are identical to those used in (Carlini & Wagner, 2017; Papernot et al., 2016b) . For both data sets, we adopt the following standard preprocessing and data augmentation scheme (He et al., 2016) : each training image is zero-padded with four pixels on each side2 , randomly cropped to produce a new image with the original dimensions and horizontally flipped with probability one half3 . We also standardize each image to have zero mean and unit variance when passing it to the classifier. Further details can be found in the Appendix. Adversarial attacks. We evaluate the classification accuracy against several adversarial attacks: Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) , Projected Gradient Descent (PGD) (Madry et al., 2017) and DeepFool (Moosavi Dezfooli et al., 2016) . The attack strength is reported in units of 1/255. The SNR reported for the DeepFool attack is computed according to Eq. ( 2) in (Moosavi Dezfooli et al., 2016) . The attacks were implemented with the open source CleverHans Library (Papernot et al., 2017) . Further details and attack hyper-parameters can be found in Sec. 7.2 in the Appendix. Figure 2 : PGD covariance functions: (Left) intra-channel covariance functions, measuring correlations between identical color channels, (the coloring in the plot matches the corresponding channel), (right) inter-channel covariance functions, measuring correlations between distinct color channels.

The rapid decay with pixel displacement, indicating very short decay length, is clearly visible.

4.2. COVARIANCE STRUCTURE OF ADVERSARIAL PERTURBATIONS:

To investigate the covariance structure of adversarial perturbations, we trained a clean classifier for 50 epochs and computed adversarial perturbations for every data point in the test set. The perturbations were flattened into 1D arrays before computing the full covariance matrix4 as in Eq. ( 15).

The short-range correlation structure of the perturbations, shown in Fig. 1 , is clearly visible: the correlations decay much faster than those of the data set covariance matrix. Figure 2 shows the corresponding PGD covariance functions. The rapid decay with pixel displacement, indicating very short decay length5 , is again evident. It thus seems that an unregularized classifier vulnerable to adversarial perturbations gives too much weight to short-range correlations (low-level patterns) and not enough weight to long-range ones (globally relevant high-level features of the data).

4.3. LONG-RANGE CORRELATED NOISE ATTACK:

To investigate the effect and potential benefit of using a structured covariance matrix in the SGR regularizer versus an "unstructured" diagonal covariance, corresponding to gradient-norm regularization, we compute the classification accuracy against a worst-of-100 attack in which the adversary samples perturbations from a long-range correlated multivariate Gaussian with covariance matrix specified through an exponentially decaying covariance function parametrized by a variable decay length ζ.

Inspired by the PGD covariance function, we chose intra-channel CovFun f (r) = exp(-r/ζ) and inter-channel CovFun f (r) = 0.5 exp(-r/ζ). The corresponding covariance matrices are depicted in Fig. 5 in the Appendix. LRC samples are shown in Fig. 3 .

The worst-of-100 attack then consists in perturbing every test set data point 100 times and evaluating the classifier accuracy against the worst of these perturbations. We tested for several decay lengths in the range ζ ∈ [1, 2, 4, 8, 16] and attack strengths ∈ [0.01, 0.7] with which the perturbations were scaled. Figure 4 in the Appendix shows the accuracy of different models as a function of . As a baseline, we also trained a model on LRC-augmented input. For SGR and GN we performed a hyper-parameter search over λ (at fixed ζ = 8) and report results for the best performing model. The LRC attack is a natural prototype for low frequency perturbations, as opposed to existing attacks which we have shown to mainly corrupt the short range (high frequency) structure of signals.

As shown in Fig. 3 (left), the low frequency range is precisely the regime where the benefit of structured regularization comes into play: SGR clearly and consistently outperforms GN on longrange correlated perturbations, achieving over 25% higher classification accuracy on LRC-perturbed input with ζ ∼ 16. As the decay length goes to zero, the synthetic covariance matrix converges to the identity matrix and SGR performance approaches GN performance.

Finally, to verify that those effects are not a result of our synthetically chosen covariance function, we also performed the same experiments with the covariance matrix computed from the CIFAR10 training set, shown in Fig. 1 . The accuracies for = 0.3 are: SGR 65.8%, GN 48.7%, LRC noise 57.9% and clean 46.5%, again confirming the benefit of using a structured covariance in the regularizer.

4.4. WHITE-BOX & TRANSFER ATTACK ACCURACY:

A summary of the white-box and transfer attack accuracies can be found in Table 1 . We train all models to achieve the highest possible white-box / transfer attack accuracies, averaged over in the range [0, 32] for the MNIST and [0, 8] for the CIFAR10 test set, i.e. the best performing models were selected according to the integrated area under the attack curve. The corresponding hyper-parameters are listed in Sec. 7.2 in the Appendix.

On MNIST, SGR/GN achieve PGD white-box accuracies of 96.5% compared to the 78.3% accuracy achieved by the clean model. On CIFAR10, SGR/GN trained models achieve an accuracy of 53.4% against targeted T-PGD attack (target is selected uniformly at random), whereas the clean model achieves 24.9% and PGD-trained model achieves 69.7% accuracy. SGR/GN trained models thus achieve white-box accuracies that are intermediate between those of the clean and adversarially trained models. The higher white-box accuracies achieved by adversarially trained models, however comes at the expense of a drop in test set accuracies. On CIFAR10, the test set accuracy of the PDG-trained model drops to 77.7%, while SGR/GN achieve 83.4% and 83.7% respectively.

An interesting observation can be made from the transfer attack accuracies (bold-face entries) on CIFAR10. As can be seen, the PGD adversarial perturbations obtained from SGR/GN trained models hurt the clean model more (60.3%) than the PGD perturbations obtained from PGD/FGSM adversarially trained models (77.0%/81.9%). Moreover, SGR achieves significantly higher accuracies against PGD transfer attacks derived from the clean model (80.6%) than the clean model does on PGD perturbations obtained from the SGR-trained model (60.3%), while accuracies achieved by SGR models on PDG samples from PGD-trained models (75.8%) are comparable to those in the other direction (74.0%).

As expected, the difference between SGR and GN in terms of white-box and transfer attack accuracies is less significant in this experiment, which can be explained by the evident short-range nature of current adversarial attacks (see discussion in Sec. 4.2). Reassuringly, we did not observe any loss of performance in all our experiments when using the covariance function instead of the full covariance matrix in the SGR regularizer.

5. RELATED WORK:

A large body of related work on distributionally robust optimization (DRO) (Sinha et al., 2017; Gao & Kleywegt, 2016) seeks a classifier that minimizes the worst-case loss against an adversary that can shift the entire training set within an uncertainty ball around the empirical distribution. Adversarial training can be viewed as a special case of DRO against a pointwise adversary that independently perturbs each example. While these methods aim at identifying the worst-case distribution, we propose to efficiently approximate expectations over corrupted distributions through structure-informed regularization.

Coincidentally, Ross & Doshi-Velez (2017) and Simon-Gabriel et al. (2018) also suggested to use gradient-norm regularization to improve the robustness against adversarial attacks. While (Simon-Gabriel et al., 2018) investigated the effect of different gradient-norms induced by constraints on the magnitude of the perturbations, we focused on a structured generalization of gradient regularization.

Similar in spirit to our work is also the recent work on adversarial training vs. weight decay regularization (Galloway et al., 2018) . While weight decay acts on the parameters, our regularizer acts on the function realized by the classifier. As discussed in Sec. 3.2, the two approaches are complementary to each other and can also be combined.

From a Bayesian perspective, regularization can be understood as imposing a prior distribution over the model parameters. As parameters in neural networks are non-identifiable, we ultimately prefer to impose priors in the space of functions directly (Teh, 2017; Neal, 2012; 1996) . As pointed out in Sec. 3.2, this is indeed one of the key properties of our regularizer.

6. CONCLUSION:

The fact that adversarial perturbations can fool classifiers while being imperceptible to the human eye, hints at a fundamental mismatch between how humans and classifiers try to make sense of the data they see. We provided evidence that current adversarial attacks act by perturbing the short-range correlations of signals. We proposed a novel structured gradient regularizer (SGR) informed by the covariance structure of adversarial noise and presented an efficient SGR implementation with low memory footprint. Devising further (e.g. gradient-based) low frequency attacks is an interesting direction of future research.

7. APPENDIX:

7.1 SGR PROPERTIES (iii) We can gain complementary insights into SGR by explicitly computing it in terms of logits

EQUATION

Defining the class average of the logit-gradients as

EQUATION

we obtain the following expression for the structured gradient regularizer

EQUATION

We can therefore see that SGR is penalizing large fluctuations of the class-conditional logit-gradients ∇ϕ y around their data-dependent class average ∇ϕ . For simple one-layer softmax classifiers ϕ y (x) = ω y x + b y , we obtain

EQUATION

This suggests an intriguing connection to variance-based regularization. Weight-decay regularization, on the other hand, simply penalizes large norms. An overview of the attack hyper-parameters can be found in Table 2 . Untargeted attacks find adversarial perturbations to an arbitrary one of the K -1 other classes. Both PGD and DeepFool are iterative attacks: they use 10 / 40 (for the white-box / transfer attack tables) and 100 iterations respectively. The numbers reported for the DeepFool attack are computed according to Eq. ( 2) in (Moosavi Dezfooli et al., 2016) . The adversarial example construction processes use the most likely label predicted by the classifier in order to avoid label leaking (Kurakin et al., 2016) during adversarially augmented training. The perturbed images returned by the attacks were clipped to lie within the original rgb range, as is commonly done in practice. The attacks were implemented with the open source CleverHans Library (Papernot et al., 2017) .

For SGR/GN we performed a hyper-parameter search over the regularization strength λ and SGR decay parameter β. 

7.3. FURTHER EXPERIMENTAL RESULTS:



Smoothness of Regularized Classifier:

At the most basic level of abstraction, adversarial vulnerability is related to an overly-steep classifier gradient, such that a tiny change in the input can lead to a large perturbation of the classifier output. In order to analyze this effect, we visualize the softmax activations along linear trajectories between various clean and corresponding adversarial examples.

As can be seen in Fig. 6 , the regularized classifier consistently leads to smooth classifier outputs.

Label leaking. Label leaking can cause models trained with adversarial examples to perform better on perturbed than on clean inputs, as the model can learn to exploit regularities in the adversarial example construction process (Kurakin et al., 2016) . This hints at a danger of overfitting to the specific attack when training with adversarial examples. SGR regularized classifiers, on the other hand, do not suffer from label leaking, as the adversarial examples are never directly fed to the classifier.

7.4. CLASSIFIER ARCHITECTURES:

Networks were trained for 50/75 epochs on MNIST/CIFAR10 using the Adam optimizer (Kingma & Ba, 2014) with learning rate 0.001 and minibatch size 128.

:

Code will be made available

We shifted the CIFAR10 data by -0.5 so that padding adds mid rgb-range pixels.

We do not flip MNIST digits.

Strictly speaking, Σ denotes the matrix of raw second moments. We will refer to Σ as the covariance matrix however, since the difference between the two is negligible.

Decay length is defined as the displacement over which the covariance function decays to 1/e of its value.

Figures and Tables:

Training methods. We train each classifier with a number of different training methods: (i) clean, i.e. through the standard softmax cross-entropy objective, (ii) clean + L 2 weight decay regularization, (iii) adversarially augmented training, i.e. training the classifier on a mixture of clean and adversarial examples (iv) GN (gradient-norm) regularized (corresponding to SGR with identity covariance matrix) and (v) SGR regularized. The SGR covariance matrix was computed from either: (a) loss gradients, (b) sign of loss gradients (corresponding to FGSM) or (c) PGD perturbations. The performance of these SGR variants was identical in all our experiments, hence we collectively report them as SGR.

Figure 1: Covariance matrices of PGD, FGSM and DeepFool perturbations as well as CIFAR10training set (for comparison). The short-range structure of the perturbations is clearly visible. It is also apparent that the first two attack methods yield perturbations with almost identical covariance structure. We are showing a center-crop for better visibility (25% trimmed on each side). For comparison the matrices were rescaled such that their maximum element has an absolute value of one.

Figure 3: (Left) Accuracy of different models as a function of the LRC attack decay length ζ ∈ [1, 16] at fixed = 0.3 (averaged over five runs). The plot clearly demonstrates the superiority of using a structured covariance matrix in the SGR regularizer. (Right) LRC perturbed samples with increasing decay lengths ζ ∈ [1, 2, 4, 8, 12] (two examples per decay length). The top row shows original test set images, the middle row shows adversarially perturbed samples, while the bottom row shows the adversarial perturbations rescaled to fit the full range of rgb values (their actual range and hence saliency is smaller). The footer indicates the classifier predictions on adversarial input.

For adversarial training, we trained models with in the full range ∈ [0, 32] for MNIST and ∈ [0, 8] for CIFAR10 and report results for the best performing one. The hyperparameters of the best performing models are: CIFAR10: SGR λ = 0.005, β = 0.1, GN γ = 0.025, PGD = 8, FGSM = 4, weight-decay 0.005. MNIST: SGR λ = 3.0, β = 0.1, GN γ = 10.0, PGD = 32, FGSM = 32, weight-decay 0.005.

Figure 4: Long Range Correlated Noise. Classification accuracy of different models as a function of the attack strength for an LRC attack with a fixed decay length ζ = 8 on CIFAR10. The SGR/GN hyper-parameters are reported in Sec. 4.3. See Fig. 3 for a plot of the accuracy of different models as a function of the LRC attack decay length ζ ∈ [1, 16] at fixed = 0.3

Test set, white-box and PGD-transfer attack accuracies. Each row corresponds to a model trained with a different training method, evaluated against the respective white-box (non-bold entries) and PGD-transfer attacks (bold-face entries) indicated above each column (T-PGD refers to targeted attack). The white-box and transfer attack accuracies are averaged over attack strengths in the range ∈ [0, 32] for MNIST and ∈ [0, 8] for CIFAR10, i.e. the reported accuracies represent the integrated area under the attack curve. For the transfer attack accuracies of a training method against itself, we trained a separate model with the same hyper-parameters to obtain the perturbations.

Table content: <table><tr><td/><td/><td/><td/><td colspan="2">MNIST = 32</td><td/><td/><td/><td/><td/></tr><tr><td>TRAIN. M.</td><td>TEST</td><td>RAND</td><td>PGD</td><td colspan="2">T-PGD CLEAN</td><td>PGD</td><td>FGSM</td><td>GN</td><td>SGR</td><td>FOOL</td></tr><tr><td>CLEAN</td><td>99.4</td><td>99.3</td><td>78.3</td><td>74.1</td><td>98.3</td><td>98.9</td><td>98.9</td><td>97.9</td><td>98.3</td><td>0.144</td></tr><tr><td>WDECAY</td><td>98.8</td><td>98.7</td><td>92.5</td><td>95.0</td><td>98.4</td><td>98.4</td><td>98.4</td><td>96.9</td><td>97.6</td><td>0.229</td></tr><tr><td>PGD</td><td>99.5</td><td>99.5</td><td>99.1</td><td>99.3</td><td>99.4</td><td>99.3</td><td>99.4</td><td>99.3</td><td>99.3</td><td>0.453</td></tr><tr><td>FGSM</td><td>99.4</td><td>99.4</td><td>98.5</td><td>98.0</td><td>99.3</td><td>99.2</td><td>99.2</td><td>99.2</td><td>99.2</td><td>0.339</td></tr><tr><td>GN</td><td>99.5</td><td>99.5</td><td>96.5</td><td>97.6</td><td>99.3</td><td>99.2</td><td>99.2</td><td>98.0</td><td>98.4</td><td>0.309</td></tr><tr><td>SGR</td><td>99.6</td><td>99.5</td><td>96.6</td><td>96.5</td><td>99.4</td><td>99.3</td><td>99.4</td><td>98.5</td><td>98.7</td><td>0.303</td></tr><tr><td/><td/><td/><td/><td colspan="2">CIFAR10 = 8</td><td/><td/><td/><td/><td/></tr><tr><td>TRAIN. M.</td><td>TEST</td><td>RAND</td><td>PGD</td><td colspan="2">T-PGD CLEAN</td><td>PGD</td><td>FGSM</td><td>GN</td><td>SGR</td><td>FOOL</td></tr><tr><td>CLEAN</td><td>84.7</td><td>83.2</td><td>28.8</td><td>24.9</td><td>72.2</td><td>77.0</td><td>81.9</td><td>61.9</td><td>60.3</td><td>0.010</td></tr><tr><td>WDECAY</td><td>81.4</td><td>79.2</td><td>27.1</td><td>32.3</td><td>69.7</td><td>71.7</td><td>68.9</td><td>54.1</td><td>52.9</td><td>0.015</td></tr><tr><td>PGD</td><td>77.7</td><td>77.7</td><td>62.4</td><td>69.7</td><td>77.1</td><td>71.5</td><td>76.2</td><td>74.0</td><td>74.0</td><td>0.061</td></tr><tr><td>FGSM</td><td>81.9</td><td>82.0</td><td>55.2</td><td>66.4</td><td>81.1</td><td>75.0</td><td>78.0</td><td>76.5</td><td>76.4</td><td>0.046</td></tr><tr><td>GN</td><td>83.7</td><td>83.7</td><td>41.9</td><td>54.1</td><td>81.6</td><td>76.9</td><td>82.0</td><td>68.1</td><td>67.9</td><td>0.039</td></tr><tr><td>SGR</td><td>83.4</td><td>82.8</td><td>41.5</td><td>53.4</td><td>80.6</td><td>75.8</td><td>81.2</td><td>66.8</td><td>65.5</td><td>0.039</td></tr></table>

Attack hyper-parameters. wb stands for white-box, t for transfer attacks.

Table content: <table><tr><td>7.2 HYPER-PARAMETERS</td><td/></tr><tr><td>Attack</td><td>Hyper-parameters</td></tr><tr><td>FGSM</td><td>: in units of 1/255</td></tr><tr><td>PGD</td><td>: in units of 1/255</td></tr><tr><td/><td>iter = /5</td></tr><tr><td/><td>nb_iter = 10 (wb) , 40 (t)</td></tr><tr><td>DEEP FOOL</td><td>overshoot : 0.02</td></tr><tr><td/><td>max_iter= 100</td></tr></table>

Peer review (iclr2019):
 Summary of the paper: This paper proposes to use structured gradient regularization to increase adversarial robustness of neural network. Here, the gradient regularization is to regularize some norm of the gradients on neural network input. "structured" means that instead of just minimizing the L2 norm of the gradients, a "mahalanobis norm" is minimized. The covariance matrix is updated continuously to track the "structure" of gradients/perturbations. Whitebox attack and blackbox attack The paper is well written, both theory and experiments are well explained. The analysis of LRC attack on SGR trained models are interesting. However, I believe the paper has major flaws in several aspects. The whitebox robustness evaluation is weak. Whitebox PGD with 10 iterations is not enough for discovering true robustness of a neural network, which makes the experiments unconvincing. PGD with 100 iterations and 50 random starts would make the evaluation much convincing wrt to whitebox attack. https://github.com/MadryLab/mnist_challenge I noticed that in Table 1, the authors reported averaged results across different epsilons. Although I see the motivation to give equal weights to small and large perturbations, it makes it hard to compare with previous papers. I think the authors should a least report commonly used eps in the literature, including MNIST eps=0.1, 0.2, 0.3 and CIFAR10 eps=8/255. Currently, for MNIST eps=32/255=0.125 is much below the standard eps for benchmarking MNIST. In my opinion, when evaluating robust optimization / gradient regularization methods, robustness under the strongest whitebox should be the major benchmark. Because "intrinsic" robustness is their goal. In contrast, black-box results are less important. This is because 1) evaluating black-box robustness on a few attacks hardly give any conclusive statements; 2) if we*re pursuing black-box robustness, there*re many randomization methods that boosts black-box robustness under various settings. How does a gradient regularization method help on top of those should be at least evaluated. So if the paper wants to claim black-box robustness, it needs at least include experiments like 2), so it provides useful benchmarks to practitioners. There*re also a few problems in the motivation / analysis. """A remedy to these problems is through the use of regularization. The basic idea is simple: instead of sampling virtual examples, one tries to calculate the corresponding integrals in closed form, at least under reasonable approximations.""" The adversarial robustness problem is not about integral over a neighborhood, it is about the maximum loss over a neighborhood. This is likely why previous attempts on gradient regularization and adversarial training on FGSM attack fails. And the success is of PGD training is largely due to that the loss minimize over the adversarial example that gives the maximum loss. """Thus, under the assumption that .phi .approx .phi^* and of small perturbations (such that we can ignore higher order terms.""" The Bayes optimal assumption seems to be arbitrary to me. If .phi is nearly Bayes-optimal, why would we worry about adversarial examples? Other relatively minor problems In the caption of Figure 1, """Covariance matrices of PGD, FGSM and DeepFool perturbations as well as CIFAR10 training set (for comparison). The short-range structure of the perturbations is clearly visible. It is also apparent that the first two attack methods yield perturbations with almost identical covariance structure.""" PGD and FGSM have very different attack power. If they are similar by any measure, wouldn*t that mean the measure (covariance structure) is too coarse? In Section 3.1, the paper talks about both centered and uncentered adversarial examples. I assumed that the authors mean that the distribution of perturbations are centered? First, I think this the authors should make this more explicit. Second, I think this is not a realistic to assume the perturbations to be centered, because for image data, the epsilon-ball usually intersects with data domain boundary. So I*m wondering in the experiments, which version was used? centered or uncentered? Figure 5 shows periodic patterns on covariance matrices. I didn*t find explanation of the periodic patterns in the covariance matrices. It would nice if the authors can explain it or point me the relevant sections in the paper. I don*t fully get the idea of LRC attack. Is it purely sampling? are there optimization involved? Figure 3, I suggest the authors show perturbations with different decay lengths on the same original images, which would make it easier to compare.