Title: ADVERSARIALLY ROBUST TRAINING THROUGH STRUCTURED GRADIENT REGULARIZATION

Abstract: We propose a novel data-dependent structured gradient regularizer to increase the robustness of neural networks vis-a-vis adversarial perturbations. Our regularizer can be derived as a controlled approximation from first principles, leveraging the fundamental link between training with noise and regularization. It adds very little computational overhead during learning and is simple to implement generically in standard deep learning frameworks. Our experiments provide strong evidence that structured gradient regularization can act as an effective first line of defense against attacks based on long-range correlated signal corruptions. 1

1. INTRODUCTION:

Deep neural networks, in particular convolutional neural networks (CNNs), have been used with great success for perceptual tasks such as image classification (Simonyan & Zisserman, 2014) or speech recognition (Hinton et al., 2012) . However, it has been shown that the accuracy of models obtained by standard training methods can dramatically deteriorate in the face of so-called adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014) , i.e. small perturbations in the input signal, typically imperceptible to humans, that are sufficient to induce large changes in the output. This apparent vulnerability is worrisome as CNNs start to proliferate in the real-world, including in safety-critical deployments.

Although the theoretical aspects of vulnerability to adversarial perturbations are not yet fully understood, a plethora of methods has been proposed to find adversarial examples. These often transfer or generalize across different architectures and datasets (Papernot et al., 2016a; Liu et al., 2016; Tramèr et al., 2017) , enabling black-box attacks even for inaccessible models. The most direct and commonly used strategy of protection is to use adversarial examples during training, see e.g. (Goodfellow et al., 2014) or (Miyato et al., 2015) . This raises the question, of whether one can generalize across such examples in order to become immune to a wider range of possible adversarial perturbations. Otherwise there appears to be a danger of overfitting to a specific attack, in particular if adversarial examples are sparsely generated from the training set.

In this paper, we pursue a different route, starting from the hypothesis that the weakness revealed by adversarial examples first and foremost hints at a data scarcity problem. Inspired by the classical work of Bishop (1995) and similar in spirit to Miyato et al. (2017) , we thus propose regularization as the preferred remedy to increase model robustness. Our regularization approach comes without rigid a priori assumptions on the model structure and filters (Bruna & Mallat, 2013) as well as strong (and often overly drastic) requirements such as layer-wise Lipschitz bounds (Cisse et al., 2017) or isotropic smoothness (Miyato et al., 2017) . Instead, we will rely on the ease of generating adversarial examples to learn an informative regularizer, focusing on the correlation structure of the adversarial noise. Our regularizer consistently outperforms structure-agnostic baselines on long-range correlated perturbations -achieving over 25% higher classification accuracy. We thus consider it an effective first line of defense on top of which other defense mechanisms may be built.

In summary, we make the following contributions:

• We provide evidence that current adversarial attacks act by perturbing the short-range covariance structure of signals.

• We propose Structured Gradient Regularization (SGR), a data-dependent regularizer informed by the covariance structure of adversarial perturbations. • We present an efficient SGR implementation with low memory footprint.

• We introduce a long-range correlated adversarial attack against which SGR-regularized classifiers are shown to be robust.

2. ADVERSARIAL ROBUST LEARNING:



2.1. VIRTUAL EXAMPLES:

Imagine we had access to a set of transformations τ ∈ T of inputs that leave outputs invariant. Then we could augment our training data by expanding each real example

(x i , y i ) into virtual examples (τ (x i ), y i ).

This approach is simple and widely-applicable, e.g. for exploiting known data invariances (Schölkopf et al., 1996) , and can greatly increase the statistical power of learning. A similar stochastic method has been suggested by Maaten et al. (2013) : a noisy channel or perturbation Q corrupts x → x with probability Q(x|x), leading to an altered (more precisely: convolved) joint distribution

P Q (x, y) = P (y) P (x|y)Q(x|x)dx.

Where could such a corruption model come from? We propose to use a generative mechanism to learn adversarial corruptions from examples. In fact, we will use a simple additive noise model in this paper,

EQUATION

Our focus will be on leveraging the correlation structure of the noise, captured by the matrix of raw second moments Σ, aggregated as a running exponentially weighted batch estimate computed from adversarial perturbations ξ, obtained by methods such as FGSM (Goodfellow et al., 2014) , PGD (Madry et al., 2017) or DeepFool (Moosavi Dezfooli et al., 2016) .

2.2. ROBUST MULTICLASS LEARNING OBJECTIVE:

Once we have an estimate of Q available, we propose to optimize any standard training criterion (e.g. a likelihood function) in expectation over a mixing distribution between the uncorrupted and the corrupted data. For the sake of concreteness, let us focus on the multiclass case, where y ∈ {1, . . . , K}. Define a probabilistic classifier

EQUATION

For a given loss function such as the negative log-loss, (y, π) = -log π y , our aim is to minimize

EQUATION

where P denotes the empirical distribution and PQ the noise corrupted distribution. The hyperparameter λ gives us a degree of freedom to pay more attention to the training data vs. our (imperfect) corruption model.

2.3. FROM VIRTUAL EXAMPLES TO REGULARIZATION:

Approaches relying on the explicit generation of adversarial examples face three statistical challenges: (i) How can enough diverse samples be generated to cover all attacks? (ii) How can we avoid a computational blow-up by adding too many virtual examples? (iii) How can we prevent overfitting to a specific attack? A remedy to these problems is through the use of regularization. The basic idea is simple: instead of sampling virtual examples, one tries to calculate the corresponding integrals in closed form, at least under reasonable approximations.

The general connection between regularization and training with noise-corrupted examples has been firmly established by Bishop (1995) . The benefits of regularization over adversarially augmented training in terms of generalization error have been hypothesized by Galloway et al. (2018) . In (Maaten et al., 2013) , properties of the loss function are exploited to achieve efficient regularization techniques for certain types of noise. Here, we follow the approach pursued in (Roth et al., 2017) in using an approximation that is accurate for small noise amplitudes.

3. STRUCTURED GRADIENT REGULARIZATION:



3.1. FROM CORRELATED NOISE TO STRUCTURED GRADIENT REGULARIZATION:

To approximate the expectation with regard to the noisy channel, we generalize a recent approach for gradient regularization of Generative Adversarial Networks (GANs) (Roth et al., 2017) , from binary classification to multiclass classification and from white noise to (adversarially) structured noise.

For the sake of simplicity of derivations we assume adversarial perturbations to be centered. Let us Taylor expand the log-component functions, making use of the shortcut ψ y := log φ y ,

ψ y (• + ξ) = ψ y + ∇ψ y ξ + 1 2 ξ [ψ y ]ξ + O( ξ 3 ) (4)

For any zero-mean distribution Q with second moments Σ,

EQUATION

)

The Hessian is calculated via the chain rule,

EQUATION

We will make a similar argument to (Roth et al., 2017) to show that it is reasonable to neglect the terms involving Hessians φ y . Let us therefore consider the Bayes-optimal classifier φ * y , for which up to a normalization constant φ * y (x) ∝ P (x|y)P (y) ,

such that

EQUATION

Exchanging summation and differentiation, we however have as a consequence of the normalization

EQUATION

Thus, under the assumption that φ ≈ φ * and of small perturbations (such that we can ignore higher order terms in Eq. ( 4)), we get

EQUATION

We can turn this into a regularizer by taking the leading terms (Roth et al., 2017) . Identifying P = P with the empirical distribution, we arrive at the following robust learning objective:

Structured Gradient Regularization (SGR) min φ → L SGR (φ) = E P [-log φ y (x)] + λΩ Σ (φ) (11) Ω Σ (φ) := 1 2 E P ∇ x log φ y (x) Σ ∇ x log φ y (x)

For uncentered adversarial perturbations E Q [ξ] = µ it is easy to see that the corresponding regularizer is given by

EQUATION

In practice, we however observed this correction to be small.

Algorithm 1 Adversarial SGR. Default values: λ ∈ [0, 1], β = 0.1

Input: Regularization strength λ, exponential decay rate β, batch size m Initial classifier parameters (weights and biases) ω 0 while ω t not converged do Sample minibatch of data {(x (1) , y (1) ), ..., (x (m) , y (m) )} ∼ P. Compute (normalized) adversarial perturbations ξ (1) , ..., ξ (m) Compute covariance matrix or covariance function Cov(ξ (1) , ...,

ξ (m) ) Update running average Σt ← (1-β) Σt-1 + β Cov(ξ (1) , ..., ξ (m) ) L(ω) = 1 m m i=1 K k=1 -y (i) k log φ k (x (i) ; ω) Ω Σt (ω) = 1 2m m i=1 K k=1 y (i) k ∇ x log φ k (x (i) ; ω) Σt ∇ x log φ k (x (i) ; ω) ω t ← ω t-1 -∇ ω L(ω) + λ Ω Σt (ω) ωt-1

return ω t The gradient-descent step can be performed with any learning rule. We used Adam in our experiments. See Sec. 3.3 for an efficient implementation of the regularizer.

3.2. PROPERTIES:

There are a few facts that are important to point out about the derived SGR regularizer.

(i) As the regularizer provides an efficiently computable approximation for an intractable expectation, it is clearly data-dependent. Ω Σ penalizes loss-gradients evaluated at training points. This is different from standard regularizers that penalize some norm of the parameter vector, such as L 2 -regularization. In the latter case, we would expect the regularizer to have the largest effect in the empty parts of the input space, where it should reduce the variability of the classifier outputs. On the contrary, Ω Σ has its main effect around the data manifold. In this sense, it is complementary to parameter regularization.

(ii) The SGR regularizer is intrinsic in the sense that it does not depend on the parameters of the classifier (for a good reason, we have not mentioned any parameterization), but instead directly acts on the function realized by the classifier φ (the relevant gradients measure the sensitivity of φ with regard to the input x and not with regard to parameters). Thus, it is parameterization invariant and can be naturally applied to any function space, whether it is finite-dimensional or not.

(iii) We can gain complementary insights into SGR by explicitly computing it in terms of classifier logits ϕ y (x). As outlined in Sec 7.1 in the Appendix, we obtain the following expression for the structured gradient regularizer

EQUATION

We can therefore see that SGR is penalizing large variations of the class-conditional logit-gradients ∇ϕ y around their data-dependent average ∇ϕ . For simple one-layer softmax classifiers we obtain

Ω Σ (φ) = 1/2 E P (ω y -ω ) Σ (ω y -ω ) .

This suggests an intriguing connection to variancebased regularization. Weight-decay regularization, on the other hand, simply penalizes large norms.

3.3. IMPLEMENTATION:

The implementation of SGR training is outlined in Algorithm 1. The matrix of raw second moments is aggregated through a running exponentially weighted average

EQUATION

with the decay rate β as a tuneable parameter trading off weighting between current (β → 1) and past (β → 0) batch averages. For low-dimensional data sets, we can directly compute the full matrix

EQUATION

For high-dimensional data sets, we have to resort to more memory-efficient representations leveraging the sparseness of the covariance matrix and covariance-gradient matrix-vector product. For image data sets, the covariance matrix can be estimated as a function of the displacement between pixels, as illustrated in Fig. 2 ,

EQUATION

)

where i = (i r , i c ) denotes the 2D pixel location and i is the covariance matrix index obtained by flattening the 2D pixel location into a 1D array such that i r and i c denote the row and column numbers of pixel i, and similar for j respectively. Rounding pixel displacements ||i -j|| 2 to the nearest integer, the covariance function simply becomes an array of real numbers storing the current estimate of the covariance between two pixels displaced by the difference in array indices.

In our experiments, we normalize the perturbations such that the average diagonal entry of the covariance matrix is one, thus allowing for more intuitive optimization over the regularization strength parameter λ. We note also that it is not necessary to update Σ at every training step and it is even possible to train with a fixed "expert"-designed covariance matrix.

The other crucial quantities needed to evaluate the regularizer are log φ k (•) for training points (x (i) , y (i) ). This is simply the per-sample cross-entropy loss, which is often available as a highly optimized callable operation in modern deep learning frameworks, reducing the implementation of the SGR regularizer to merely a few lines of code at very little computational overhead.

4. EXPERIMENTS:



4.1. EXPERIMENTAL SETUP:

Classifier architectures & data preprocessing. We trained Convolutional Neural Networks (CNNs) with seven hidden layers (and different numbers of filters for each data set), on CIFAR10 (Krizhevsky & Hinton, 2009) and MNIST (LeCun, 1998) . Our models are identical to those used in (Carlini & Wagner, 2017; Papernot et al., 2016b) . For both data sets, we adopt the following standard preprocessing and data augmentation scheme (He et al., 2016) : each training image is zero-padded with four pixels on each side2 , randomly cropped to produce a new image with the original dimensions and horizontally flipped with probability one half3 . We also standardize each image to have zero mean and unit variance when passing it to the classifier. Further details can be found in the Appendix. Adversarial attacks. We evaluate the classification accuracy against several adversarial attacks: Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) , Projected Gradient Descent (PGD) (Madry et al., 2017) and DeepFool (Moosavi Dezfooli et al., 2016) . The attack strength is reported in units of 1/255. The SNR reported for the DeepFool attack is computed according to Eq. ( 2) in (Moosavi Dezfooli et al., 2016) . The attacks were implemented with the open source CleverHans Library (Papernot et al., 2017) . Further details and attack hyper-parameters can be found in Sec. 7.2 in the Appendix. Figure 2 : PGD covariance functions: (Left) intra-channel covariance functions, measuring correlations between identical color channels, (the coloring in the plot matches the corresponding channel), (right) inter-channel covariance functions, measuring correlations between distinct color channels.

The rapid decay with pixel displacement, indicating very short decay length, is clearly visible.

4.2. COVARIANCE STRUCTURE OF ADVERSARIAL PERTURBATIONS:

To investigate the covariance structure of adversarial perturbations, we trained a clean classifier for 50 epochs and computed adversarial perturbations for every data point in the test set. The perturbations were flattened into 1D arrays before computing the full covariance matrix4 as in Eq. ( 15).

The short-range correlation structure of the perturbations, shown in Fig. 1 , is clearly visible: the correlations decay much faster than those of the data set covariance matrix. Figure 2 shows the corresponding PGD covariance functions. The rapid decay with pixel displacement, indicating very short decay length5 , is again evident. It thus seems that an unregularized classifier vulnerable to adversarial perturbations gives too much weight to short-range correlations (low-level patterns) and not enough weight to long-range ones (globally relevant high-level features of the data).

4.3. LONG-RANGE CORRELATED NOISE ATTACK:

To investigate the effect and potential benefit of using a structured covariance matrix in the SGR regularizer versus an "unstructured" diagonal covariance, corresponding to gradient-norm regularization, we compute the classification accuracy against a worst-of-100 attack in which the adversary samples perturbations from a long-range correlated multivariate Gaussian with covariance matrix specified through an exponentially decaying covariance function parametrized by a variable decay length ζ.

Inspired by the PGD covariance function, we chose intra-channel CovFun f (r) = exp(-r/ζ) and inter-channel CovFun f (r) = 0.5 exp(-r/ζ). The corresponding covariance matrices are depicted in Fig. 5 in the Appendix. LRC samples are shown in Fig. 3 .

The worst-of-100 attack then consists in perturbing every test set data point 100 times and evaluating the classifier accuracy against the worst of these perturbations. We tested for several decay lengths in the range ζ ∈ [1, 2, 4, 8, 16] and attack strengths ∈ [0.01, 0.7] with which the perturbations were scaled. Figure 4 in the Appendix shows the accuracy of different models as a function of . As a baseline, we also trained a model on LRC-augmented input. For SGR and GN we performed a hyper-parameter search over λ (at fixed ζ = 8) and report results for the best performing model. The LRC attack is a natural prototype for low frequency perturbations, as opposed to existing attacks which we have shown to mainly corrupt the short range (high frequency) structure of signals.

As shown in Fig. 3 (left), the low frequency range is precisely the regime where the benefit of structured regularization comes into play: SGR clearly and consistently outperforms GN on longrange correlated perturbations, achieving over 25% higher classification accuracy on LRC-perturbed input with ζ ∼ 16. As the decay length goes to zero, the synthetic covariance matrix converges to the identity matrix and SGR performance approaches GN performance.

Finally, to verify that those effects are not a result of our synthetically chosen covariance function, we also performed the same experiments with the covariance matrix computed from the CIFAR10 training set, shown in Fig. 1 . The accuracies for = 0.3 are: SGR 65.8%, GN 48.7%, LRC noise 57.9% and clean 46.5%, again confirming the benefit of using a structured covariance in the regularizer.

4.4. WHITE-BOX & TRANSFER ATTACK ACCURACY:

A summary of the white-box and transfer attack accuracies can be found in Table 1 . We train all models to achieve the highest possible white-box / transfer attack accuracies, averaged over in the range [0, 32] for the MNIST and [0, 8] for the CIFAR10 test set, i.e. the best performing models were selected according to the integrated area under the attack curve. The corresponding hyper-parameters are listed in Sec. 7.2 in the Appendix.

On MNIST, SGR/GN achieve PGD white-box accuracies of 96.5% compared to the 78.3% accuracy achieved by the clean model. On CIFAR10, SGR/GN trained models achieve an accuracy of 53.4% against targeted T-PGD attack (target is selected uniformly at random), whereas the clean model achieves 24.9% and PGD-trained model achieves 69.7% accuracy. SGR/GN trained models thus achieve white-box accuracies that are intermediate between those of the clean and adversarially trained models. The higher white-box accuracies achieved by adversarially trained models, however comes at the expense of a drop in test set accuracies. On CIFAR10, the test set accuracy of the PDG-trained model drops to 77.7%, while SGR/GN achieve 83.4% and 83.7% respectively.

An interesting observation can be made from the transfer attack accuracies (bold-face entries) on CIFAR10. As can be seen, the PGD adversarial perturbations obtained from SGR/GN trained models hurt the clean model more (60.3%) than the PGD perturbations obtained from PGD/FGSM adversarially trained models (77.0%/81.9%). Moreover, SGR achieves significantly higher accuracies against PGD transfer attacks derived from the clean model (80.6%) than the clean model does on PGD perturbations obtained from the SGR-trained model (60.3%), while accuracies achieved by SGR models on PDG samples from PGD-trained models (75.8%) are comparable to those in the other direction (74.0%).

As expected, the difference between SGR and GN in terms of white-box and transfer attack accuracies is less significant in this experiment, which can be explained by the evident short-range nature of current adversarial attacks (see discussion in Sec. 4.2). Reassuringly, we did not observe any loss of performance in all our experiments when using the covariance function instead of the full covariance matrix in the SGR regularizer.

5. RELATED WORK:

A large body of related work on distributionally robust optimization (DRO) (Sinha et al., 2017; Gao & Kleywegt, 2016) seeks a classifier that minimizes the worst-case loss against an adversary that can shift the entire training set within an uncertainty ball around the empirical distribution. Adversarial training can be viewed as a special case of DRO against a pointwise adversary that independently perturbs each example. While these methods aim at identifying the worst-case distribution, we propose to efficiently approximate expectations over corrupted distributions through structure-informed regularization.

Coincidentally, Ross & Doshi-Velez (2017) and Simon-Gabriel et al. (2018) also suggested to use gradient-norm regularization to improve the robustness against adversarial attacks. While (Simon-Gabriel et al., 2018) investigated the effect of different gradient-norms induced by constraints on the magnitude of the perturbations, we focused on a structured generalization of gradient regularization.

Similar in spirit to our work is also the recent work on adversarial training vs. weight decay regularization (Galloway et al., 2018) . While weight decay acts on the parameters, our regularizer acts on the function realized by the classifier. As discussed in Sec. 3.2, the two approaches are complementary to each other and can also be combined.

From a Bayesian perspective, regularization can be understood as imposing a prior distribution over the model parameters. As parameters in neural networks are non-identifiable, we ultimately prefer to impose priors in the space of functions directly (Teh, 2017; Neal, 2012; 1996) . As pointed out in Sec. 3.2, this is indeed one of the key properties of our regularizer.

6. CONCLUSION:

The fact that adversarial perturbations can fool classifiers while being imperceptible to the human eye, hints at a fundamental mismatch between how humans and classifiers try to make sense of the data they see. We provided evidence that current adversarial attacks act by perturbing the short-range correlations of signals. We proposed a novel structured gradient regularizer (SGR) informed by the covariance structure of adversarial noise and presented an efficient SGR implementation with low memory footprint. Devising further (e.g. gradient-based) low frequency attacks is an interesting direction of future research.

7. APPENDIX:

7.1 SGR PROPERTIES (iii) We can gain complementary insights into SGR by explicitly computing it in terms of logits

EQUATION

Defining the class average of the logit-gradients as

EQUATION

we obtain the following expression for the structured gradient regularizer

EQUATION

We can therefore see that SGR is penalizing large fluctuations of the class-conditional logit-gradients ∇ϕ y around their data-dependent class average ∇ϕ . For simple one-layer softmax classifiers ϕ y (x) = ω y x + b y , we obtain

EQUATION

This suggests an intriguing connection to variance-based regularization. Weight-decay regularization, on the other hand, simply penalizes large norms. An overview of the attack hyper-parameters can be found in Table 2 . Untargeted attacks find adversarial perturbations to an arbitrary one of the K -1 other classes. Both PGD and DeepFool are iterative attacks: they use 10 / 40 (for the white-box / transfer attack tables) and 100 iterations respectively. The numbers reported for the DeepFool attack are computed according to Eq. ( 2) in (Moosavi Dezfooli et al., 2016) . The adversarial example construction processes use the most likely label predicted by the classifier in order to avoid label leaking (Kurakin et al., 2016) during adversarially augmented training. The perturbed images returned by the attacks were clipped to lie within the original rgb range, as is commonly done in practice. The attacks were implemented with the open source CleverHans Library (Papernot et al., 2017) .

For SGR/GN we performed a hyper-parameter search over the regularization strength λ and SGR decay parameter β. 

7.3. FURTHER EXPERIMENTAL RESULTS:



Smoothness of Regularized Classifier:

At the most basic level of abstraction, adversarial vulnerability is related to an overly-steep classifier gradient, such that a tiny change in the input can lead to a large perturbation of the classifier output. In order to analyze this effect, we visualize the softmax activations along linear trajectories between various clean and corresponding adversarial examples.

As can be seen in Fig. 6 , the regularized classifier consistently leads to smooth classifier outputs.

Label leaking. Label leaking can cause models trained with adversarial examples to perform better on perturbed than on clean inputs, as the model can learn to exploit regularities in the adversarial example construction process (Kurakin et al., 2016) . This hints at a danger of overfitting to the specific attack when training with adversarial examples. SGR regularized classifiers, on the other hand, do not suffer from label leaking, as the adversarial examples are never directly fed to the classifier.

7.4. CLASSIFIER ARCHITECTURES:

Networks were trained for 50/75 epochs on MNIST/CIFAR10 using the Adam optimizer (Kingma & Ba, 2014) with learning rate 0.001 and minibatch size 128.

:

Code will be made available

We shifted the CIFAR10 data by -0.5 so that padding adds mid rgb-range pixels.

We do not flip MNIST digits.

Strictly speaking, Σ denotes the matrix of raw second moments. We will refer to Σ as the covariance matrix however, since the difference between the two is negligible.

Decay length is defined as the displacement over which the covariance function decays to 1/e of its value.

Figures and Tables:

Training methods. We train each classifier with a number of different training methods: (i) clean, i.e. through the standard softmax cross-entropy objective, (ii) clean + L 2 weight decay regularization, (iii) adversarially augmented training, i.e. training the classifier on a mixture of clean and adversarial examples (iv) GN (gradient-norm) regularized (corresponding to SGR with identity covariance matrix) and (v) SGR regularized. The SGR covariance matrix was computed from either: (a) loss gradients, (b) sign of loss gradients (corresponding to FGSM) or (c) PGD perturbations. The performance of these SGR variants was identical in all our experiments, hence we collectively report them as SGR.

Figure 1: Covariance matrices of PGD, FGSM and DeepFool perturbations as well as CIFAR10training set (for comparison). The short-range structure of the perturbations is clearly visible. It is also apparent that the first two attack methods yield perturbations with almost identical covariance structure. We are showing a center-crop for better visibility (25% trimmed on each side). For comparison the matrices were rescaled such that their maximum element has an absolute value of one.

Figure 3: (Left) Accuracy of different models as a function of the LRC attack decay length ζ ∈ [1, 16] at fixed = 0.3 (averaged over five runs). The plot clearly demonstrates the superiority of using a structured covariance matrix in the SGR regularizer. (Right) LRC perturbed samples with increasing decay lengths ζ ∈ [1, 2, 4, 8, 12] (two examples per decay length). The top row shows original test set images, the middle row shows adversarially perturbed samples, while the bottom row shows the adversarial perturbations rescaled to fit the full range of rgb values (their actual range and hence saliency is smaller). The footer indicates the classifier predictions on adversarial input.

For adversarial training, we trained models with in the full range ∈ [0, 32] for MNIST and ∈ [0, 8] for CIFAR10 and report results for the best performing one. The hyperparameters of the best performing models are: CIFAR10: SGR λ = 0.005, β = 0.1, GN γ = 0.025, PGD = 8, FGSM = 4, weight-decay 0.005. MNIST: SGR λ = 3.0, β = 0.1, GN γ = 10.0, PGD = 32, FGSM = 32, weight-decay 0.005.

Figure 4: Long Range Correlated Noise. Classification accuracy of different models as a function of the attack strength for an LRC attack with a fixed decay length ζ = 8 on CIFAR10. The SGR/GN hyper-parameters are reported in Sec. 4.3. See Fig. 3 for a plot of the accuracy of different models as a function of the LRC attack decay length ζ ∈ [1, 16] at fixed = 0.3

Test set, white-box and PGD-transfer attack accuracies. Each row corresponds to a model trained with a different training method, evaluated against the respective white-box (non-bold entries) and PGD-transfer attacks (bold-face entries) indicated above each column (T-PGD refers to targeted attack). The white-box and transfer attack accuracies are averaged over attack strengths in the range ∈ [0, 32] for MNIST and ∈ [0, 8] for CIFAR10, i.e. the reported accuracies represent the integrated area under the attack curve. For the transfer attack accuracies of a training method against itself, we trained a separate model with the same hyper-parameters to obtain the perturbations.

Table content: <table><tr><td/><td/><td/><td/><td colspan="2">MNIST = 32</td><td/><td/><td/><td/><td/></tr><tr><td>TRAIN. M.</td><td>TEST</td><td>RAND</td><td>PGD</td><td colspan="2">T-PGD CLEAN</td><td>PGD</td><td>FGSM</td><td>GN</td><td>SGR</td><td>FOOL</td></tr><tr><td>CLEAN</td><td>99.4</td><td>99.3</td><td>78.3</td><td>74.1</td><td>98.3</td><td>98.9</td><td>98.9</td><td>97.9</td><td>98.3</td><td>0.144</td></tr><tr><td>WDECAY</td><td>98.8</td><td>98.7</td><td>92.5</td><td>95.0</td><td>98.4</td><td>98.4</td><td>98.4</td><td>96.9</td><td>97.6</td><td>0.229</td></tr><tr><td>PGD</td><td>99.5</td><td>99.5</td><td>99.1</td><td>99.3</td><td>99.4</td><td>99.3</td><td>99.4</td><td>99.3</td><td>99.3</td><td>0.453</td></tr><tr><td>FGSM</td><td>99.4</td><td>99.4</td><td>98.5</td><td>98.0</td><td>99.3</td><td>99.2</td><td>99.2</td><td>99.2</td><td>99.2</td><td>0.339</td></tr><tr><td>GN</td><td>99.5</td><td>99.5</td><td>96.5</td><td>97.6</td><td>99.3</td><td>99.2</td><td>99.2</td><td>98.0</td><td>98.4</td><td>0.309</td></tr><tr><td>SGR</td><td>99.6</td><td>99.5</td><td>96.6</td><td>96.5</td><td>99.4</td><td>99.3</td><td>99.4</td><td>98.5</td><td>98.7</td><td>0.303</td></tr><tr><td/><td/><td/><td/><td colspan="2">CIFAR10 = 8</td><td/><td/><td/><td/><td/></tr><tr><td>TRAIN. M.</td><td>TEST</td><td>RAND</td><td>PGD</td><td colspan="2">T-PGD CLEAN</td><td>PGD</td><td>FGSM</td><td>GN</td><td>SGR</td><td>FOOL</td></tr><tr><td>CLEAN</td><td>84.7</td><td>83.2</td><td>28.8</td><td>24.9</td><td>72.2</td><td>77.0</td><td>81.9</td><td>61.9</td><td>60.3</td><td>0.010</td></tr><tr><td>WDECAY</td><td>81.4</td><td>79.2</td><td>27.1</td><td>32.3</td><td>69.7</td><td>71.7</td><td>68.9</td><td>54.1</td><td>52.9</td><td>0.015</td></tr><tr><td>PGD</td><td>77.7</td><td>77.7</td><td>62.4</td><td>69.7</td><td>77.1</td><td>71.5</td><td>76.2</td><td>74.0</td><td>74.0</td><td>0.061</td></tr><tr><td>FGSM</td><td>81.9</td><td>82.0</td><td>55.2</td><td>66.4</td><td>81.1</td><td>75.0</td><td>78.0</td><td>76.5</td><td>76.4</td><td>0.046</td></tr><tr><td>GN</td><td>83.7</td><td>83.7</td><td>41.9</td><td>54.1</td><td>81.6</td><td>76.9</td><td>82.0</td><td>68.1</td><td>67.9</td><td>0.039</td></tr><tr><td>SGR</td><td>83.4</td><td>82.8</td><td>41.5</td><td>53.4</td><td>80.6</td><td>75.8</td><td>81.2</td><td>66.8</td><td>65.5</td><td>0.039</td></tr></table>

Attack hyper-parameters. wb stands for white-box, t for transfer attacks.

Table content: <table><tr><td>7.2 HYPER-PARAMETERS</td><td/></tr><tr><td>Attack</td><td>Hyper-parameters</td></tr><tr><td>FGSM</td><td>: in units of 1/255</td></tr><tr><td>PGD</td><td>: in units of 1/255</td></tr><tr><td/><td>iter = /5</td></tr><tr><td/><td>nb_iter = 10 (wb) , 40 (t)</td></tr><tr><td>DEEP FOOL</td><td>overshoot : 0.02</td></tr><tr><td/><td>max_iter= 100</td></tr></table>

Peer review (iclr2019):
 Short paper summary: This work proposes a novel method of gradient regularization (SGR) which utilizes the covariance structure of adversarial examples generated during training. The authors propose simple techniques to reduce the computational overhead of SGR. Empirically, the authors compare their method to standard adversarial training and gradient norm regularization. Brief review summary: There are some interesting ideas in this work but I feel that the some practical aspects lack formal justification and the comparison to existing work is inconclusive. Detailed comments: In addition to some minor comments, I have two concerns. First, with the SGR algorithm itself. And second with the empirical analysis. While I suspect that the first concern may be clarified with discussion I think that the second is more serious and is the primary factor behind my review score. 1) As the SGR algorithm is written I wonder whether the regularization term may be computed more efficiently using something like a Hutchinson trace estimation trick. I suspect that if the random vector used to estimate the trace was the xi from Algorithm 1 then the same Mahalanobis gradient norm would be recovered. This would hold only in the case beta=1, bringing me to my second point. 2) What is the purpose of the running average of the covariance? A relatively small beta value is used in practice but I do not see any strong justification for this. Is there a good reason why we do not want the covariance matrix to be a close approximation for the local gradient landscape? This seems like an important part of the algorithm, especially as it may shed light on my next note. 3) In practice, Algorithm 1 uses adversarial attack schemes to generate the perturbations. In simple cases like FGM, this would give the covariance of the input-output gradient which seems that it would have a direct interpretation as a form of classical gradient regularization. To this extent, I also wonder how the SGR algorithm could be related to interpretations of adversarial training as gradient smoothing (when using small perturbations). I recognize that the above points are (so far as I could tell) not directly addressed in the work, and some may be fairly considered out of scope. However, due to the direct comparison to adversarial training later and the need to tie SGR to adversarial attacks I feel that it would be important to distinguish these cases. Overall, I felt that the first three sections did well to introduce the motivation and techniques used and were was easy to follow. The derivation of the SGR algorithm was clear and concise but I believe that some of the practical details (covariance running average, computational efficiency [at first glance, it looks like the full Jacobian must be computed, but practically the sum over K reduces this to a single backprop call]) could have been elaborated on. For the empirical evaluation the authors provided ample detail on the experimental set up and have performed a fairly thorough investigation in terms of existing defenses and attacks. I felt that the bulk of the study which is contained in Table 1 is fairly inconclusive or at the very least, difficult to interpret completely. Additional comments: 4) I felt that Figure 1 and 2 are a little difficult to interpret at first. It would help to clearly define what is meant by short- and long-range signal corruptions. However, they do suggest some interesting findings. As these covariance matrices depend directly on the model itself, I think it is worth investigate (or commenting on) how this structure may change when introducing things like SGR (or GN). The authors claim that unregularized classifiers give too much weight to short range correlations but they should show that SGN (or other methods) correct this. 5) My biggest concern with this work is with the results presented in Table 1. In terms of how they are presented: first I think that the fool column requires further explanation, or perhaps more simply the column could show accuracy instead of the average perturbation size. Second, I am not sure why the reported accuracies are averaged over attack strengths in a range. So far as I am aware, this is not standard and makes it difficult to interpret the performance of the models in this way. Figure 4 in the appendix does a better job of describing the behavior over a range of attack strengths. 6) From the table, it is not obvious to me that SGR provides any improvements to robustness over existing techniques. Indeed, the authors write that SGR achieves white-box accuracies which are between those of the clean and adversarially trained models and claim that SGR improves on the clean accuracy for CIFAR-10. But in the table the gap between FGSM and GN/SGR clean accuracies seem fairly small with FGSM providing better robustness (for most source attacks). Even more concerning, is the fact that GN seems to outperform SGR. I do not find these results substantial enough to motivate SGR as a robustness defense compared with adversarial training (or even GN), especially as SGR has the same computational limitations involved with expensive adversarial perturbations. I felt that the study into the covariance structure of adversarial perturbations was interesting but as it stands was not complete enough to be informative in general. In the conclusion the authors write that they provide evidence that current adversarial attacks act by perturbing the short-range correlations of signals but this has only been confirmed for unregularized classifiers. Despite these issues, I thought that the paper was well written and hope that the empirical study can be improved and clarified. Minor comments: - Section 2.1, set of transformations only introduced briefly then forgotten. Leaving output invariant confused me, as this does not apply to adversarial examples. - Section 2.3, second paragraph l3: In Maaten et al. should be citet. - Section 3.1, should make clear that derivative is with respect to the data. - Section 3.1, define delta as the Hessian clearly (it is used for the simplex in the previous section). Though this is easy to figure out. - Section 7.1, starts with (iii), is this intentional? Perhaps an introductory sentence could make this clearer. - Section 7.3, for label leaking, I*m not convinced by this argument alone. Assuming the covariance structure is still computed from a particular adversarial example, I see no compelling reason that this would not occur. Clarity: The paper is very clearly written and is easy to follow.