Title: ADVERSARIALLY ROBUST TRAINING THROUGH STRUCTURED GRADIENT REGULARIZATION

Abstract: We propose a novel data-dependent structured gradient regularizer to increase the robustness of neural networks vis-a-vis adversarial perturbations. Our regularizer can be derived as a controlled approximation from first principles, leveraging the fundamental link between training with noise and regularization. It adds very little computational overhead during learning and is simple to implement generically in standard deep learning frameworks. Our experiments provide strong evidence that structured gradient regularization can act as an effective first line of defense against attacks based on long-range correlated signal corruptions. 1

1. INTRODUCTION:

Deep neural networks, in particular convolutional neural networks (CNNs), have been used with great success for perceptual tasks such as image classification (Simonyan & Zisserman, 2014) or speech recognition (Hinton et al., 2012) . However, it has been shown that the accuracy of models obtained by standard training methods can dramatically deteriorate in the face of so-called adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014) , i.e. small perturbations in the input signal, typically imperceptible to humans, that are sufficient to induce large changes in the output. This apparent vulnerability is worrisome as CNNs start to proliferate in the real-world, including in safety-critical deployments.

Although the theoretical aspects of vulnerability to adversarial perturbations are not yet fully understood, a plethora of methods has been proposed to find adversarial examples. These often transfer or generalize across different architectures and datasets (Papernot et al., 2016a; Liu et al., 2016; Tramèr et al., 2017) , enabling black-box attacks even for inaccessible models. The most direct and commonly used strategy of protection is to use adversarial examples during training, see e.g. (Goodfellow et al., 2014) or (Miyato et al., 2015) . This raises the question, of whether one can generalize across such examples in order to become immune to a wider range of possible adversarial perturbations. Otherwise there appears to be a danger of overfitting to a specific attack, in particular if adversarial examples are sparsely generated from the training set.

In this paper, we pursue a different route, starting from the hypothesis that the weakness revealed by adversarial examples first and foremost hints at a data scarcity problem. Inspired by the classical work of Bishop (1995) and similar in spirit to Miyato et al. (2017) , we thus propose regularization as the preferred remedy to increase model robustness. Our regularization approach comes without rigid a priori assumptions on the model structure and filters (Bruna & Mallat, 2013) as well as strong (and often overly drastic) requirements such as layer-wise Lipschitz bounds (Cisse et al., 2017) or isotropic smoothness (Miyato et al., 2017) . Instead, we will rely on the ease of generating adversarial examples to learn an informative regularizer, focusing on the correlation structure of the adversarial noise. Our regularizer consistently outperforms structure-agnostic baselines on long-range correlated perturbations -achieving over 25% higher classification accuracy. We thus consider it an effective first line of defense on top of which other defense mechanisms may be built.

In summary, we make the following contributions:

• We provide evidence that current adversarial attacks act by perturbing the short-range covariance structure of signals.

• We propose Structured Gradient Regularization (SGR), a data-dependent regularizer informed by the covariance structure of adversarial perturbations. • We present an efficient SGR implementation with low memory footprint.

• We introduce a long-range correlated adversarial attack against which SGR-regularized classifiers are shown to be robust.

2. ADVERSARIAL ROBUST LEARNING:



2.1. VIRTUAL EXAMPLES:

Imagine we had access to a set of transformations τ ∈ T of inputs that leave outputs invariant. Then we could augment our training data by expanding each real example

(x i , y i ) into virtual examples (τ (x i ), y i ).

This approach is simple and widely-applicable, e.g. for exploiting known data invariances (Schölkopf et al., 1996) , and can greatly increase the statistical power of learning. A similar stochastic method has been suggested by Maaten et al. (2013) : a noisy channel or perturbation Q corrupts x → x with probability Q(x|x), leading to an altered (more precisely: convolved) joint distribution

P Q (x, y) = P (y) P (x|y)Q(x|x)dx.

Where could such a corruption model come from? We propose to use a generative mechanism to learn adversarial corruptions from examples. In fact, we will use a simple additive noise model in this paper,

EQUATION

Our focus will be on leveraging the correlation structure of the noise, captured by the matrix of raw second moments Σ, aggregated as a running exponentially weighted batch estimate computed from adversarial perturbations ξ, obtained by methods such as FGSM (Goodfellow et al., 2014) , PGD (Madry et al., 2017) or DeepFool (Moosavi Dezfooli et al., 2016) .

2.2. ROBUST MULTICLASS LEARNING OBJECTIVE:

Once we have an estimate of Q available, we propose to optimize any standard training criterion (e.g. a likelihood function) in expectation over a mixing distribution between the uncorrupted and the corrupted data. For the sake of concreteness, let us focus on the multiclass case, where y ∈ {1, . . . , K}. Define a probabilistic classifier

EQUATION

For a given loss function such as the negative log-loss, (y, π) = -log π y , our aim is to minimize

EQUATION

where P denotes the empirical distribution and PQ the noise corrupted distribution. The hyperparameter λ gives us a degree of freedom to pay more attention to the training data vs. our (imperfect) corruption model.

2.3. FROM VIRTUAL EXAMPLES TO REGULARIZATION:

Approaches relying on the explicit generation of adversarial examples face three statistical challenges: (i) How can enough diverse samples be generated to cover all attacks? (ii) How can we avoid a computational blow-up by adding too many virtual examples? (iii) How can we prevent overfitting to a specific attack? A remedy to these problems is through the use of regularization. The basic idea is simple: instead of sampling virtual examples, one tries to calculate the corresponding integrals in closed form, at least under reasonable approximations.

The general connection between regularization and training with noise-corrupted examples has been firmly established by Bishop (1995) . The benefits of regularization over adversarially augmented training in terms of generalization error have been hypothesized by Galloway et al. (2018) . In (Maaten et al., 2013) , properties of the loss function are exploited to achieve efficient regularization techniques for certain types of noise. Here, we follow the approach pursued in (Roth et al., 2017) in using an approximation that is accurate for small noise amplitudes.

3. STRUCTURED GRADIENT REGULARIZATION:



3.1. FROM CORRELATED NOISE TO STRUCTURED GRADIENT REGULARIZATION:

To approximate the expectation with regard to the noisy channel, we generalize a recent approach for gradient regularization of Generative Adversarial Networks (GANs) (Roth et al., 2017) , from binary classification to multiclass classification and from white noise to (adversarially) structured noise.

For the sake of simplicity of derivations we assume adversarial perturbations to be centered. Let us Taylor expand the log-component functions, making use of the shortcut ψ y := log φ y ,

ψ y (• + ξ) = ψ y + ∇ψ y ξ + 1 2 ξ [ψ y ]ξ + O( ξ 3 ) (4)

For any zero-mean distribution Q with second moments Σ,

EQUATION

)

The Hessian is calculated via the chain rule,

EQUATION

We will make a similar argument to (Roth et al., 2017) to show that it is reasonable to neglect the terms involving Hessians φ y . Let us therefore consider the Bayes-optimal classifier φ * y , for which up to a normalization constant φ * y (x) ∝ P (x|y)P (y) ,

such that

EQUATION

Exchanging summation and differentiation, we however have as a consequence of the normalization

EQUATION

Thus, under the assumption that φ ≈ φ * and of small perturbations (such that we can ignore higher order terms in Eq. ( 4)), we get

EQUATION

We can turn this into a regularizer by taking the leading terms (Roth et al., 2017) . Identifying P = P with the empirical distribution, we arrive at the following robust learning objective:

Structured Gradient Regularization (SGR) min φ → L SGR (φ) = E P [-log φ y (x)] + λΩ Σ (φ) (11) Ω Σ (φ) := 1 2 E P ∇ x log φ y (x) Σ ∇ x log φ y (x)

For uncentered adversarial perturbations E Q [ξ] = µ it is easy to see that the corresponding regularizer is given by

EQUATION

In practice, we however observed this correction to be small.

Algorithm 1 Adversarial SGR. Default values: λ ∈ [0, 1], β = 0.1

Input: Regularization strength λ, exponential decay rate β, batch size m Initial classifier parameters (weights and biases) ω 0 while ω t not converged do Sample minibatch of data {(x (1) , y (1) ), ..., (x (m) , y (m) )} ∼ P. Compute (normalized) adversarial perturbations ξ (1) , ..., ξ (m) Compute covariance matrix or covariance function Cov(ξ (1) , ...,

ξ (m) ) Update running average Σt ← (1-β) Σt-1 + β Cov(ξ (1) , ..., ξ (m) ) L(ω) = 1 m m i=1 K k=1 -y (i) k log φ k (x (i) ; ω) Ω Σt (ω) = 1 2m m i=1 K k=1 y (i) k ∇ x log φ k (x (i) ; ω) Σt ∇ x log φ k (x (i) ; ω) ω t ← ω t-1 -∇ ω L(ω) + λ Ω Σt (ω) ωt-1

return ω t The gradient-descent step can be performed with any learning rule. We used Adam in our experiments. See Sec. 3.3 for an efficient implementation of the regularizer.

3.2. PROPERTIES:

There are a few facts that are important to point out about the derived SGR regularizer.

(i) As the regularizer provides an efficiently computable approximation for an intractable expectation, it is clearly data-dependent. Ω Σ penalizes loss-gradients evaluated at training points. This is different from standard regularizers that penalize some norm of the parameter vector, such as L 2 -regularization. In the latter case, we would expect the regularizer to have the largest effect in the empty parts of the input space, where it should reduce the variability of the classifier outputs. On the contrary, Ω Σ has its main effect around the data manifold. In this sense, it is complementary to parameter regularization.

(ii) The SGR regularizer is intrinsic in the sense that it does not depend on the parameters of the classifier (for a good reason, we have not mentioned any parameterization), but instead directly acts on the function realized by the classifier φ (the relevant gradients measure the sensitivity of φ with regard to the input x and not with regard to parameters). Thus, it is parameterization invariant and can be naturally applied to any function space, whether it is finite-dimensional or not.

(iii) We can gain complementary insights into SGR by explicitly computing it in terms of classifier logits ϕ y (x). As outlined in Sec 7.1 in the Appendix, we obtain the following expression for the structured gradient regularizer

EQUATION

We can therefore see that SGR is penalizing large variations of the class-conditional logit-gradients ∇ϕ y around their data-dependent average ∇ϕ . For simple one-layer softmax classifiers we obtain

Ω Σ (φ) = 1/2 E P (ω y -ω ) Σ (ω y -ω ) .

This suggests an intriguing connection to variancebased regularization. Weight-decay regularization, on the other hand, simply penalizes large norms.

3.3. IMPLEMENTATION:

The implementation of SGR training is outlined in Algorithm 1. The matrix of raw second moments is aggregated through a running exponentially weighted average

EQUATION

with the decay rate β as a tuneable parameter trading off weighting between current (β → 1) and past (β → 0) batch averages. For low-dimensional data sets, we can directly compute the full matrix

EQUATION

For high-dimensional data sets, we have to resort to more memory-efficient representations leveraging the sparseness of the covariance matrix and covariance-gradient matrix-vector product. For image data sets, the covariance matrix can be estimated as a function of the displacement between pixels, as illustrated in Fig. 2 ,

EQUATION

)

where i = (i r , i c ) denotes the 2D pixel location and i is the covariance matrix index obtained by flattening the 2D pixel location into a 1D array such that i r and i c denote the row and column numbers of pixel i, and similar for j respectively. Rounding pixel displacements ||i -j|| 2 to the nearest integer, the covariance function simply becomes an array of real numbers storing the current estimate of the covariance between two pixels displaced by the difference in array indices.

In our experiments, we normalize the perturbations such that the average diagonal entry of the covariance matrix is one, thus allowing for more intuitive optimization over the regularization strength parameter λ. We note also that it is not necessary to update Σ at every training step and it is even possible to train with a fixed "expert"-designed covariance matrix.

The other crucial quantities needed to evaluate the regularizer are log φ k (•) for training points (x (i) , y (i) ). This is simply the per-sample cross-entropy loss, which is often available as a highly optimized callable operation in modern deep learning frameworks, reducing the implementation of the SGR regularizer to merely a few lines of code at very little computational overhead.

4. EXPERIMENTS:



4.1. EXPERIMENTAL SETUP:

Classifier architectures & data preprocessing. We trained Convolutional Neural Networks (CNNs) with seven hidden layers (and different numbers of filters for each data set), on CIFAR10 (Krizhevsky & Hinton, 2009) and MNIST (LeCun, 1998) . Our models are identical to those used in (Carlini & Wagner, 2017; Papernot et al., 2016b) . For both data sets, we adopt the following standard preprocessing and data augmentation scheme (He et al., 2016) : each training image is zero-padded with four pixels on each side2 , randomly cropped to produce a new image with the original dimensions and horizontally flipped with probability one half3 . We also standardize each image to have zero mean and unit variance when passing it to the classifier. Further details can be found in the Appendix. Adversarial attacks. We evaluate the classification accuracy against several adversarial attacks: Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) , Projected Gradient Descent (PGD) (Madry et al., 2017) and DeepFool (Moosavi Dezfooli et al., 2016) . The attack strength is reported in units of 1/255. The SNR reported for the DeepFool attack is computed according to Eq. ( 2) in (Moosavi Dezfooli et al., 2016) . The attacks were implemented with the open source CleverHans Library (Papernot et al., 2017) . Further details and attack hyper-parameters can be found in Sec. 7.2 in the Appendix. Figure 2 : PGD covariance functions: (Left) intra-channel covariance functions, measuring correlations between identical color channels, (the coloring in the plot matches the corresponding channel), (right) inter-channel covariance functions, measuring correlations between distinct color channels.

The rapid decay with pixel displacement, indicating very short decay length, is clearly visible.

4.2. COVARIANCE STRUCTURE OF ADVERSARIAL PERTURBATIONS:

To investigate the covariance structure of adversarial perturbations, we trained a clean classifier for 50 epochs and computed adversarial perturbations for every data point in the test set. The perturbations were flattened into 1D arrays before computing the full covariance matrix4 as in Eq. ( 15).

The short-range correlation structure of the perturbations, shown in Fig. 1 , is clearly visible: the correlations decay much faster than those of the data set covariance matrix. Figure 2 shows the corresponding PGD covariance functions. The rapid decay with pixel displacement, indicating very short decay length5 , is again evident. It thus seems that an unregularized classifier vulnerable to adversarial perturbations gives too much weight to short-range correlations (low-level patterns) and not enough weight to long-range ones (globally relevant high-level features of the data).

4.3. LONG-RANGE CORRELATED NOISE ATTACK:

To investigate the effect and potential benefit of using a structured covariance matrix in the SGR regularizer versus an "unstructured" diagonal covariance, corresponding to gradient-norm regularization, we compute the classification accuracy against a worst-of-100 attack in which the adversary samples perturbations from a long-range correlated multivariate Gaussian with covariance matrix specified through an exponentially decaying covariance function parametrized by a variable decay length ζ.

Inspired by the PGD covariance function, we chose intra-channel CovFun f (r) = exp(-r/ζ) and inter-channel CovFun f (r) = 0.5 exp(-r/ζ). The corresponding covariance matrices are depicted in Fig. 5 in the Appendix. LRC samples are shown in Fig. 3 .

The worst-of-100 attack then consists in perturbing every test set data point 100 times and evaluating the classifier accuracy against the worst of these perturbations. We tested for several decay lengths in the range ζ ∈ [1, 2, 4, 8, 16] and attack strengths ∈ [0.01, 0.7] with which the perturbations were scaled. Figure 4 in the Appendix shows the accuracy of different models as a function of . As a baseline, we also trained a model on LRC-augmented input. For SGR and GN we performed a hyper-parameter search over λ (at fixed ζ = 8) and report results for the best performing model. The LRC attack is a natural prototype for low frequency perturbations, as opposed to existing attacks which we have shown to mainly corrupt the short range (high frequency) structure of signals.

As shown in Fig. 3 (left), the low frequency range is precisely the regime where the benefit of structured regularization comes into play: SGR clearly and consistently outperforms GN on longrange correlated perturbations, achieving over 25% higher classification accuracy on LRC-perturbed input with ζ ∼ 16. As the decay length goes to zero, the synthetic covariance matrix converges to the identity matrix and SGR performance approaches GN performance.

Finally, to verify that those effects are not a result of our synthetically chosen covariance function, we also performed the same experiments with the covariance matrix computed from the CIFAR10 training set, shown in Fig. 1 . The accuracies for = 0.3 are: SGR 65.8%, GN 48.7%, LRC noise 57.9% and clean 46.5%, again confirming the benefit of using a structured covariance in the regularizer.

4.4. WHITE-BOX & TRANSFER ATTACK ACCURACY:

A summary of the white-box and transfer attack accuracies can be found in Table 1 . We train all models to achieve the highest possible white-box / transfer attack accuracies, averaged over in the range [0, 32] for the MNIST and [0, 8] for the CIFAR10 test set, i.e. the best performing models were selected according to the integrated area under the attack curve. The corresponding hyper-parameters are listed in Sec. 7.2 in the Appendix.

On MNIST, SGR/GN achieve PGD white-box accuracies of 96.5% compared to the 78.3% accuracy achieved by the clean model. On CIFAR10, SGR/GN trained models achieve an accuracy of 53.4% against targeted T-PGD attack (target is selected uniformly at random), whereas the clean model achieves 24.9% and PGD-trained model achieves 69.7% accuracy. SGR/GN trained models thus achieve white-box accuracies that are intermediate between those of the clean and adversarially trained models. The higher white-box accuracies achieved by adversarially trained models, however comes at the expense of a drop in test set accuracies. On CIFAR10, the test set accuracy of the PDG-trained model drops to 77.7%, while SGR/GN achieve 83.4% and 83.7% respectively.

An interesting observation can be made from the transfer attack accuracies (bold-face entries) on CIFAR10. As can be seen, the PGD adversarial perturbations obtained from SGR/GN trained models hurt the clean model more (60.3%) than the PGD perturbations obtained from PGD/FGSM adversarially trained models (77.0%/81.9%). Moreover, SGR achieves significantly higher accuracies against PGD transfer attacks derived from the clean model (80.6%) than the clean model does on PGD perturbations obtained from the SGR-trained model (60.3%), while accuracies achieved by SGR models on PDG samples from PGD-trained models (75.8%) are comparable to those in the other direction (74.0%).

As expected, the difference between SGR and GN in terms of white-box and transfer attack accuracies is less significant in this experiment, which can be explained by the evident short-range nature of current adversarial attacks (see discussion in Sec. 4.2). Reassuringly, we did not observe any loss of performance in all our experiments when using the covariance function instead of the full covariance matrix in the SGR regularizer.

5. RELATED WORK:

A large body of related work on distributionally robust optimization (DRO) (Sinha et al., 2017; Gao & Kleywegt, 2016) seeks a classifier that minimizes the worst-case loss against an adversary that can shift the entire training set within an uncertainty ball around the empirical distribution. Adversarial training can be viewed as a special case of DRO against a pointwise adversary that independently perturbs each example. While these methods aim at identifying the worst-case distribution, we propose to efficiently approximate expectations over corrupted distributions through structure-informed regularization.

Coincidentally, Ross & Doshi-Velez (2017) and Simon-Gabriel et al. (2018) also suggested to use gradient-norm regularization to improve the robustness against adversarial attacks. While (Simon-Gabriel et al., 2018) investigated the effect of different gradient-norms induced by constraints on the magnitude of the perturbations, we focused on a structured generalization of gradient regularization.

Similar in spirit to our work is also the recent work on adversarial training vs. weight decay regularization (Galloway et al., 2018) . While weight decay acts on the parameters, our regularizer acts on the function realized by the classifier. As discussed in Sec. 3.2, the two approaches are complementary to each other and can also be combined.

From a Bayesian perspective, regularization can be understood as imposing a prior distribution over the model parameters. As parameters in neural networks are non-identifiable, we ultimately prefer to impose priors in the space of functions directly (Teh, 2017; Neal, 2012; 1996) . As pointed out in Sec. 3.2, this is indeed one of the key properties of our regularizer.

6. CONCLUSION:

The fact that adversarial perturbations can fool classifiers while being imperceptible to the human eye, hints at a fundamental mismatch between how humans and classifiers try to make sense of the data they see. We provided evidence that current adversarial attacks act by perturbing the short-range correlations of signals. We proposed a novel structured gradient regularizer (SGR) informed by the covariance structure of adversarial noise and presented an efficient SGR implementation with low memory footprint. Devising further (e.g. gradient-based) low frequency attacks is an interesting direction of future research.

7. APPENDIX:

7.1 SGR PROPERTIES (iii) We can gain complementary insights into SGR by explicitly computing it in terms of logits

EQUATION

Defining the class average of the logit-gradients as

EQUATION

we obtain the following expression for the structured gradient regularizer

EQUATION

We can therefore see that SGR is penalizing large fluctuations of the class-conditional logit-gradients ∇ϕ y around their data-dependent class average ∇ϕ . For simple one-layer softmax classifiers ϕ y (x) = ω y x + b y , we obtain

EQUATION

This suggests an intriguing connection to variance-based regularization. Weight-decay regularization, on the other hand, simply penalizes large norms. An overview of the attack hyper-parameters can be found in Table 2 . Untargeted attacks find adversarial perturbations to an arbitrary one of the K -1 other classes. Both PGD and DeepFool are iterative attacks: they use 10 / 40 (for the white-box / transfer attack tables) and 100 iterations respectively. The numbers reported for the DeepFool attack are computed according to Eq. ( 2) in (Moosavi Dezfooli et al., 2016) . The adversarial example construction processes use the most likely label predicted by the classifier in order to avoid label leaking (Kurakin et al., 2016) during adversarially augmented training. The perturbed images returned by the attacks were clipped to lie within the original rgb range, as is commonly done in practice. The attacks were implemented with the open source CleverHans Library (Papernot et al., 2017) .

For SGR/GN we performed a hyper-parameter search over the regularization strength λ and SGR decay parameter β. 

7.3. FURTHER EXPERIMENTAL RESULTS:



Smoothness of Regularized Classifier:

At the most basic level of abstraction, adversarial vulnerability is related to an overly-steep classifier gradient, such that a tiny change in the input can lead to a large perturbation of the classifier output. In order to analyze this effect, we visualize the softmax activations along linear trajectories between various clean and corresponding adversarial examples.

As can be seen in Fig. 6 , the regularized classifier consistently leads to smooth classifier outputs.

Label leaking. Label leaking can cause models trained with adversarial examples to perform better on perturbed than on clean inputs, as the model can learn to exploit regularities in the adversarial example construction process (Kurakin et al., 2016) . This hints at a danger of overfitting to the specific attack when training with adversarial examples. SGR regularized classifiers, on the other hand, do not suffer from label leaking, as the adversarial examples are never directly fed to the classifier.

7.4. CLASSIFIER ARCHITECTURES:

Networks were trained for 50/75 epochs on MNIST/CIFAR10 using the Adam optimizer (Kingma & Ba, 2014) with learning rate 0.001 and minibatch size 128.

:

Code will be made available

We shifted the CIFAR10 data by -0.5 so that padding adds mid rgb-range pixels.

We do not flip MNIST digits.

Strictly speaking, Σ denotes the matrix of raw second moments. We will refer to Σ as the covariance matrix however, since the difference between the two is negligible.

Decay length is defined as the displacement over which the covariance function decays to 1/e of its value.

Figures and Tables:

Training methods. We train each classifier with a number of different training methods: (i) clean, i.e. through the standard softmax cross-entropy objective, (ii) clean + L 2 weight decay regularization, (iii) adversarially augmented training, i.e. training the classifier on a mixture of clean and adversarial examples (iv) GN (gradient-norm) regularized (corresponding to SGR with identity covariance matrix) and (v) SGR regularized. The SGR covariance matrix was computed from either: (a) loss gradients, (b) sign of loss gradients (corresponding to FGSM) or (c) PGD perturbations. The performance of these SGR variants was identical in all our experiments, hence we collectively report them as SGR.

Figure 1: Covariance matrices of PGD, FGSM and DeepFool perturbations as well as CIFAR10training set (for comparison). The short-range structure of the perturbations is clearly visible. It is also apparent that the first two attack methods yield perturbations with almost identical covariance structure. We are showing a center-crop for better visibility (25% trimmed on each side). For comparison the matrices were rescaled such that their maximum element has an absolute value of one.

Figure 3: (Left) Accuracy of different models as a function of the LRC attack decay length ζ ∈ [1, 16] at fixed = 0.3 (averaged over five runs). The plot clearly demonstrates the superiority of using a structured covariance matrix in the SGR regularizer. (Right) LRC perturbed samples with increasing decay lengths ζ ∈ [1, 2, 4, 8, 12] (two examples per decay length). The top row shows original test set images, the middle row shows adversarially perturbed samples, while the bottom row shows the adversarial perturbations rescaled to fit the full range of rgb values (their actual range and hence saliency is smaller). The footer indicates the classifier predictions on adversarial input.

For adversarial training, we trained models with in the full range ∈ [0, 32] for MNIST and ∈ [0, 8] for CIFAR10 and report results for the best performing one. The hyperparameters of the best performing models are: CIFAR10: SGR λ = 0.005, β = 0.1, GN γ = 0.025, PGD = 8, FGSM = 4, weight-decay 0.005. MNIST: SGR λ = 3.0, β = 0.1, GN γ = 10.0, PGD = 32, FGSM = 32, weight-decay 0.005.

Figure 4: Long Range Correlated Noise. Classification accuracy of different models as a function of the attack strength for an LRC attack with a fixed decay length ζ = 8 on CIFAR10. The SGR/GN hyper-parameters are reported in Sec. 4.3. See Fig. 3 for a plot of the accuracy of different models as a function of the LRC attack decay length ζ ∈ [1, 16] at fixed = 0.3

Test set, white-box and PGD-transfer attack accuracies. Each row corresponds to a model trained with a different training method, evaluated against the respective white-box (non-bold entries) and PGD-transfer attacks (bold-face entries) indicated above each column (T-PGD refers to targeted attack). The white-box and transfer attack accuracies are averaged over attack strengths in the range ∈ [0, 32] for MNIST and ∈ [0, 8] for CIFAR10, i.e. the reported accuracies represent the integrated area under the attack curve. For the transfer attack accuracies of a training method against itself, we trained a separate model with the same hyper-parameters to obtain the perturbations.

Table content: <table><tr><td/><td/><td/><td/><td colspan="2">MNIST = 32</td><td/><td/><td/><td/><td/></tr><tr><td>TRAIN. M.</td><td>TEST</td><td>RAND</td><td>PGD</td><td colspan="2">T-PGD CLEAN</td><td>PGD</td><td>FGSM</td><td>GN</td><td>SGR</td><td>FOOL</td></tr><tr><td>CLEAN</td><td>99.4</td><td>99.3</td><td>78.3</td><td>74.1</td><td>98.3</td><td>98.9</td><td>98.9</td><td>97.9</td><td>98.3</td><td>0.144</td></tr><tr><td>WDECAY</td><td>98.8</td><td>98.7</td><td>92.5</td><td>95.0</td><td>98.4</td><td>98.4</td><td>98.4</td><td>96.9</td><td>97.6</td><td>0.229</td></tr><tr><td>PGD</td><td>99.5</td><td>99.5</td><td>99.1</td><td>99.3</td><td>99.4</td><td>99.3</td><td>99.4</td><td>99.3</td><td>99.3</td><td>0.453</td></tr><tr><td>FGSM</td><td>99.4</td><td>99.4</td><td>98.5</td><td>98.0</td><td>99.3</td><td>99.2</td><td>99.2</td><td>99.2</td><td>99.2</td><td>0.339</td></tr><tr><td>GN</td><td>99.5</td><td>99.5</td><td>96.5</td><td>97.6</td><td>99.3</td><td>99.2</td><td>99.2</td><td>98.0</td><td>98.4</td><td>0.309</td></tr><tr><td>SGR</td><td>99.6</td><td>99.5</td><td>96.6</td><td>96.5</td><td>99.4</td><td>99.3</td><td>99.4</td><td>98.5</td><td>98.7</td><td>0.303</td></tr><tr><td/><td/><td/><td/><td colspan="2">CIFAR10 = 8</td><td/><td/><td/><td/><td/></tr><tr><td>TRAIN. M.</td><td>TEST</td><td>RAND</td><td>PGD</td><td colspan="2">T-PGD CLEAN</td><td>PGD</td><td>FGSM</td><td>GN</td><td>SGR</td><td>FOOL</td></tr><tr><td>CLEAN</td><td>84.7</td><td>83.2</td><td>28.8</td><td>24.9</td><td>72.2</td><td>77.0</td><td>81.9</td><td>61.9</td><td>60.3</td><td>0.010</td></tr><tr><td>WDECAY</td><td>81.4</td><td>79.2</td><td>27.1</td><td>32.3</td><td>69.7</td><td>71.7</td><td>68.9</td><td>54.1</td><td>52.9</td><td>0.015</td></tr><tr><td>PGD</td><td>77.7</td><td>77.7</td><td>62.4</td><td>69.7</td><td>77.1</td><td>71.5</td><td>76.2</td><td>74.0</td><td>74.0</td><td>0.061</td></tr><tr><td>FGSM</td><td>81.9</td><td>82.0</td><td>55.2</td><td>66.4</td><td>81.1</td><td>75.0</td><td>78.0</td><td>76.5</td><td>76.4</td><td>0.046</td></tr><tr><td>GN</td><td>83.7</td><td>83.7</td><td>41.9</td><td>54.1</td><td>81.6</td><td>76.9</td><td>82.0</td><td>68.1</td><td>67.9</td><td>0.039</td></tr><tr><td>SGR</td><td>83.4</td><td>82.8</td><td>41.5</td><td>53.4</td><td>80.6</td><td>75.8</td><td>81.2</td><td>66.8</td><td>65.5</td><td>0.039</td></tr></table>

Attack hyper-parameters. wb stands for white-box, t for transfer attacks.

Table content: <table><tr><td>7.2 HYPER-PARAMETERS</td><td/></tr><tr><td>Attack</td><td>Hyper-parameters</td></tr><tr><td>FGSM</td><td>: in units of 1/255</td></tr><tr><td>PGD</td><td>: in units of 1/255</td></tr><tr><td/><td>iter = /5</td></tr><tr><td/><td>nb_iter = 10 (wb) , 40 (t)</td></tr><tr><td>DEEP FOOL</td><td>overshoot : 0.02</td></tr><tr><td/><td>max_iter= 100</td></tr></table>

Peer review (iclr2019):
 The authors propose a new defense against adversarial examples that relies on a data-dependent regularization (instead of adversarial training). They then benchmark the performance of this new defense against popular white-box and transfer attacks, as well as propose a new long range correlated adversarial attack. Comments: I find the premise of this paper interesting - developing regularization strategies to help with generalization to adversarial perturbations. For instance, it is well known that state-of-the-art defenses such as PGD have generalization gaps as large as 50% between robust train and test accuracies. It has also been previously hypothesized that this could be due to a data scarcity problem [Schmidt et al., 2018]. The authors here propose to tackle this problem using a new data-dependent regularization technique. My primary issue with this paper is that the authors do not clearly illustrate what the advantage of their method over standard methods is - The problem this paper aims to solve is overfitting to a specific attack/virtual adversarial examples presented during adversarial training by using regularization instead. However, the authors do not actually illustrate that their technique reduces overfitting. For instance, the authors do not contrast the robust train-test accuracies using their method to other standard methods. Thus it is not clear that this paper met the objectives laid out in the introduction. - The claim in this paper is that SGR helps against attacks with long range dependencies. However, in their experiments (e.g., in Figure 3), the authors do not evaluate other standard defenses. It is thus unclear whether other standard methods are already robust to such attacks. In fact, based on the results of Table 1, it doesn’t seem like attacks from SGR are able to reduce the robustness of PGD/FGSM trained models. Because of these two points, along with the lower robustness to various attacks (in Table 1) as compared to approaches such as PGD, it is not really clear to me what the real merit of this new approach is. Ultimately, having a defense which is more robust to a particular attack is not very meaningful if there exists an alternative attack that reduces the robustness of the defense. I am also surprised that the authors chose to use this regularization as an alternative to adversarial training instead of complementary to it. I would be interested to see if such regularization could actually help to bridge the generalization gap observed while using adversarial training. The paper is at times is poorly written and confusing. For instance, the description of CovFun is hard to parse. The authors should make this explanation more clear. The authors also do not state what their attack model is - Linf vs L2 perturbations. They also choose to evaluate attacks differently, using an average accuracy over different epsilons rather than reporting individual accuracies. This does make the results harder to compare to other work. The authors should include a full table of individual accuracies (at least in the appendix) to make the numbers easier to parse and compare. In the derivation in Section 3.1, the authors use the assumption that the robust classifier is almost equal to the Bayes optimal classifier to justify dropping terms corresponding to the Hessian(.phi_y). I am not sure how realistic this assumption is in the adversarial setting - one can construct simple distributions for which the Bayes optimal classifier is not the robust classifier. With regards to Figure 3, the authors state - “As the decay length goes to zero, the synthetic covariance matrix converges to the identity matrix and SGR performance approaches GN performance” Could the authors clarify why this is obvious? After all these two models are trained very differently. The plot in Figure 3 and the results in Table 1 seems to illustrate that SGR is no better than GN as you can find an attack where they perform as well/badly. The authors say that this is due to the short-range nature of current attacks. I do not understand this rationale though - the goal of the defenses should be to be more robust to all attacks, both short range and long range. Thus arguing that there may be an attack under which their model performs better is not sufficient. I do agree that finding long range attacks that can break current SOTA robust models would be interesting, however the authors do not seem to achieve that in this work. I find the observation on transfer attacks interesting - PGD attacks from SGR/GN models are better than PGD models. Do the authors have any insight as to why this is the case? In general, my concern about gradient regularization based defenses is that they only give a very local picture of the landscape and thus can only protect against small eps attacks. This could probably explain why the SGR/GN models are less robust than PGD. As mentioned previously, it would be valuable to see accuracies against individual eps values (rather than averaged) to understand this better. If this is the case, this regularization would not provide any additional benefits when combined with adversarial training either. References: Schmidt, Ludwig, et al. "Adversarially Robust Generalization Requires More Data." arXiv preprint arXiv:1804.11285 (2018).