Title: ADEF: AN ITERATIVE ALGORITHM TO CONSTRUCT ADVERSARIAL DEFORMATIONS

Abstract: While deep neural networks have proven to be a powerful tool for many recognition and classification tasks, their stability properties are still not well understood. In the past, image classifiers have been shown to be vulnerable to so-called adversarial attacks, which are created by additively perturbing the correctly classified image. In this paper, we propose the ADef algorithm to construct a different kind of adversarial attack created by iteratively applying small deformations to the image, found through a gradient descent step. We demonstrate our results on MNIST with convolutional neural networks and on ImageNet with Inception-v3 and ResNet-101.

1. INTRODUCTION:

In a first observation in Szegedy et al. (2013) it was found that deep neural networks exhibit unstable behavior to small perturbations in the input. For the task of image classification this means that two visually indistinguishable images may have very different outputs, resulting in one of them being misclassified even if the other one is correctly classified with high confidence. Since then, a lot of research has been done to investigate this issue through the construction of adversarial examples: given a correctly classified image x, we look for an image y which is visually indistinguishable from x but is misclassified by the network. Typically, the image y is constructed as y = x + r, where r is an adversarial perturbation that is supposed to be small in a suitable sense (normally, with respect to an p norm). Several algorithms have been developed to construct adversarial perturbations, see Goodfellow et al. (2014); Moosavi Dezfooli et al. (2016) ; Kurakin et al. (2017b) ; Madry et al. (2018) ; Carlini & Wagner (2017b) and the review paper Akhtar & Mian (2018) .

Even though such pathological cases are very unlikely to occur in practice, their existence is relevant since malicious attackers may exploit this drawback to fool classifiers or other automatic systems. Further, adversarial perturbations may be constructed in a black-box setting (i.e., without knowing the architecture of the DNN but only its outputs) (Papernot et al., 2017; Moosavi-Dezfooli et al., 2017) and also in the physical world (Kurakin et al., 2017b; Athalye & Sutskever, 2017; Brown et al., 2017; Sharif et al., 2016) . This has motivated the investigation of defenses, i.e., how to make the network invulnerable to such attacks, see Kurakin et al. (2017a) ; Carlini & Wagner (2017a) ; Madry et al. (2018) ; Tramèr et al. (2018) ; Wong & Kolter (2018) ; Raghunathan et al. (2018) ; Athalye et al. (2018) ; Kannan et al. (2018) . In most cases, adversarial examples are artificially created and then used to retrain the network, which becomes more stable under these types of perturbations.

Most of the work on the construction of adversarial examples and on the design of defense strategies has been conducted in the context of small perturbations r measured in the ∞ norm. However, this is not necessarily a good measure of image similarity: e.g., for two translated images x and y, the norm of x-y is not small in general, even though x and y will look indistinguishable if the translation is small. Several papers have investigated the construction of adversarial perturbations not designed for norm proximity (Rozsa et al., 2016; Sharif et al., 2016; Brown et al., 2017; Engstrom et al., 2017; Xiao et al., 2018) .

In this work, we build up on these ideas and investigate the construction of adversarial deformations. In other words, the misclassified image y is not constructed as an additive perturbation y = x + r, but as a deformation y = x • (id + τ ), where τ is a vector field defining the transformation. In this case, the similarity is not measured through a norm of yx, but instead through a norm of τ , which quantifies the deformation between y and x.

We develop an efficient algorithm for the construction of adversarial deformations, which we call ADef. It is based on the main ideas of DeepFool (Moosavi Dezfooli et al., 2016) , and iteratively constructs the smallest deformation to misclassify the image. We test the procedure on MNIST (LeCun) (with convolutional neural networks) and on ImageNet (Russakovsky et al., 2015) (with Inception-v3 (Szegedy et al., 2016) and ResNet-101 (He et al., 2016) ). The results show that ADef can succesfully fool the classifiers in the vast majority of cases (around 99%) by using very small and imperceptible deformations. We also test our adversarial attacks on adversarially trained networks for MNIST. Our implementation of the algorithm can be found at https://gitlab.math. ethz.ch/tandrig/ADef.

The results of this work have initially appeared in the master's thesis Gauksson (2017) , to which we refer for additional details on the mathematical aspects of this construction. While writing this paper, we have come across Xiao et al. (2018) , in which a similar problem is considered and solved with a different algorithm. Whereas in Xiao et al. (2018) the authors use a second order solver to find a deforming vector field, we show how a first order method can be formulated efficiently and justify a smoothing operation, independent of the optimization step. We report, for the first time, success rates for adversarial attacks with deformations on ImageNet. The topic of deformations has also come up in Jaderberg et al. (2015) , in which the authors introduce a class of learnable modules that deform inputs in order to increase the performance of existing DNNs, and Fawzi & Frossard (2015) , in which the authors introduce a method to measure the invariance of classifiers to geometric transformations.

2. ADVERSARIAL DEFORMATIONS:



2.1. ADVERSARIAL PERTURBATIONS:

Let K be a classifier of images consisting of P pixels into L ≥ 2 categories, i.e. a function from the space of images X = R cP , where c = 1 (for grayscale images) or c = 3 (for color images), and into the set of labels L = {1, . . . , L}. Suppose x ∈ X is an image that is correctly classified by K and suppose y ∈ X is another image that is imperceptible from x and such that K(y) = K(x), then y is said to be an adversarial example. The meaning of imperceptibility varies, but generally, proximity in p -norm (with 1 ≤ p ≤ ∞) is considered to be a sufficient substitute. Thus, an adversarial perturbation for an image x ∈ X is a vector r ∈ X such that K(x + r) = K(x) and r p is small, where

EQUATION

Given such a classifier K and an image x, an adversary may attempt to find an adversarial example y by minimizing xy p subject to K(y) = K(x), or even subject to K(y) = k for some target label k = K(x). Different methods for finding minimal adversarial perturbations have been proposed, most notably FGSM (Goodfellow et al., 2014) and PGD (Madry et al., 2018) for ∞ , and the DeepFool algorithm (Moosavi Dezfooli et al., 2016) for general p -norms.

2.2. DEFORMATIONS:

Instead of constructing adversarial perturbations, we intend to fool the classifier by small deformations of correctly classified images. Our procedure is in the spirit of the DeepFool algorithm. Before we explain it, let us first clarify what we mean by a deformation of an image. The discussion is at first more intuitive if we model images as functions ξ :

[0, 1] 2 → R c (with c = 1 or c = 3) instead Original ∞ : 1.00

Translation by (-2, 1)

∞ : 0.98

Rotation by 10 of discrete vectors x in R cP . In this setting, perturbing an image ξ : [0, 1] 2 → R c corresponds to adding to it another function ρ : [0, 1] 2 → R c with a small L p -norm.

While any transformation of an image ξ can be written as a perturbation ξ + ρ, we shall restrict ourselves to a particular class of transformations. A deformation with respect to a vector field τ : [0, 1] 2 → R 2 is a transformation of the form ξ → ξ τ , where for any image ξ :

[0, 1] 2 → R c , the image ξ τ : [0, 1] 2 → R c is defined by ξ τ (u) = ξ (u + τ (u)) for all u ∈ [0, 1] 2 ,

extending ξ by zero outside of [0, 1] 2 . Deformations capture many natural image transformations. For example, a translation of the image ξ by a vector v ∈ R 2 is a deformation with respect to the constant vector field τ = v. If v is small, the images ξ and ξ v may look similar, but the corresponding perturbation ρ = ξ vξ may be arbitrarily large in the aforementioned L p -norms. Figure 1 shows three minor deformations, all of which yield large L ∞ -norms.

In the discrete setting, deformations are implemented as follows. We consider square images of W × W pixels and define the space of images to be

X = R W ×W c . A discrete vector field is a function τ : {1, . . . , W } 2 → R 2 .

In what follows we will only consider the set T of vector fields that do not move points on the grid {1, . . . , W } 2 outside of [1, W ] 2 . More precisely,

T := τ : {1, . . . , W } 2 → R 2 | τ (s, t) + (s, t) ∈ [1, W ] 2 for all s, t ∈ {1, . . . , W } .

An image x ∈ X can be viewed as the collection of values of a function ξ :

[0, 1] 2 → R c on a regular grid { 1 /(W +1), . . . , W /(W +1)} 2 ⊆ [0, 1] 2 , i.e. x s,t = ξ( s /(W +1), t /(W +1))

for s, t = 1, . . . , W . Such a function ξ can be computed by interpolating from x. Thus, the deformation of an image x with respect to the discrete vector field τ can be defined as the discrete deformed image x τ in X by

EQUATION

It is not straightforward to measure the size of a deformation such that it captures the visual difference between the original image x and its deformed counterpart x τ . We will use the size of the corresponding vector field, τ , in the norm defined by

τ T = max s,t=1,...,W τ (s, t) 2 (3)

as a proxy. The p -norms defined in (1), adapted to vector fields, can be used as well. (We remark, however, that none of these norms define a distance between x and x τ , since two vector fields τ, σ ∈ T with τ T = σ T may produce the same deformed image x τ = x σ .)

2.3. THE ALGORITHM ADEF:

We will now describe our procedure for finding deformations that will lead a classifier to yield an output different from the original label.

Let F = (F 1 , . . . , F L ) : X → R L be the underlying model for the classifier K, such that

K(x) = arg max k=1,...,L F k (x).

Let x ∈ X be the image of interest and fix ξ : [0, 1] 2 → R c obtained by interpolation from x. Let l = K(x) denote the true label of x, let k ∈ L be a target label and set f = F k -F l . We assume that x does not lie on a decision boundary, so that we have f (x) < 0.

We define the function g :

T → R, τ → f (x τ ) and note that g(0) = f (x 0 ) = f (x) < 0.

Our goal is to find a small vector field τ ∈ T such that g(τ ) = f (x τ ) ≥ 0. We can use a linear approximation of g around the zero vector field as a guide:

EQUATION

for small enough τ ∈ T and D 0 g : T → R the derivative of g at τ = 0. Hence, if τ is a vector field such that (D 0 g) τ = -g(0)

(5) and τ T is small, then the classifier K has approximately equal confidence for the deformed image x τ to have either label l or k. This is a scalar equation with unknown in T , and so has infinitely many solutions. In order to select τ with small norm, we solve it in the least-squares sense.

In view of (2), we have

∂x τ ∂τ | τ =0 (s, t) = 1 W +1 ∇ξ (s,t) W +1

∈ R c×2 . Thus, by applying the chain rule to g(τ ) = f (x τ ), we obtain that its derivative at τ = 0 can, with a slight abuse of notation, be identified with the vector field

EQUATION

where ∇f (x) s,t ∈ R 1×c is the derivative of f in x calculated at (s, t). With this, (D 0 g) τ stands for W s,t=1 D 0 g(s, t) • τ (s, t), and the solution to (5) in the least-square sense is given by

EQUATION

Finally, we define the deformed image x τ ∈ X according to (2).

One might like to impose some degree of smoothness on the deforming vector field. In fact, it suffices to search in the range of a smoothing operator S : T → T . However, this essentially amounts to applying S to the solution from the larger search space T . Let α = S(D 0 g) = ϕ * (D 0 g), where S denotes the componentwise application of a two-dimensional Gaussian filter ϕ (of any standard deviation). Then the vector field

τ = - f (x) W s,t=1 |α(s, t)| 2 Sα = - f (x) W s,t=1 |α(s, t)| 2 S 2 (D 0 g)

also satisfies (5), since S is self-adjoint. We can hence replace τ by τ to obtain a smooth deformation of the image x.

We iterate the deformation process until the deformed image is misclassified. More explicitly, let x (0) = x and for n ≥ 1 let τ (n) be given by ( 7) for x (n-1) . Then we can define the iteration as

x (n) = x (n-1) • (id + τ (n)

). The algorithm terminates and outputs an adversarial example y

= x (n) if K(x (n) ) = l.

The iteration also terminates if x (n) lies on a decision boundary of K, in which case we propose to introduce an overshoot factor 1 + η on the total deforming vector field. Provided that the number of iterations is moderate, the total vector field can be well approximated by τ * = τ (1) +• • •+τ (n) and the process can be altered to output the deformed image y = x•(id+(1+η)τ * ) instead.

The target label k may be chosen in each iteration to minimize the vector field to obtain a better approximation in the linearization (4). More precisely, for a candidate set of labels k 1 , . . . , k m , we compute the corresponding vectors fields τ 1 , . . . , τ m and select k = arg min j=1,...,m τ j T .

The candidate set consists of the labels corresponding to the indices of the m smallest entries of F -F l , in absolute value.

Algorithm ADef:

Input: Classification model F , image x, correct label l, candidate labels k 1 , . . . , k m Output: Deformed image y Initialize y ← x while K(y) = l do for j = 1, . . . , m do

α j ← S c i=1 (∇F kj ) i -(∇F l ) i • ∇y i τ j ← - F k j (y)-F l (y) αj 2 2 Sα j

end for i ← arg min j=1,...,m τ j T y ← y • (id + τ i ) end while return y By equation ( 6), provided that ∇f is moderate, the deforming vector field takes small values wherever ξ has a small derivative. This means that the vector field will be concentrated on the edges in the image x (see e.g. the first row of figure 2 ). Further, note that the result of a deformation is always a valid image in the sense that it does not violate the pixel value bounds. This is not guaranteed for the perturbations computed with DeepFool.

3. EXPERIMENTS:



3.1. SETUP:

We evaluate the performance of ADef by applying the algorithm to classifiers trained on the MNIST (LeCun) and ImageNet (Russakovsky et al., 2015) datasets. Below, we briefly describe the setup of the experiments and in tables 1 and 2 we summarize their results.

MNIST::

We train two convolutional neural networks based on architectures that appear in Madry et al. (2018) and Tramèr et al. (2018) respectively. The network MNIST-A consists of two convolutional layers of sizes 32 × 5 × 5 and 64 × 5 × 5, each followed by 2 × 2 max-pooling and a rectifier activation function, a fully connected layer into dimension 1024 with a rectifier activation function, and a final linear layer with output dimension 10. The network MNIST-B consists of two convolutional layers of sizes 128 × 3 × 3 and 64 × 3 × 3 with a rectifier activation function, a fully connected layer into dimension 128 with a rectifier activation function, and a final linear layer with output dimension 10. During training, the latter convolutional layer and the former fully connected layer of MNIST-B are subject to dropout of drop probabilities 1 /4 and 1 /2. We use ADef to produce adversarial deformations of the images in the test set. The algorithm is configured to pursue any label different from the correct label (all incorrect labels are candidate labels). It performs smoothing by a Gaussian filter of standard deviation 1 /2, uses bilinear interpolation to obtain intermediate pixel intensities, and it overshoots by η = 2 /10 whenever it converges to a decision boundary.

Table 1 : The results of applying ADef to the images in the MNIST test set and the ILSVRC2012 validation set. The accuracy of the Inception and ResNet models is defined as the top-1 accuracy on the center-cropped and resized images. The success rate of ADef is shown as a percentage of the correctly classified inputs. The pixel range is scaled to [0, 1], so the perturbation r = yx, where x is the input and y the output of ADef, has values in [-1, 1] . The averages in the three last columns are computed over the set of images on which ADef is successful. Recall the definition of the vector field norm in equation ( 3).

Model:

Accuracy ADef success Avg. Second row: A larger deformation of the same image, obtained by using a wider Gaussian filter (standard deviation 6) for smoothing.

ImageNet: We apply ADef to pretrained Inception-v3 (Szegedy et al., 2016) and ResNet-101 (He et al., 2016) models to generate adversarial deformations for the images in the ILSVRC2012 validation set. The images are preprocessed by first scaling so that the smaller axis has 299 pixels for the Inception model and 224 pixels for ResNet, and then they are center-cropped to a square image. The algorithm is set to focus only on the label of second highest probability. It employs a Gaussian filter of standard deviation 1, bilinear interpolation, and an overshoot factor η = 1 /10.

We only consider inputs that are correctly classified by the model in question, and, since τ * = τ (1) +• • •+τ (n) approximates the total deforming vector field, we declare ADef to be successful if its output is misclassified and τ * T ≤ ε, where we choose ε = 3. Observe that, by (3), a deformation with respect to a vector field τ does not displace any pixel further away from its original position than τ T . Hence, for high resolution images, the choice ε = 3 indeed produces small deformations if the vector fields are smooth. In appendix A, we illustrate how the success rate of ADef depends on the choice of ε.

When searching for an adversarial example, one usually searches for a perturbation with ∞ -norm smaller than some small number ε > 0. Common choices of ε range from 1 /10 to 3 /10 for MNIST classifiers (Goodfellow et al., 2014; Madry et al., 2018; Wong & Kolter, 2018; Tramèr et al., 2018; Kannan et al., 2018) and 2 /255 to 16 /255 for ImageNet classifiers (Goodfellow et al., 2014; Kurakin et al., 2017a; Tramèr et al., 2018; Kannan et al., 2018) . Table 1 shows that on average, the perturbations obtained by ADef are quite large compared to those constraints. However, as can be seen in figure 2 , the relatively high resolution images of the ImageNet dataset can be deformed into adversarial examples that, while corresponding to large perturbations, are not visibly different from the original images. In appendices B and C, we give more examples of adversarially deformed images.

3.2. ADVERSARIAL TRAINING:

In addition to training MNIST-A and MNIST-B on the original MNIST data, we train independent copies of the networks using the adversarial training procedure described by Madry et al. (2018) . That is, before each step of the training process, the input images are adversarially perturbed using the PGD algorithm. This manner of training provides increased robustness against adversarial perturbations of low ∞ -norm. Moreover, we train networks using ADef instead of PGD as an adversary. In table 2 we show the results of attacking these adversarially trained networks, using ADef on the one hand, and PGD on the other. We use the same configuration for ADef as above, and for PGD we use 40 iterations, step size 1 /100 and 3 /10 as the maximum ∞ -norm of the perturbation. Interestingly, using these configurations, the networks trained against PGD attacks are more resistant to adversarial deformations than those trained against ADef.

3.3. TARGETED ATTACKS:

ADef can also be used for targeted adversarial attacks, by restricting the deformed image to have a particular target label instead of any label which yields the optimal deformation. Figure 3 demonstrates the effect of choosing different target labels for a given MNIST image, and figure 4 shows the result of targeting the label of lowest probability for an image from the ImageNet dataset. T from the MNIST experiments. Deformations that fall to the left of the vertical line at ε = 3 are considered successful. The networks in the first column were trained using the original MNIST data, and the networks in the second and third columns were adversarially trained using ADef and PGD, respectively. A DISTRIBUTION OF VECTOR FIELD NORMS Figures 5 and 6 show the distribution of the norms of the total deforming vector fields, τ * , from the experiments in section 3. For networks that have not been adversarially trained, most deformations fall well below the threshold of = 3. Out of the adversarially trained networks, only MNIST-A trained against PGD is truly robust against ADef. Further, a comparison between the first column of figure 5 and figure 6 indicates that ImageNet is much more vulnerable to adversarial deformations than MNIST, also considering the much higher resolution of the images in ImageNet. Thus, it would be very interesting to study the performance of ADef with adversarially trained network for ImageNet, as mentioned in the Conclusion. 

B SMOOTH DEFORMATIONS:

The standard deviation of the Gaussian filter used for smoothing in the update step of ADef has significant impact on the resulting vector field. To explore this aspect of the algorithm, we repeat the experiment from section 3 on the Inception-v3 model, using standard deviations σ = 0, 1, 2, 4, 8 (where σ = 0 stands for no smoothing). The results are shown in table 3, and the effect of varying σ is illustrated in figures 7 and 8. We observe that as σ increases, the adversarial distortion steadily increases both in terms of vector field norm and perturbation norm. Likewise, the success rate of ADef decreases with larger σ. However, from figure 8 we see that the constraint τ * T ≤ 3 on the total vector field may provide a rather conservative measure of the effectiveness of ADef in the case of smooth high dimensional vector fields.

C ADDITIONAL DEFORMED IMAGES:

C.1 MNIST Figures 9 and 10 show adversarial deformations for the models MNIST-A and MNIST-B, respectively. The attacks are performed using the same configuration as in the experiments in section 3. Observe that in some cases, features resembling the target class have appeared in the deformed image. For example, the top part of the 4 in the fifth column of figure 10 has been curved slightly to more resemble a 9.

C.2 IMAGENET:

Figures 11 -15 show additional deformed images resulting from attacking the Inception-v3 model using the same configuration as in the experiments in section 3. Similarly, figures 16 -20 show deformed images resulting from attacking the ResNet-10 model. However, in order to increase variability in the output labels, we perform a targeted attack, targeting the label of 50th highest probability.

:



Figures and Tables:

Figure 1: First row: The original 28 × 28 pixel image from the MNIST database, and the same image translated by (-2, 1), rotated by an angle of 10 • , and deformed w.r.t. an arbitrary smooth vector field τ . The ∞ -norm of the corresponding perturbation is shown under each deformed image. The pixel values range from 0 (white) to 1 (black), so the deformed images all lie far from the original image in the ∞ -norm. Second row: The vector fields corresponding to the above deformations and their T -norms (cf. equation (3)).

Figure2: Sample deformations for the Inception-v3 model. The vector fields and perturbations have been amplified for visualization. First row: An image from the ILSVRC2012 validation set, the output of ADef with a Gaussian filter of standard deviation 1, the corresponding vector field and perturbation. The rightmost image is a close-up of the vector field around the nose of the ape. Second row: A larger deformation of the same image, obtained by using a wider Gaussian filter (standard deviation 6) for smoothing.

Figure 3: Targeted ADef against MNIST-A. First row: The original image and deformed images produced by restricting ADef to the target labels 0 to 8. The ∞ -norms of the corresponding perturbations are shown under the deformed images. Second row: The vector fields corresponding to the deformations and their T -norms.

Figure 5: The (normalized) distribution of τ *T from the MNIST experiments. Deformations that fall to the left of the vertical line at ε = 3 are considered successful. The networks in the first column were trained using the original MNIST data, and the networks in the second and third columns were adversarially trained using ADef and PGD, respectively.

Figure 6: The (normalized) distribution of τ * T from the ImageNet experiments. Deformations that fall to the left of the vertical line at ε = 3 are considered successful.

τ *

Table content: <table><tr><td>MNIST-A</td><td>98.99%</td><td>99.85%</td><td>1.1950</td><td>T</td><td colspan="2">Avg. r ∞ Avg. # iterations 0.7455 7.002</td></tr><tr><td>MNIST-B</td><td>98.91%</td><td>99.51%</td><td>1.0841</td><td/><td>0.7654</td><td>4.422</td></tr><tr><td>Inception-v3</td><td>77.56%</td><td>98.94%</td><td>0.5984</td><td/><td>0.2039</td><td>4.050</td></tr><tr><td>ResNet-101</td><td>76.97%</td><td>99.78%</td><td>0.5561</td><td/><td>0.1882</td><td>4.176</td></tr><tr><td>Original: orangutan</td><td colspan="2">Deformed: chimpanzee</td><td/><td/><td/></tr></table>

Success rates for PGD and ADef attacks on adversarially trained networks.

Table content: <table><tr><td>Model</td><td colspan="4">Adv. training Accuracy PGD success ADef success</td></tr><tr><td>MNIST-A</td><td>PGD ADef</td><td>98.36% 98.95%</td><td>5.81% 100.00%</td><td>6.67% 54.16%</td></tr><tr><td>MNIST-B</td><td>PGD ADef</td><td>98.74% 98.79%</td><td>5.84% 100.00%</td><td>20.35% 45.07%</td></tr></table>

The results of applying ADef to the images in the ILSVRC2012 validation set and the Inception model, using different values for the standard deviation σ of the Gaussian filter. As before, we define ADef to be successful if τ * T ≤ 3.

Table content: <table><tr><td colspan="3">σ ADef success Avg. τ  *  0 99.12% 0.5272</td><td>T</td><td colspan="2">Avg. r ∞ Avg. # iterations 0.1628 5.247</td></tr><tr><td>1</td><td>98.94%</td><td>0.5984</td><td/><td>0.2039</td><td>4.050</td></tr><tr><td>2</td><td>95.91%</td><td>0.7685</td><td/><td>0.2573</td><td>3.963</td></tr><tr><td>4</td><td>86.66%</td><td>0.9632</td><td/><td>0.3128</td><td>4.379</td></tr><tr><td>8</td><td>67.54%</td><td>1.1684</td><td/><td>0.3687</td><td>5.476</td></tr></table>

Peer review (iclr2019):
 The paper proposes a new way to construct adversarial examples: do not change the intensity of the input image directly, but deform the image plane (i.e. compose the image with Id + tau where tau is a small amplitude vector field). The paper originates from a document provably written in late 2017, which is before the deposit on arXiv of another article (by different authors, early 2018) which was later accepted to ICLR 2018 [Xiao and al.]. This remark is important in that it changes my rating of the paper (being more indulgent with papers proposing new ideas, as otherwise the novelty is rather low compared to [Xiao and al.]). Pros: - the paper is well written, very easy to read, well explained (and better formalized than [Xiao and al.]); - the idea of deforming images is new (if we forget about [Xiao and al.]) and simple; - experiments show what such a technique can achieve on MNIST and ImageNet. Interestingly, one can see on MNIST the parts of the numbers that the adversarial attack is trying to delete/create. Cons: - the paper is a bit weak, in that it is not very dense, and in that there is not much more content than the initial idea; - for instance, more discussions about the results obtained could have been appreciated (such as my remark above about MNIST); - for instance, a study of the impact of the regularization would have been interesting (how does the sigma of the Gaussian smoothing affect the type of adversarial attacks obtained and their performance -- is it possible to fool the network with [very] smooth deformations?); - for instance, what about generating adversarial examples for which the network would be fully (wrongly) confident? (instead of just borderline unsure); etc. - The interpolation scheme (how is defined the intensity I(x,y) for a non-integer location (x,y) within the image I) is rather important (linear interpolation, etc.) and should be at least mentioned in the main paper, and at best studied (it might impact the gradient descent path and the results); - question: does the algorithm converge? could there be a proof of this? This is not obvious, as the objective potentially changes with time (selection of the current m best indices k of |F_k - F_l|). Also, the final overshoot factor (1+eta) is not very elegant, and not guaranteed to perform well if tau* starts being not small compared to the second derivative (i.e. g**.tau^2 not small) while I guess that for image intensities, spatial derivatives can be very high if no intensity smoothing scheme is used. - note: the approximation tau* = sum_i tau_i (section 2.3) does not stand in the case of non-small deformations. - still in section 2.3, I do not understand the statement "given that .nabla f is moderate": where does this property come from? or is "given" meant to be understood as "provided..." (i.e. under the assumption that...)? - computational times could have been given (though I guess they are reasonable). Other remarks: - suggestion: I find the "slight abuse of notation" (of confusing the derivative with the gradient) a bit annoying and suggest to use a different symbol, such as .nabla g. This could be useful in particular in the following perspective: - Mathematical side note: the "gradient" of a functional is not a uniquely-defined object in that it depends on the metric chosen in the tangent space. More clearly: the space of small deformations tau comes with an inner product (here L2, but one could choose another one), and the gradient .nabla g obtained depends on this inner product choice M, even though the derivative Dg is the same (they are related by Dg(tau) = < .nabla_M g | tau >_M for any tau). The choice of the metric can then be seen as a prior over desired gradient descent paths. In the paper, the deformation fields get smoothed by a Gaussian filter at some point (eq. 7), in order to be smoother: this can be interpreted as a prior (gradient descent paths should be made of smooth deformations) and as an associated inner product change (there do exist a metric M such that the gradient for that metric is .nabla_M g = S .nabla_L2 g). It is possible to favor other kind of deformations (not just smooth ones, but for instance rigid ones, etc. [and by the way this could make the link with "Manitest: Are classifiers really invariant?" by Fawzi and Frossard, BMVC 2015, who observe that a rigid motion can affect the classifier output]). If interested, you can check "Generalized Gradients: Priors on Minimization Flows" by Charpiat et al. for general inner products on deformations (in particular favoring rigid motion), and "Sobolev active contours" by Sundaramoorthi et al. for inner products more dedicated to smoothing (such as with the H1 norm). - Note: about the remark in section 3.2: deformation-induced transformations are a subset of all possible transformations of the image (which are all representable with intensity changes), so it is expected that a training against attacks on the intensity performs better than a training against attacks on spatial deformations.