S PATIALLY T RANSFORMED A DVERSARIAL E XAMPLES
Chaowei Xiao∗
University of Michigan

Jun-Yan Zhu∗
MIT CSAIL

Mingyan Liu
University of Michigan

Bo Li
UC Berkeley

Warren He
UC Berkeley

Dawn Song
UC Berkeley

arXiv:1801.02612v2 [cs.CR] 9 Jan 2018

A BSTRACT
Recent studies show that widely used deep neural networks (DNNs) are vulnerable to carefully crafted adversarial examples. Many advanced algorithms have
been proposed to generate adversarial examples by leveraging the Lp distance for
penalizing perturbations. Researchers have explored different defense methods to
defend against such adversarial attacks. While the effectiveness of Lp distance
as a metric of perceptual quality remains an active research area, in this paper
we will instead focus on a different type of perturbation, namely spatial transformation, as opposed to manipulating the pixel values directly as in prior works.
Perturbations generated through spatial transformation could result in large Lp
distance measures, but our extensive experiments show that such spatially transformed adversarial examples are perceptually realistic and more difficult to defend
against with existing defense systems. This potentially provides a new direction
in adversarial example generation and the design of corresponding defenses. We
visualize the spatial transformation based perturbation for different examples and
show that our technique can produce realistic adversarial examples with smooth
image deformation. Finally, we visualize the attention of deep networks with different types of adversarial examples to better understand how these examples are
interpreted.

1

I NTRODUCTION

Deep neural networks (DNNs) have demonstrated their outstanding performance in different domains, ranging from image processing (Krizhevsky et al., 2012; He et al., 2016), text analysis (Collobert & Weston, 2008) to speech recognition (Hinton et al., 2012). Though deep networks have
exhibited high performance for these tasks, recently they have been shown to be particularly vulnerable to adversarial perturbations added to the input images (Szegedy et al., 2013; Goodfellow
et al., 2015). These perturbed instances are called adversarial examples, which can lead to undesirable consequences in many practical applications based on DNNs. For example, adversarial
examples can be used to subvert malware detection, fraud detection, or even potentially mislead
autonomous navigation systems (Papernot et al., 2016b; Evtimov et al., 2017; Grosse et al., 2016;
Li & Vorobeychik, 2014; 2015) and therefore pose security risks when applied to security-related
applications. A comprehensive study about adversarial examples is required to motivate effective defenses. Different methods have been proposed to generate adversarial examples such as fast gradient
sign methods (FGSM) (Goodfellow et al., 2015), which can produce adversarial instances rapidly,
and optimization-based methods (C&W) (Carlini & Wagner, 2017a), which search for adversarial
examples with smaller magnitude of perturbation.
One important criterion for adversarial examples is that the perturbed images should “look like"
the original instances. The traditional attack strategies adopt L2 (or other Lp ) norm distance as a
perceptual similarity metric to evaluate the distortion (Gu & Rigazio, 2014). However, this is not
an ideal metric (Johnson et al., 2016; Isola et al., 2017), as L2 similarity is sensitive to lighting and
viewpoint change of a pictured object. For instance, an image can be shifted by one pixel, which will
lead to large L2 distance, while the translated image actually appear “the same" to human perception.
∗

indicates equal contributions

1

Motivated by this example, in this paper we aim to look for other types of adversarial examples
and propose to create perceptually realistic examples by changing the positions of pixels instead
of directly manipulating existing pixel values. This has been shown to better preserve the identity
and structure of the original image (Zhou et al., 2016b). Thus, the proposed spatially transformed
adversarial example optimization method (stAdv) can keep adversarial examples less distinguishable
from real instances (such examples can be found in Figure 3).
Various defense methods have also been proposed to defend against adversarial examples. Adversarial training based methods have so far achieved the most promising results (Goodfellow et al.,
2015; Tramèr et al., 2017; Madry et al., 2017). They have demonstrated the robustness of improved
deep networks under certain constraints. However, the spatially transformed adversarial examples
are generated through a rather different principle, whereby what is being minimized is the local
geometric distortion rather than the Lp pixel error between the adversarial and original instances.
Thus, the previous adversarial training based defense method may appear less effective against this
new attack given the fact that these examples generated by stAdv have never been seen before. This
opens a new challenge about how to defend against such attacks, as well as other attacks that are not
based on direct pixel value manipulation.
We visualize the spatial deformation generated by stAdv; it is seen to be locally smooth and virtually imperceptible to the human eye. In addition, to better understand the properties of deep neural
networks on different adversarial examples, we provide visualizations of the attention of the DNN
given adversarial examples generated by different attack algorithms. We find that the spatial transformation based attack is more resilient across different defense models, including adversarially
trained robust models.
Our contributions are summarized as follows:
• We propose to generate adversarial examples based on spatial transformation instead of
direct manipulation of the pixel values, and we show realistic and effective adversarial
examples on MNIST, CIFAR-10, and ImageNet datasets.
• We provide visualizations of optimized transformations and show that such geometric
changes are small and locally smooth, leading to high perceptual quality.
• We empirically show that, compared to other attacks, adversarial examples generated by
stAdv are more difficult to detect with current defense systems.
• Finally, we visualize the attention maps of deep networks on different adversarial examples
and demonstrate that adversarial examples based on stAdv can more consistently mislead
the adversarial trained robust deep networks compared to other existing attack methods.

2

R ELATED WORK

Here we first briefly summarize the existing adversarial attack algorithms as well as the current
defense methods. We then discuss the spatial transformation model used in our adversarial attack.
Adversarial Examples Given a benign sample x, an attack instance xadv is referred to as an
adversarial example, if a small magnitude of perturbation  is added to x (i.e. xadv = x + )
so that xadv is misclassified by the targeted classifier g. Based on the adversarial goal, attacks
can be classified into two categories: targeted and untargeted attacks. In a targeted attack, the
adversary’s objective is to modify an input x such that the target model g classifies the perturbed
input xadv in a targeted class chosen, which differs from its ground truth. In a untargeted attack, the
adversary’s objective is to cause the perturbed input xadv to be misclassified in any class other than
its ground truth. Based on the adversarial capabilities, these attacks can be categorized as whitebox and black-box attacks, where an adversary has full knowledge of the classifier and training
data in the white-box setting (Szegedy et al., 2014; Goodfellow et al., 2015; Carlini & Wagner,
2017a; Moosavi-Dezfooli et al., 2015; Papernot et al., 2016b; Biggio et al., 2013; Kurakin et al.,
2016); while having zero knowledge about them in the black-box setting (Papernot et al., 2016a;
Liu et al., 2017; Moosavi-Dezfooli et al., 2016; Mopuri et al., 2017). In this work, we will focus
on the white-box setting to explore what a powerful adversary can do based on the Kerckhoffs’s
principle (Shannon, 1949) to better motivate defense methods.
2

Spatial Transformation In computer vision and graphics literature, Two main aspects determine
the appearance of a pictured object (Szeliski, 2010): (1) the lighting and material, which determine
the brightness of a point as a function of illumination and object material properties, and (2) the
geometry, which determines where the projection of a point will be located in the scene. Most
previous adversarial attacks (Goodfellow et al., 2015) build on changing the lighting and material
aspect, while assuming the underlying geometry stays the same during the adversarial perturbation
generation process.
Modeling geometric transformation with neural networks was first explored by “capsules,” computational units that locally transform their input for modeling 2D and 3D geometric changes (Hinton
et al., 2011). Later, Jaderberg et al. (2015) demonstrated that similar computational units, named
spatial transformers, can benefit many visual recognition tasks. Zhou et al. (2016a) adopted the
spatial transformers for synthesizing novel views of the same object and has shown that a geometric
method can produce more realistic results compared to pure pixel-based methods. Inspired by these
successes, we also use the spatial transformers to deform the input images, but with a different goal:
to generate realistic adversarial examples.
Defensive Methods Following the emergence of adversarial examples, various defense methods
have been studied, including adversarial training (Goodfellow et al., 2015), distillation (Papernot
et al., 2016c), gradient masking (Gu & Rigazio, 2014) and feature squeezing (Xu et al., 2017).
However, these defenses can either be evaded by C&W attacks or only provide marginal improvements (Carlini & Wagner, 2017b; He et al., 2017). Among these defenses, adversarial training has
achieved the state-of-the-art performance. Goodfellow et al. (2015) proposed to use the fast gradient
sign attack as an adversary to perform adversarial training, which is much faster, followed by ensemble adversarial training (Tramèr et al., 2017) and projected gradient descent (PGD) adversarial
training (Madry et al., 2017). In this work, we explicitly analyze how effective the spatial transformation based adversarial examples are under these adversarial training based defense methods.

3

G ENERATING A DVERSARIAL E XAMPLES

Here we first introduce several existing attack methods and then present our formulation for producing spatially transformed adversarial examples.
3.1

P ROBLEM D EFINITION

Given a learned classifier g : X → Y from a feature space X to a set of classification outputs Y (e.g.,
Y = {0, 1} for binary classification), an adversary aims to generate adversarial example xadv for an
original instance x ∈ X with its ground truth label y ∈ Y, so that the classifier predicts g(xadv ) 6= y
(untargeted attack) or g(xadv ) = t (targeted attack) where t is the target class.
3.2

BACKGROUND : C URRENT PIXEL - VALUE BASED ATTACK METHODS

All of the current methods for generating adversarial examples are built on directly modifying the
pixel values of the original image.
The fast gradient sign method (FGSM) (Goodfellow et al., 2015) uses a first-order approximation
of the loss function to construct adversarial samples for the adversary’s target classifier g. The
algorithm achieves untargeted attack by performing a single gradient ascent step: xadv = x +
 · sign(∇x `g (x, y)), where `g (x, y) is the loss function (e.g. cross-entropy loss) used to train the
original model g, y denotes the ground truth label, and the hyper-parameter  controls the magnitude
of the perturbation. A targeted version of it can be done similarly.
Optimization-based attack (C&W) produces an adversarial perturbation for a targeted attack based
on certain constraints (Carlini & Wagner, 2017a; Liu et al., 2017) as formulated below:
min ||δ||2p

s.t.

g(x + δ) = t and

x + δ ∈ X,

where the Lp norm penalty ensures that the added perturbation  is small. The same optimization
procedure can achieve untargeted attacks with a modified constraint g(x + δ) 6= y.
3

3.3

O UR A PPROACH : S PATIALLY T RANSFORMED A DVERSARIAL E XAMPLES
Benign image 𝑥

(u

i

Adversarial image 𝑥678
*

, v (i) )

(*)

(u'() , v'() )

Bilinear
Interpolation
Estimated flow 𝑓

(Δu i , Δv (i) )

𝐮 𝐢 ,𝐯

Flow calculation
(𝐢)
(𝐢)
= 𝐮𝐚𝐝𝐯 + 𝚫𝐮(𝐢) , 𝐯𝐚𝐝𝐯 + 𝚫𝐯 (𝐢)

𝐢
𝐢

(𝐢)

(𝐮𝐚𝐝𝐯 , 𝐯𝐚𝐝𝐯 )
𝚫𝐯 (𝐢)

𝚫𝐮(𝐢)
(𝐮(𝐢) , 𝐯 (𝐢) )

Figure 1: Generating adversarial examples with spatial transformation: the blue point denotes the
coordinate of a pixel in the output adversarial image and the green point is its corresponding pixel
in the input image. Red flow field represents the displacement from pixels in adversarial image to
pixels in the input image.
All the existing approaches directly modify pixel values, which may sometimes produce noticeable
artifacts. Instead, we aim to smoothly change the geometry of the scene while keeping the original
appearance, producing more perceptually realistic adversarial examples. In this section, we introduce our geometric image formation model and then describe the objective function for generating
spatially transformed adversarial examples.
(i)

Spatial transformation We use xadv to denote the pixel value of the i-th pixel and 2D coordinate
(i)
(i)
(i)
(uadv , vadv ) to denote its location in the adversarial image xadv . We assume that xadv is transformed
(i)
from the pixel x from the original image. We use the per-pixel flow (displacement) field f to
synthesize the adversarial image xadv using pixels from the input x. For the i-th pixel within xadv
(i)
(i)
at the pixel location (uadv , vadv ), we optimize the amount of displacement in each image dimension,
with the pair denoted by the flow vector fi := (∆u(i) , ∆v (i) ). Note that the flow vector fi goes
(i)
from a pixel xadv in the adversarial image to its corresponding pixel x(i) in the input image. Thus,
(i)
the location of its corresponding pixel x(i) can be derived as xadv of the adversarial image xadv
(i)
(i)
as the pixel value of input x from location (u(i) , v (i) ) = (uadv + ∆u(i) , vadv + ∆v (i) ). As the
(u(i) , v (i) ) can be fractional numbers and does not necessarily lie on the integer image grid, we use
the differentiable bilinear interpolation (Jaderberg et al., 2015) to transform the input image with the
(i)
flow field. We calculate xadv as:
X
(i)
xadv =
x(q) (1 − |u(i) − u(q) |)(1 − |v (i) − v (q) |),
(1)
q∈N (u(i) ,v (i) )

where N (u(i) , v (i) ) are the indices of the 4-pixel neighbors at the location (u(i) , v (i) ) (top-left, topright, bottom-left, bottom-right). We can obtain the adversarial image xadv by calculating Equation
(i)
1 for every pixel xadv . Note that xadv is differentiable with respect to the flow field f (Jaderberg
et al., 2015; Zhou et al., 2016b). The estimated flow field essentially captures the amount of spatial
transformation required to fool the classifier.
Objective function Most of the previous methods constrain the added perturbation to be small
regarding a Lp metric. Here instead of imposing the Lp norm on pixel space, we introduce a new
regularization loss Lflow on the local distortion f , producing higher perceptual quality for adversarial examples. Therefore, the goal of the attack is to generate adversarial examples which can
mislead the classifier as well as minimizing the local distortion introduced by the flow field f . Formally, given a benign instance x, we obtain the flow field f by minimize the following objective:
4

f ∗ = argmin

Ladv (x, f ) + τ Lflow (f ),

(2)

f

where Ladv encourages the generated adversarial examples to be misclassified by the target classifier. Lflow ensures that the spatial transformation distance is minimized to preserve high perceptual
quality, and τ balances these two losses.
The goal of Ladv is to guarantee the targeted attack g(xadv ) = t where t is the targeted class,
different from the ground truth label y. Recall that we transform the input image x to xadv with the
flow field f (Equation 1). In practice, directly enforcing g(xadv ) = t during optimization is highly
non-linear, we adopt the objective function suggested in Carlini & Wagner (2017a).
Ladv (x, f ) = max(max g(xadv )i − g(xadv )t , κ),
(3)
i6=t

where g(x) represents the logit output of model g, g(x)i denotes the i-th element of the logit vector,
and κ is used to control the attack confidence level.
To compute Lflow , we calculate the sum of spatial movement distance for any two adjacent pixels. Given an arbitrary pixel p and its neighbors q ∈ N (p), we enforce the locally smooth spatial
transformation perturbation Lflow based on the total variation (Rudin et al., 1992):
all pixels

Lflow (f ) =

X
p

X q

||∆u(p) − ∆u(q) ||22 + ||∆v (p) − ∆v (q) ||22 .

(4)

q∈N (p)

Intuitively, minimizing the spatial transformation can help ensure the high perceptual quality for
stAdv, since adjacent pixels tend to move towards close direction and distance. We solve the above
optimization with L-BFGS solver (Liu & Nocedal, 1989).

4

E XPERIMENTAL R ESULTS

In this section, we first show adversarial examples generated by the proposed spatial transformation
method and analyze the properties of these examples from different perspectives. We then visualize
the estimated flows for adversarial examples and show that with small and smooth transformation,
the generated adversarial examples can already achieve a high attack success rate against deep networks. We also show that stAdv can preserve a high attack success rate against current defense
methods, which motivates more sophisticated defense methods in the future. Finally, we analyze the
attention regions of DNNs, to better understand the attack properties of stAdv.
Experiment Setup We set τ as 0.05 for all our experiments. We use confidence κ = 0 for both
C&W and stAdv for a fair comparison. We leverage L-BFGS (Liu & Nocedal, 1989) as our solver
with backtracking linear search.
4.1

A DVERSARIAL E XAMPLES BASED ON S PATIAL T RANSFORMATIONS

We show adversarial examples with high perceptual quality for both MNIST (LeCun & Cortes,
1998) and CIFAR-10 (Krizhevsky et al., 2014) datasets.
stAdv on MNIST In our experiments, we generate adversarial examples againsts three target models in the white-box setting on the MNIST dataset. Model A, B, and C are derived from the prior
work (Tramèr et al., 2017), which represent different architectures. See Appendix A and Table 4 for
more details about their network architectures. Table 1 presents the accuracy of pristine MNIST test
data on each model as well as the attack success rate of adversarial examples generated by stAdv on
these models. Figure 2 shows the adversarial examples against different models where the original
instances appear in the diagonal. Each adversarial example achieves a targeted attack, with the target
class shown on the top of the column. It is clear that the generated adversarial examples still appear
to be in the same class as the original instance for humans. Another advantage for stAdv compared
with traditional attacks is that examples based on stAdv seldom show noise pattern within the adversarial examples. Instead, stAdv smoothly deforms the digits and since such natural deformation
also exists in the dataset digits, humans can barely notice such manipulation.
5

Table 1: Top: accuracy of pristine data (p) on different models; bottom: attack success rate of
adversarial examples generated by stAdv on MNIST dataset.
Model
Accuracy (p)
Attack Success Rate

A
98.58%
99.95%

B
98.94%
99.98%

C
99.11%
100.00%

Target class
0 1 2 3 4 5 6 7 8 9

Target class
0 1 2 3 4 5 6 7 8 9

Target class
0 1 2 3 4 5 6 7 8 9

(a) Model A

(b) Model B

(c) Model C

Figure 2: Adversarial examples generated by stAdv against different models on MNIST. The ground
truth images are shown in the diagonal and the rest are adversarial examples that are misclassified
to the targeted class shown on the top.

stAdv on CIFAR-10 For CIFAR-10, we use ResNet-321 and wide ResNet-342 as the target classifers (Zagoruyko & Komodakis, 2016; He et al., 2016; Madry et al., 2017). We show the classification accuracy of pristine CIFAR-10 test data (p) and attack success rate of adversarial examples
generated by stAdv on different models in Table 2. Figure 3 shows the generated examples on
CIFAR-10 against different models. The original images are shown in the diagonal. The other
images are targeted adversarial examples, with the index of the target class shown at the top of the
column. Here we use “0-9” to denote the ground truth labels of images lying in the diagonal for each
corresponding column. These adversarial examples based on stAdv are randomly selected from the
instances that can successfully attack the corresponding classifier. Humans can hardly distinguish
these adversarial examples from the original instances
Table 2: Top: accuracy of pristine data (p) on different models; bottom: attack success rate of
adversarial examples generated by stAdv on CIFAR-10 dataset. The number in parentheses denotes
the number of parameters in each target model.
Model
Accuracy (p)
Attack Success Rate

ResNet32 (0.47M)
93.16%
99.56%

Wide ResNet34 (46.16M)
95.82%
98.84%

Comparison of different adversarial examples In Figure 4, we show adversarial examples that
are targeted attacked to the same class (“0” for MNIST and “airplane” for CIFAR-10), which is different from their ground truth. We compare adversarial examples generated from different methods
and show that those based on stAdv look more visually realistic compared with FGSM (Goodfellow
et al., 2015) and C&W (Carlini & Wagner, 2017b) methods.
1
2

https://github.com/tensorflow/models/blob/master/research/ResNet/ResNet_model.py
https://github.com/MadryLab/cifar10_challenge/blob/master/model.py

6

0

1

2

3

Target class
4 5 6

7

8

9

0

(a) wide ResNet34

1

2

3

Target class
4 5 6

7

8

9

(b) ResNet32

Figure 3: Adversarial examples generated by stAdv against different models on CIFAR-10. The
ground truth images are shown in the diagonal while the adversarial examples on each column are
classified into the same class as the ground truth image within that column.
FGSM
C&W
StAdv
Figure 4: Comparison of adversarial examples generated by FGSM, C&W and stAdv. (Left:
MNIST, right: CIFAR-10) The target class for MNIST is “0" and “air plane" for cifar. We generate adversarial examples by FGSM and C&W with perturbation bounded in terms of L∞ as 0.3
on MNIST and 8 on CIFAR-10.

4.2

V ISUALIZING S PATIAL T RANSFORMATION

To better understand the spatial transformation applied to the original images, we visualize the optimized transformation flow for different datasets, respectively. Figure 5 visualizes a transformation
on an MNIST instance, where the digit “0” is misclassified as “2.” We can see that the adjacent
flows move in a similar direction in order to generate smooth results.

Figure 5: Flow visualization on MNIST. The digit “0" is misclassified as “2".
7

Figure 6: Flow visualization on CIFAR-10. The example is misclassified as bird.

(a) mountain bike

(b) goldfish

(c) Maltese dog

(d) tabby cat

Figure 7: Flow visualization on ImageNet. (a): the original image, (b)-(c): images are misclassified
into goldfish, dog and cat, respectively. Note that to display the flows more clearly, we fade out the
color of the original image.

The flows are more focused on the edge of the digit and sometimes these flows move in different
directions along the edge, which implies that the object boundary plays an important role in our
stAdv optimization. Figure 6 illustrates a similar visualization on CIFAR-10. It shows that the
optimized flows often focus on the area of the main object, such as the airplane. We also observe
that the magnitude of flows near the edge are usually larger, which similarly indicates the importance
of edges for misleading the classifiers. This observation confirms the observation that when DNNs
extract edge information in the earlier layers for visual recognition tasks (Viterbi, 1998). In addition,
we visualize the similar flow for ImageNet (Krizhevsky et al., 2012) in Figure 7. The top-1 label
of the original image in Figure 7 (a) is “mountain bike". Figure 7 (b)-(d) show targeted adversarial
examples generated by stAdv, which have target classes “goldfish,” “Maltese dog,” and “tabby cat,”
respectively, and which are predicted as such as the top-1 class. An interesting observation is that,
although there are other objects within the image, nearly 90% of the spatial transformation flows
tend to focus on the target object bike. Different target class corresponds to different directions for
these flows, which still fall into the similar area.

4.3

H UMAN PERCEPTUAL STUDY

To quantify the perceptual realism of stAdv’s adversarial examples, we perform a user study with
human participants on Amazon Mechanical Turk (AMT). We follow the same perceptual study
protocol used in prior image synthesis work (Zhang et al., 2016; Isola et al., 2017). We generate 600 images from an ImageNet-compatible dataset, described in Appendix C. In our study, the
participants are asked to choose the more visually realistic image between an adversarial example
generated by stAdv and its original image. During each trial, these two images appear side-by-side
for 2 seconds. After the images disappear, our participants are given unlimited time to make their
decision. To avoid labeling bias, we allow each user to conduct at most 50 trails. For each pair of an
original image and its adversarial example, we collect about 5 annotations from different users.
In total, we collected 2, 740 annotations from 93 AMT users,. Examples generated by our method
were chosen as the more realistic in 47.01% ± 1.96% of the trails (perfectly realistic results would
achieve 50%). This indicates that our adversarial examples are almost indistinguishable from natural
images.
8

Table 3: Attack success rate of adversarial examples generated by stAdv against models A, B, and
C under standard defenses on MNIST, and against ResNet and wide ResNet on CIFAR-10.
Model
A
B
C

4.4

Def.
Adv.
Ens.
PGD
Adv.
Ens.
PGD
Adv.
Ens.
PGD

FGSM
4.3%
1.6%
4.4%
6.0%
2.7%
9.0%
3.22%
1.45%
2.1%

C&W.
4.6%
4.2%
2.96%
4.5%
3.18%
3.0%
0.86%
0.98%
0.98%

stAdv
32.62%
48.07%
48.38%
50.17%
46.14%
49.82%
30.44%
28.82%
28.13%

Model
ResNet32
wide
ResNet34

Def.
Adv.
Ens.
PGD
Adv.
Ens.
PGD

FGSM
13.10%
10.00%
22.8%
5.04%
4.65%
14.9%

C&W.
11.9%
10.3%
21.4%
7.61%
8.43%
13.90%

stAdv
43.36%
36.89%
49.19%
31.66%
29.56%
31.6%

ATTACK E FFICIENCY U NDER D EFENSE M ETHODS

Here we generate adversarial examples in the white-box setting and test different defense methods
against these samples to evaluate the strength of these attacks under defenses.
We mainly focus on the adversarial training defenses due to their state-of-the-art performance. We
apply three defense strategies in our evaluation: the FGSM adversarial training (Adv.) (Goodfellow et al., 2015), ensemble adversarial training (Ens.) (Tramèr et al., 2017), and projectile gradient descent (PGD) adversarial training (Madry et al., 2017) methods. For adversarial training
purposes, we generate adversarial examples based on L∞ bound (Carlini & Wagner, 2017a) as 0.3
on MNIST and 8 on CIFAR-10. We test adversarial examples generated against model A, B, and C
on MNIST as shown in Table 4, and similarly adversarial examples generated against ResNet32 and
wide ResNet34 on CIFAR-10.
The results on MNIST and CIFAR-10 dataset are shown in Table 3. We observe that the three
defense strategies can achieve high performance (less than 10% attack success rate) against FGSM
and C&W attacks.
These defense methods only achieve low defense performance on stAdv, which improve the attack
success rate to more than 30% among all defense strategies. These results indicate that new type
of adversarial strategy, such as our spatial transformation-based attack, may open new directions
for developing better defense systems. However, for stAdv, we cannot use Lp norm to bound the
distance as translating an image by one pixel may introduce large Lp penalty. We instead constrain
the spatial transformation flow and show that our adversarial examples have high perceptual quality
in Figures 2, 3, and 4 as well as Section 4.3.
Mean blur defense We also test our adversarial examples against the 3×3 average pooling restoration mechanism (Li & Li, 2016). Table 5 in Appendix B shows the classification accuracy of recovered images after performing 3 × 3 average filter on different models. We find that the simple 3 × 3
average pooing restoration mechanism can recover the original class from fast gradient sign examples and improve the classification accuracy up to around 70%. Carlini & Wagner have also shown
that such mean blur defense strategy can defend against adversarial examples generated by their
attack and improve the model accuracy to around 80% (2017b). From Table 5, we can see that the
mean blur defense method can only improve the model accuracy to around 50% on stAdv examples,
which means adversarial examples generated by stAdv are more robust compared to other attacks.
We also perform a perfect knowledge adaptive attack against the mean blur defense following the
same attack strategy suggested in (Carlini & Wagner, 2017b), where we add the 3 × 3 average pooling layer into the original network and apply stAdv to attack the new network again. We observe that
the success rate of an adaptive attack is nearly 100%, which is consistent with Carlini & Wagner’s
findings (2017b).
4.5

V ISUALIZING ATTENTION OF N ETWORKS ON ADVERSARIAL EXAMPLES

In addition to the analyzing adversarial examples themselves, in this section, we further characterize
these spatially transformed adversarial examples from the perspective of deep neural networks.
9

(a) mountain bike

(b) goldfish

(c) Maltese dog

(d) tabby cat

(e)

(f)

(g)

(h)

Figure 8: CAM attention visualization for ImageNet inception_v3 model. (a) the original image
and (b)-(d) are stAdv adversarial examples targeting different classes. Row 2 shows the attention
visualization for the corresponding images above.
Here we apply Class Activation Mapping (CAM) (Zhou et al., 2016a), an implicit attention visualization technique for localizing the discriminative regions implicitly detected by a DNN. We use
it to show the attention of the target ImageNet inception_v3 model (Szegedy et al., 2016)) for both
original images and generated adversarial examples. Figure 8(a) shows an input bike image and Figure 8(b)–(d) show the targeted adversarial examples based on stAdv targeting three different classes
(goldfish, dog, and cat). Figure 8(e) illustrates that the target model draws attention to the bicycle
region. Interestingly, attention regions on examples generated by stAdv varies for different target
classes as shown in Figure 8(f)–(h). Though humans can barely distinguish between the original
image and the ones generated by stAdv, CAM map focus on completely different regions, implying
that our attack can mislead the network’s attention.
In addition, we also compare and visualize the attention regions of both naturally trained and the adversarial trained inception_v3 model3 on adversarial images generated by different attack algorithms
(Figure 9). The ground truth top-1 label is “cinema,” so the attention region for the original image
(Figure 9 (a)) includes both tower and building regions. However, when the adversarial examples
are targeted attacked into the adversarial label “missile,” the attention region focuses on only the
tower for all the attack algorithms as shown in Figure 9 (b)-(d) with slight different attention region
sizes. More interestingly, we also test these adversarial examples on the public adversarial trained
robust inception_v3 model. The result appears in Figure 9 (f)–(h). This time, the attention regions
are drawn to the building again for both FGSM and C&W methods, which are close to the attention
regions of the original image. The top-1 label for Figure 9 (f) and (g) are again the ground truth
“cinema", which means both FGSM and C&W fail to attack the robust model. However, Figure 9
(h) is still misclassified as “missile” under the robust model and the CAM visualization shows that
the attention region still focuses on the tower. This example again implies that adversarial examples
generated by stAdv are challenging to defend for the current “robust” ImageNet models.

5

C ONCLUSIONS

Different from the previous works that generate adversarial examples by directly manipulating pixel
values, in this work we propose a new type of perturbation based on spatial transformation, which
aims to preserve high perceptual quality for adversarial examples. We have shown that adversarial
examples generated by stAdv are more difficult for humans to distinguish from original instances.
3

https://github.com/tensorflow/cleverhans/tree/master/examples/nips17_adversarial_competition/

10

(a) Benign

(b) FGSM

(c) C&W

(d) StAdv

(e) Benign

(f) FGSM

(g) C&W

(h) StAdv

Figure 9: CAM attention visualization for ImageNet inception_v3 model. Column 1 shows the CAM
map corresponding to the original image. Column 2-4 show the adversarial examples generated by
different methods. The visualizations are drawn for Row 1: inception_v3 model; Row 2: (robust)
adversarial trained inception_v3 model. (a) and (e)-(g) are labeled as the ground truth “cinema”,
while (b)-(d) and (h) are labeled as the adversarial target “missile.”
We also analyze the attack success rate of these examples under existing defense methods and
demonstrate they are harder to defend against, which opens new directions for developing more
robust defense algorithms. Finally, we visualize the attention regions of DNNs on our adversarial
examples to better understand this new attack.
Acknowledgment We thank Zhuang Liu, Richard Shin, and George Philipp for their valuable discussions on this work. This work is partially supported by the NSF under grants CNS-1422211
and CNS-1616575, and Berkeley DeepDrive.

R EFERENCES
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint
European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 387–
402. Springer, 2013.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE
Symposium on Security and Privacy, 2017, 2017a.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten
detection methods. arXiv preprint arXiv:1705.07263, 2017b.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep
neural networks with multitask learning. In Proceedings of the 25th international conference on
Machine learning, pp. 160–167. ACM, 2008.
Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir
Rahmati, and Dawn Song. Robust physical-world attacks on machine learning models. arXiv
preprint arXiv:1707.08945, 2017.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial
examples. In International Conference on Learning Representations, 2015.
11

Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick McDaniel.
Adversarial perturbations against deep neural networks for malware classification. arXiv preprint
arXiv:1606.04435, 2016.
Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial
examples. arXiv preprint arXiv:1412.5068, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
770–778, 2016.
Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial example
defenses: Ensembles of weak defenses are not strong. arXiv preprint arXiv:1706.04701, 2017.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly,
Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks
for acoustic modeling in speech recognition: The shared views of four research groups. IEEE
Signal Processing Magazine, 29(6):82–97, 2012.
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In International Conference on Artificial Neural Networks, pp. 44–51. Springer, 2011.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with
conditional adversarial networks. CVPR, 2017.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NIPS,
pp. 2017–2025, 2015.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and
super-resolution. In European Conference on Computer Vision, 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105,
2012.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http://www. cs.
toronto. edu/kriz/cifar. html, 2014.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world.
arXiv preprint arXiv:1607.02533, 2016.
Yann LeCun and Corrina Cortes. The MNIST database of handwritten digits. 1998.
Bo Li and Yevgeniy Vorobeychik. Feature cross-substitution in adversarial classification. In Advances in Neural Information Processing Systems, pp. 2087–2095, 2014.
Bo Li and Yevgeniy Vorobeychik. Scalable optimization of randomized operational decisions in
adversarial classification settings. In AISTATS, 2015.
Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter
statistics. arXiv preprint arXiv:1612.07767, 2016.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization.
Mathematical programming, 45(1):503–528, 1989.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In ICLR, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.
Towards deep learning models resistant to adversarial attacks. arXiv:1706.06083 [cs, stat], June
2017.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and
accurate method to fool deep neural networks. arXiv preprint arXiv:1511.04599, 2015.
12

Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal
adversarial perturbations. arXiv preprint arXiv:1610.08401, 2016.
Konda Reddy Mopuri, Utsav Garg, and R Venkatesh Babu. Fast feature fool: A data independent
approach to universal adversarial perturbations. arXiv preprint arXiv:1707.05572, 2017.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from
phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277,
2016a.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram
Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 372–387. IEEE, 2016b.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a
defense to adversarial perturbations against deep neural networks. In Security and Privacy (SP),
2016 IEEE Symposium on, pp. 582–597. IEEE, 2016c.
Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal
algorithms. Physica D: Nonlinear Phenomena, 60(1-4):259–268, 1992.
Claude E Shannon. Communication theory of secrecy systems. Bell Labs Technical Journal, 28(4):
656–715, 1949.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow,
and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on
Learning Representations, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pp. 2818–2826, 2016.
Richard Szeliski. Computer vision: algorithms and applications. Springer Science & Business
Media, 2010.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Dan Boneh, and Patrick McDaniel. Ensemble
adversarial training: Attacks and defenses. arXiv preprint arXiv:1705.07204, 2017.
Andrew J. Viterbi. An intuitive justification and a simplified implementation of the map decoder for
convolutional codes. IEEE Journal on Selected Areas in Communications, 16(2):260–264, 1998.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep
neural networks. arXiv preprint arXiv:1704.01155, 2017.
Sergey Zagoruyko and Nikos Komodakis.
arXiv:1605.07146, 2016.

Wide residual networks.

arXiv preprint

Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Conference on Computer Vision, pp. 649–666. Springer, 2016.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep
features for discriminative localization. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pp. 2921–2929, 2016a.
Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis
by appearance flow. In ECCV, pp. 286–301. Springer, 2016b.

13

A

M ODEL A RCHITECTURES
Table 4: Architecture of models applied on MNIST
A
Conv(64,5,5) + Relu
Conv(64,5,5) + Relu
Dropout(0.25)
FC(128) + Relu
Dropout(0.5)
FC(10) + Softmax

B

B
Conv(64,8,8) + Relu
Dropout(0.2)
Conv(128, 6, 6) + Relu
Conv(128, 5, 5) + Relu
Dropout(0.5)
FC(10) +Softmax

C
Conv(128,3,3) + Relu
Conv(64,3,3) + Relu
Dropout(0.25)
FC(128) + Relu
Dropout(0.5)
FC(10)+Softmax

A NALYSIS FOR M EAN B LUR D EFENSE

Here we evaluated adversarial examples generated by stAdv against the 3 × 3 average pooling
restoration mechanism suggested in Li & Li (2016). Table 5 shows the classification accuracy of
recovered images after performing 3 × 3 average pooling on different models.
Table 5: Performance of adversarial examples against the mean blur defense strategy with 3 × 3
mean filter.
Accuracy on
recovered images
3 × 3 Average Filter

C

A
59.00%

MNIST
B
64.22%

C
79.71%

CIFAR-10
Resnet32 wide ResNet34
45.12%
50.12%

A DVERSARIAL E XAMPLES FOR AN I MAGE N ET-C OMPATIBLE SET,
MNIST, AND CIFAR-10

Experiment settings. In the following experiments, we perform a grid search of hyper-parameter τ
so that the adversarial examples can attack the target model with minimal deformation. Values of τ
are searched from 0.0005 to 0.05.
ImageNet-compatible. We use benign images from the DEV set from the NIPS 2017 targeted
adversarial attack competition.4 This competition provided a dataset compatible with ImageNet and
containing target labels for a targeted attack. We generate targeted adversarial examples for the
target inception_v3 model. In Figure 10 below, we show the original images on the left with the
correct label, and we show adversarial examples generated by stAdv on the right with the target
label.
MNIST. We generate adversarial examples for the target Model B. In Figure 11, we show original
images with ground truth classes 0–9 in the diagonal, and we show adversarial examples generated
by stAdv targeting the class of the original image within that column.
CIFAR-10. We generate adversarial examples for the target ResNet-32 model. In Figure 12, we
show the original images in the diagonal, and we show adversarial examples generated by stAdv
targeting the class of the original image within that column.
Table 6 shows the magnitude of the generated flow regarding total variation (TV) and L2 distance
on the ImageNet-compatible set, MNIST, CIFAR-10, respectively. These metrics are calculated by
the following equations, where n is the number of pixels:
v
u all pixels
u1 X X
(5)
||∆u(p) − ∆u(q) ||22 + ||∆v (p) − ∆v (q) ||22 .
TV = t
n p
q∈N (p)

4
https://github.com/tensorflow/cleverhans/tree/master/examples/nips17_
adversarial_competition/dataset

14

v
u all pixels
u1 X
L2 = t
||∆u(p) ||22 + ||∆v (p) ||22
n p

(6)

Table 6: Evaluation Metric (the number in bracket is image size)
Metric
flow TV
flow L2

ImageNet-compatible (299x299)
2.85 × 10−4 ± 7.28 × 10−5
2.11 × 10−4 ± 5.19 × 10−5

MNIST (28x28)
8.26 × 10−3 ± 4.95 × 10−3
5.18 × 10−2 ± 5.66 × 10−2

15

CIFAR-10 (32x32)
2.21 × 10−3 ± 1.26 × 10−3
2.76 × 10−3 ± 2.31 × 10−3

(a) Benign image (labeled as dung beetle)

(b) Adversarial image (labeled as scale)

(c) Benign image (labeled as jeep)

(d) Adversarial image (labeled as coil)

(e) Benign image (labeled as bull mastiff)

(f) Adversarial image (labeled as American lobster)

16

(g) Benign image (labeled as buckeye)

(h) Adversarial image (labeled as goose)

(i) Benign image (labeled as thatch)

(j) Adversarial image (labeled as miniature poodle)

(k) Benign image (labeled as beaker)

(l) Adversarial image (labeled as padlock)

17

(m) Benign image (labeled as strawberry)

(n) Adversarial image (labeled as tench)

(o) Benign image (labeled as folding chair)

(p) Adversarial image (labeled as power drill)

(q) Benign image (labeled as jeep)

(r) Adversarial image (labeled as house finch)

18

(s) Benign image (labeled as vase)

(t) Adversarial image (labeled as marmoset)

Figure 10: Examples from an ImageNet-compatible set. Left: original image; right: adversarial
image generated by stAdv against inception_v3.

19

20

21

22

23

Figure 11: Adversarial examples generated by stAdv against Model B on MNIST. The original
images are shown in the diagonal; the rest are adversarial examples that are classified into the same
class as the original image within that column.

24

25

26

27

28

Figure 12: Adversarial examples generated by stAdv against a ResNet-32 on CIFAR-10. The original images are shown in the diagonal; the rest are adversarial examples that are classified into the
same class as the original image within that column.

29

