Semantics Preserving Adversarial Attacks
Authors: Ousmane Amadou Dia
Elnaz Barshan
Reza Babanezhad
Venue:  (2020)
Keywords: black-box adversarial attacks
stein variational inference
adversarial images and tex

Abstract: While progress has been made in crafting visually imperceptible adversarial examples, constructing semantically meaningful ones remains a challenge. In this paper, we propose a framework to generate semantics preserving adversarial examples. First, we present a manifold learning method to capture the semantics of the inputs. The motivating principle is to learn the low-dimensional geometric summaries of the inputs via statistical inference. Then, we perturb the elements of the learned manifold using the Gram-Schmidt process to induce the perturbed elements to remain in the manifold. To produce adversarial examples, we propose an efficient algorithm whereby we leverage the semantics of the inputs as a source of knowledge upon which we impose adversarial constraints. We apply our approach on toy data, images and text, and show its effectiveness in producing semantics preserving adversarial examples which evade existing defenses against adversarial attacks.

Under review as a conference paper at ICLR 2020
SEMANTICS PRESERVING ADVERSARIAL ATTACKS
Anonymous authors
Paper under double-blind review
ABSTRACT
While progress has been made in crafting visually imperceptible adversarial exam- ples, constructing semantically meaningful ones remains challenge. In this paper, we propose framework to generate semantics preserving adversarial examples. First, we present manifold learning method to capture the semantics of the inputs. The motivating principle is to learn the www-dimesionnl geometric summaries of the inputs via statistical inference. Then, we perturb the elements of the learned manifold using the Gram-Schmidt process to induce the perturbed elements to remain in the manifold. To produce adversarial examples, we propose an efficient algorithm whereby we leverage the semantics of the inputs as a source of knowl- edge upon which we impose adversarial constraints. We apply our approach on toy data, images and text, and show its effectiveness in producing semantics preserving adversarial examples which evade existing defenses against adversarial attacks.
INTRODUCTION
In response to the susceptibility of deep neural networks to small adversarial perturbations (Szegedy
cct al. 2014), several defenses have been proposed (Liu et al. 2019 Sinha et al. 2018 Raghunathan
ct al. 2018 Madry et al. 2017: Kolter & Wong| 2017 7). Recent attacks have, however, cast serious doubts on the robustness of these defenses (Athalye et al. 2018 Carlini & Wagner 2016). A standard way to increase robustness is to inject adversarial examples into the training inputs (Goodfellow ct al. 2014a). This method, known as adversarial training. is however sensitive to distributional shifts between the inputs and their adversarial examples (Ilya et al., 2019). Indeed, distortions, occlusions or changes of illumination in an image, to name a few, do not always preserve the nature of the image. In text, slight changes to a sentence often alter its readability or lead to substantial differences in meaning. Constructing semantics preserving adversarial examples would provide reliable adversarial training signals to robustify deep learning models, and make them generalize better. However, several approaches in adversarial attacks fail to enforce the semantic relatedness that ought to exist between the inputs and their adversarial counterparts. This is due to inadequate characterizations of the semantics of the inputs and the adversarial examples -ann--Soon -aanleess-SSnn------ al. (2018) and Zhao et al. (201 8b) confine the distribution of the latents of the adversarial examples to a Gaussian. Moreover, the search for adversarial examples is customarily restricted to
regions or conducted along suboptimal gradient directions (Szegedy et al. 2014 Kurakin et al. 2016 Goodfellow et al.) 2014b).
In this study, we introduce method to address the limitations of previous approaches by constructing adversarial examples that explicitly preserve the semantics of the inputs. We achieve this by char- acterizing and aligning the low dimensional geometric summaries of the inputs and the adversarial examples. The summaries capture the semantics of the inputs and the adversarial examples. The alignment ensures that the adversarial examples reflect the unbiased semantics of the inputs. We decompose our attack mechanism into: (i.) manifold learning, (ii.) perturbution invariance, and (iii.) adversarial attack. The motivating principle behind step (i.) to learn the low dimensional geometric summaries of the inputs via statistical inference. Thus, we present a variational inference technique that relaxes the rigid Gaussian prior assumption typically placed on VAES encoder networks (Kingma) & Welling 2014) to capture faithfully such summaries. In step (ii.), we develop an approach around the manifold invariance concept of (Roussel 2019) to perturb the elements of the learned manifold while ensuring the perturbed elements remain within the manifold. Finally, in step (iii.), we propose a learning algorithm whereby we leverage the rich semantics of the inputs and the perturbations as a source of knowledge upon which we impose adversarial constraints to produce adversarial examples. Unlike (Song et al.) 2018 Carlini & Wagner 2016 Zhao et al.) 2018b Goodfellow et al., 2014b) that resort to costly search of adversarial examples, our algorithm is efficient and end-to-end.

---

Under review as a conference paper at ICLR 2020
The main contributions of our work are thus: (i.) a variational inference method for manifold learning in the presence of continuous latent variables with minimal assumptions about their distribution, (ii.) an intuitive perturbation strategy that encourages perturbed elements of manifold to remain within the manifold, (iii.) an end-to-end and computationally efficient algorithm that combines (i.) and (ii.) to generate adversarial examples in a black-box setting. and (iv.) illustration on toy data, images and text, as well as empirical validation against strong certified and non-certified adversarial defenses.
2 PRELIMINARIES & ARCHITECTURE
Notations. Let be a sample from the input space X. with label from set of possible labels V. and D {ar 1 a set of N such samples æ. Also, let d be a distance measure on X' capturing closeness in input space, on Z, the embedding space of X', capturing semantics similarity.
Adversarial Examples. Given a classifier g. and its loss function f, an adversarial example of x is produced by maximizing the objective below over an e-radius ball around æ (Athalye et al. 2017).
x' = arg max E(g(x'). y) such that x' E ((x;e)
2'EX
Above, the search region for adversarial examples confined to maiformly-boundee ball (xx; c). In reality, however, the shape imposed on B is quite restrictive as the optimal search region may have a different topology. It is also common practice to produce adversarial examples in the input space
X'' via an exhaustive and costly search procedure (Shaham et al.] |018|] [Song et al. [0188] Zhao (ct al. 2018b Athalye et al. 2017 Carlini & Wagner 2016; Goodfellow et al. (0144)) Unlike these approaches, however, we wish to operate in Z, the lower dimensional embedding space of X, with minimal computational overhead. Our primary intuition is that 2 captures well the semantics of D.
Attack Model. Given sample x E D and its class E V, we want to construct an adversarial example x' that shares the same semanties as 2. We assume the semantics of x (resp. x') is modeled by a learned latent variable model p(2) (resp. p' p²(='), where 2, z' E Z. In this setting, observing x (resp. x²) is conditioned on the observation model p(æ[2)) (resp. p(x'|2 that is: p(x12) and x' ~ p(a'|='), with 2 ~ p(s) and x' ~ (='). We learn this model in way that d(x x') is small and g(x) = y A g(x") + y while ensuring also that d(2, 2²) is small.
Decoder pe
Rem p(2la;;0m)) p(sla
A
Zm p lx; km))
mn ~ (mm)
E
E'
O'm ~ Sui (Em)
x E D
Intuitively, we get the latent 2 ~ p(z) which en- codes the semantics of x. Then, we perturb in a way that its perturbed version e' ~ p'(2') lies in the manifold that supports p(2). We define a manifold as a set of points in Z where every point is locally Euclidean (Roussel 2019). We devise our perturbation procedure by generaliz- ing the manifold invariance concept of (Roussel 2019) to Z. For that, we consider two embedding maps h: X + Z and h': X + Z, parameterized by # and 0', as surrogates for p(=) and p(='). We assume A and 6' follow the implicit distributions p(0) and p(0'). In the following, we consider M such embedding maps h and h'
Figure 1: Architecture. The set of model parameters θ == [00m M and e' = [O',, J are sampled from the recognition networks fa and fay Given input x D. we use E' to sample the latent codes 21, ..., 2M via
O. These codes are passed to E' to learn their perturbed versions .i., Z using O', The output a' ~ po(x') (za)) generated via posterior sampling of 2' (in red).
"/I we let hxx; 0)
and Æ' = 1 (a;; 0'), we ensure that z' is in the manifold that supports p(z) by constraining d(z, z²) to be small. Then, given map dec;: Z + X, we craft x' deca(22) (2') in way that d(r, x') is small and g(x) = y A g(x') + y. Then, we say that x' is adversarial to x and preserves its semantics.
Model Architecture. To implement our attack model, we propose as framework the architecture illustrated in Figure [ Our framework is essentially a variational auto-encoder with two encoders and E' that learn the geometric summaries of D via statistical inference. We present two inference mechanisms - implicit manifold learning via Stein variational gradient descent (Liu & Wang 2016) draw instances of model parameters from
and Gram-Schmidt basis sign method (Dukes) 2014
'The reason why consider M instances of h and h' will hecome apparent in Section
2

---

Under review as a conference paper at ICLR 2020
the implicit distributions p(0) and p(0") that we parameterize E and E' with. Both encoders optimize the uncertainty inherent to embedding D in 2 while guaranteeing easy sampling via Bayesian ensembling. Finally, the decoder Ps acts generative model for constructing adversarial examples.
Threat Model. We consider in this paper a black-box scenario where we, as an attacker, have only access to the predictions of classifier g. As the attacker, we want to construct adversarial examples not knowing the intricacies of ! such as its loss function, nor having access to its gradient. We focus on this scenario because is challenging and more plausible in real-life than the white-box case. This threat model serves to evaluate both certified defenses and non-certified ones under our attack model
3 IMPLICIT MANIFOLD LEARNING
Manifold learning is based on the assumption that high dimensional data lies on or near lower dimensional manifolds in data embedding space. In the variational auto-encoder (VAE) (Kingma & Welling 2014) setting, the datapoints Z.. E 'D are modeled via a decoder 2n <> p( Zm; Φ). To learn the parameters Ø, one typically maximizes variational approximation the empirical expected log- likelihoood 1/N E
log p(xn; Φ), called evidence lower bound (ELBO), defined as:
p(z| ())(2)
Co(Ø, 17 - log
(q(]=) (1) ())pp(>( + (o) (1)
The expectation Eaja can be re-expressed as a sum of a reconstruction loss, or expected negative log- ongliikeihhod of x; and a (T((;=; w) [[)))) term. The KIL term acts as a regularizer and forces the encoder (=|aa;; la;; 1)) to follow distribution similar p(=). In VAEs, p(z) is defined
spherical Gaussian. That is, VAEs learn an encoding function that maps the data manifold to an isotropic Gaus- Gloeever,Dmenez p(z) leads to uninformative latent codes; hence to poorly learning the semantics of 'D (Zhao et al. 2017). To sidestep this issue, we minimize the divergence term [KL[q(=]x; ())lp(() o)) using Stein Variational Gradient Descent (Liu & Wang. 2016) instead of explicitly optimizing the ELBO.
sian. However, Dimenez Rezende & Mohamed (2015) have shown that the Gaussian form imposed on
Stein Variational Gradient Descent (SVGD) is a nonparametric variational inference method that combines the advantages of MCMC sampling and variational inference. Unlike ELBO (Kingma & WWelling 2014). SVGD does not confine a target distribution p(z) it approximates to simple or tractable parametric distributions. It remains yet an efficient algorithm. To approximate p(z), SVGD maintains M particles z {zm J initially sampled from a simple distribution, iteratively transports via functional gradient descent. At iteration t, each particle 21 z, is updated follows:
M
k(22" Vay logp(>"") 2t+11 2t + (z)) where ((x) M Σ
Vagok(""" 24)
where a₂ is a step-size and k(...) is a positive-definite kernel. In the equation above, each particle determines its update direction by consulting with other particles and asking their gradients. The importance of the latter particles is weighted according to the distance measure k(...). Closer particles
given higher consideration than those lying further away. The term V k(<2") ;) ) is regularizer that acts as a repulsive force between the particles to prevent them from collapsing into one particle. Upon convergence, the particles Zy will be unbiased samples of the true implicit distribution p(z).
Manifold Learning via SVGD. To characterize the manifold of D, which we denote M, we learn the encoding function q( v)) Similar to Pu al. (017)) we optimize the divergence [L(g(=)a;; {)) ø)) using SVGD. As an MCMC method, SVGD, however, induces inher- ent uncertainty we ought to capture in order to learn M efficiently. To potentially capture such uncertainty, Puu et al. (2017) use dropout. However, according to Hron et al. (2017), dropout is not principled. Bayesian methods, on the contrary, provide a principled way to model uncertainty through the posterior distribution over model parameters. Kim et al. (018)) have shown that SVGD can be cast as a Bayesian approach for parameter estimation and uncertainty quantification. Since SVGD always maintains M particles, we introduce thus M instances of model parameters @ = (0.. 1M where every 6m E θ is a particle that defines the weights and biases of a Bayesian neural network.
m==1
For large M. however, maintaining θ can be computationally prohibitive because of the memory footprint. Furthermore, the need to generate the particles during inference for each test case is undesirable. To sidestep these issues, we maintain only one (recognition) network fo that takes as
3

---

Under review as a conference paper at ICLR 2020
p( 2; ())
Algorithm 1 Inversion with one particle O.
Require: Input D
Require: Model parameters
Sample & (0 I)
n(E:n)
p(a z 0)
2: Sample 0 ~ (E)
3
Given æ, sample
p(z|2r; 0)
4: Sample J ~ p(r]2,, Φ)
Sample p(z]ū, 0)
E
x
6: Use and Σ to compute p(
Figure 2: Inversion. Process for computing the likelihood p(D]0). As the decoder gets accurate, the error |x Ø1|2 becomes small (see Algorithm and we get closer to sampling the optimal
input f. NO, I) and outputs particle Om. The recognition network Jn learns the trajectories of the particles as they get updated via SVGD. Sn serves as a proxy to SVGD sampling strategy, and refined through small number of gradient steps to get good generalization.
M
rf + arg min
Σ m=1
[(Emin')
g++1 m
with gt+1 m <- 0% + aCT(((t))
"
(2)
M
where
T(0")
[K00) 0') logr(0)) A(0) Σ M
We use the notation (V) to denote an SVGD update of o using the operator 7(-). As the particles 0 are Bayesian, upon observing D, we update the prior p(05) obtain the posterior p(0)(D) x p(D 0* ((0)) which captures the uncertainty. We refer the reader to Appendix A for formulation of p( (D) and p(D)A)). The data likelihood p(D)0)) is evaluated over all pairs (z, where D and E is dependent variable. However, is not given. Thus, we introduce the inversion process described in Figure 2 to generate such & using Algorithm | For any input x E 'D we sample its latent code: z from p(z|2;; D), which we approximate by Monte Carlo over O; that is:
D( / p(El=:0)p(01)))= =
1
Σ
M
p(z|:; 0m) where On ~ p(0(D).
(3)
M
4 PERTURBATION INVARIANCE
Here, we focus perturbing the elements of M. We want the perturbed elements to reside in M and exhibit the semantics DD that M captures. Formally, we seek linear mapping h/: M + M such that for any point In this case, we say that M is preserved under h'. Trivial examples of such mappings are linear combinations of the basis vectors of subspaces S of M called linear spans of S.
M, neighborhood Ul of is invariant under hW; that is: z' E = h (222)) u.
Rather than finding a linear span h directly, we introduce a new set of instances of model parameters O' (8' MM m==- Each 8%,, denotes the weights and biases of Bayesian neural network. Then, for any input x E 'D and its latent code ~ p(z|2; D), a point in M, we set h h'(z) = x' where z' p(='<=; D). We approximate p(='|æ: D) by Monte Carlo using O' as in Equation 3 We leverage
the local smoothness of M to learn each 8', in a way to encourage z' to reside in M in a close neighborhood of z using a technique called
Basis Sign Method.
Gram-Schmidt Basis Sign Method (GBSM). Let X be a batch of samples of D, Z,, a set of latent codes 2m p(2|2; (m)) where x X. and 0,.. θ. For any m E {1.., M..,M,,, ). we learn 6', generate perturbed versions of 2m E Zm along the directions of an orthonormal basis Um. As M is locally Euclidean, we compute the dimensions of the subspace Zn by applying Gram-Schmidt (Dukes) 2014) to orthogonalize the span of representative local points. We formalize GBSM follows:
arg min e(6m, 8'm) = Σ z/m Anc 8%
[ + 8m O (ign((.m)]] where z'm ~ p(2 (18; P'm.).
Zm
4

---

Under review as a conference paper at ICLR 2020
The intuition behind GBSM is to utilize the fact that topological spaces are closed under their basis vectors to render M invariant to the perturbations 6m. To elaborate more on GBSM. we first sample model instance f' Then, generate z'n, p(22(1x; 0 for all X. We orthogonalize Zm and find the perturbations 6m that minimizes along the directions of the basis vectors Wim E Un- We want the perturbations 8m be small. With 6n fixed, we update On by minimizing @ again. We use the notation GBSM(@', A) where A = {6m } to denote one update of O' via GBSM.
Manifold Alignment. Although GBSM confers us latent noise imperceptibility and sampling speed, o' may deviate from θ; in which case the manifolds they learn will mis-align. To mitigate this issue, we regularize each 8 O' after every GBSM update. In essence, we apply one SVGD update on O to ensure that O' follows the transform maps constructed by the particles è (Han & Liu 2017
1 441 + 0g + ac((()) where (6,)) M Σ
[<(0e, 07 Vor log p(g"( Vor 677)
(4)
We use the notation SV to refer the gradient update rule in Equation 4 In this rule, the model instances @' determine their own update direction by consulting only the particles Ô instead of consulting each other. Maintaining O' ( m=1 for large M is, however, computationally prohibitive. Thus, as in Section 3 we keep only one (recognition) network fnr that takes input Sn (0, I) and outputs 6' : Here too we refine vy' through small number of gradient steps to learn the trajectories that O' follows it gets updated via GBSM and SVGD,.
M
n+ arg min
Σ
0'4+1 1|2 [(m:n"")
where 0' +- 6 (0())
(5)
GENERATING ADVERSARIAL EXAMPLES
In this paper, a black-box scenario considered. In this scenario, we have only access to the predictions of the classifier We produce adversarial examples by optimizing the loss below. The first term is the reconstruction loss inherent VAEs. This loss accounts here for the dissimilarity between any input
DD and its adversarial counterpart z', and is constrained to be smaller than Cattack so that x' resides within an Canack -radius ball of Unless otherwise specified, we shall use norm L2 as reconstruction loss. The second term is an auxiliary log- log-likelihood loss (for g) of target class y E E {y) where y is the class of æ. This loss defines the cost incurred for failing to fool g.
E = x æ'112
+ min E'EY
1y-y log (1 (()/
such that || x'112 S Cattack
(6)
In Algorithm/ 2 we show how we unify our manifold learning and perturbation strategy into one learning procedure to generate adversarial examples without resorting an exhaustive search.
RELATED WORK
Manifold Learning. VAEs are generally used learn manifolds (Yu al. 2018 Falorsi al.
2018: Higgins et al.) 2016) by maximizing the ELBO of the data log-likelihood Alemi et al. Chen al.]/2017 Optimizing the ELBO entails reparameterizing the encoder to a Gaussian distribution (Kingma & Welling 2014). This reparameterization is, however, restrictive (Jimenez / as it may lead to learning poorly the manifold of the data (Zhao et al. 2017). To alleviate this issue, we use SVGD, similar to Pu et al. (2017). While our approach and that of Pu et al. (2017) may look similar, ours is more principled. As discussed in (Hron et al., 2017). dropout which Pu et al. (2017) use is not Bayesian. Since model instances are Bayesian, better equipped to capture the uncertainty. Capturing the uncertainty requires, however, evaluating the data likelihood. As are operating in latent space, this raises the interesting challenge of assigning target-dependent variables to the inputs. We overcome this challenge using our inversion process.
2017
Rezende
Adversarial Examples. Studies in adversarial deep learning (Athalye al. 2018 Kurakin et al. 2016 Goodfellow et al., 2214b..A.. Athalye et al.) 2017) can be categorized into two groups The first
group (Carlini & Wagner 2016 Athalye al. 2017 Moosavi-Dezfooli et al. 2016) proposes to
5

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Generating Adversarial Examples. Lines and 4 compute distances between sets keeping a one-to-one mapping between them. is adversarial to æ when Ex S fattack and y + y'.
1: function INNERTRATNINGGN n
local gradient updates of Jn Sn ,
Require: Learning rates 3
2:
- 7 3V, - SVGD, (+)]l2 A, O' t GBSM A)
apply inversion on 7 and update yf update A and O' using GBSM align O' with O and update ry
3:
4:
n/ + - I SVGD,
5:
return 77, n/
Require:
Training samples (z, y) D y
Require: Number of model instances M
Require:
Number of inner updates T
Require: Initialize weights 7, ,
recognition nets fnz foy decoder pe
Require: Initialize perturbations A = 10mmm=
m==
latent (adversarial) perturbations
Require: Learning rates e, a, a', and noise margin Eattack
6: Sample E1,..., EM from (0,
inputs to recognition nets fy, fay
7: for = 1 to I do
8:
Sample è
{0m }
m= where 6m ~ fn)(mm
9: Sample e' = {0 1M where O' ~ f.r (Em)
10: Use @ and @' in Equation to sample and z'
Sample ã p(r] Φ) and x p(='|=', O', 1el-2112 2212
clean and perturbed reconstructions
12: n. A InnerTrainingg 20, A. ;)
13:
Cs
E||2i
C. = 1 -
reconstruction losses on Ø and x'
Cz',
L >
Cattack
14: D.
C
+mnn [ log (1 P()(2''))]
otherwise
y'EY
15: f n αV 11 Lgi n/ s- n/ a L/
SGD update using Adam optimizer
16: o - Ø
<V
a [C + tz))
>> SGD update using Adam optimizer
generate adversarial examples directly in the input space of the original data by distorting, occluding or changing illumination in images to cause changes in classification. The second group (Song et al., 2018: Zhao et al. (0188), where our work belongs, uses generative models to search for adversarial examples in the dense and continuous representations of the data rather than in its input space.
Adversarial Images. Song et al. 2018 propose to construct unrestricted adversarial examples in the image domain by training conditional GAN that constrains the search region for latent code z' in the neighborhood a target 2. IZhao et al. (2018b) use also a GAN to map input images to a latent space where they conduct their search for adversarial examples. These studies are the closest ours. Unlike in (Song et al.) (2088) and (Zhao et al., (018b). however, our adversarial perturbations are learned, and we do not constrain the search for adversarial examples uniformly-bounded regions. In stark contrast to Song et al. (2018) and Zhao et al. (2018b) approaches also, where the search for adversarial examples is exhaustive and decoupled from the training of the GANs, our approach is efficient and end-to-end. Lastly, by capturing the uncertainty induced by embedding the data, characterize the semantics of the data better, allowing us thus to generate sound adversarial examples.
Adversarial Text. Previous studies adversarial text generation (Zhao et al. 201 8a Jia & Liang] 2017; Alvarez-Melis & Jaakkola, 2017 Li et al.) 2016) perform word crasures and replacements directly in the input space using
rules or heuristics, they require manual curation. Similar to us, IZhoo et al.) (2018b) propose to search for textual adversarial examples in the latent representation of the data. However, in addition the differences aforementioned for images, the search for adversarial examples is handled more gracefully in our case thanks to an efficient tradient-basee optimization method in lieu of computationally expensive search in the latent space.
EXPERIMENTS & RESULTS
Before, presented attack model whereby align the semantics of the inputs with their adversarial counterparts. As reminder, our attack model is black-box. restricted and non-ta targeted Our adversarial examples reside within catee radius ball of the inputs our reconstruction

---

Under review as a conference paper at ICLR 2020
loss, which measures the amount of changes in the inputs, is bounded by Canack (see Equation 6 We validate the adversarial examples we produce based on three evaluation criteria: (i.) manifold preservation, (ii.) adversarial strength, and (iii.) soundness via manual evaluation. We provide in Appendix A examples of the adversarial images and sentences that we construct.
7.1 MANIFOLD PRESERVATION
We experiment with 3D non-linear Swiss Roll dataset which comprises 1600 datapoints grouped in 4 classes. We show in Figure 3 on the left, the 2D plots of the manifold we learn. In the middle, we plot the manifold and its elements that perturbed and whose reconstructions
adversarial. On
the right, we show the manifold overlaid with the latent codes of the adversarial examples produced by PGD (Goodfellow al., 2014b) with eanack I 0.3. Observe in Figure 3 in the middle, how the latent codes of our adversarial examples espouse the Swiss Roll manifold, unlike the plot on the right.
Legend latent codes
Legand
lezent codes
pigil
latent codes
-25
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
Figure 3: Invariance. Swiss Roll manifold learned with our encoder E' (left), and after perturbing its elements with our encoder E' (middle) vs. that of PGD adversarial examples (right) learned using E.
7.2 ADVERSARIAL STRENGTH
Setup. As argued in AAthalye al. 2018). the strongest non-certified defense against adversarial
attacks is adversarial training with Projected Gradient Descent (PGD) (Goodfellow et al., 2014b)) Thus, we evaluate the strength of our MNIST, CelebA and SVHN adversarial examples against adversarially trained ResNets (He et al. 2015) with 40-step PGD and noise margin <attack < 0.3.
The ResNet models follow the architecture design of [Song et al. (2018). Similar to Song et al. also target the certified defenses (Raghunathan et al. 2018 Kolter & Wong 2017) with fatack = 0.1 using norm Tose as reconstruction loss. For all the datasets, the accuracies of the models target higher than 96.3%. Next, present our attack success rates and give examples of our adversarial images in Figure 4
(2018)-wal whose attack model resembles our for MNIST,
Attack Success Rate (ASR) is the percentage of examples misclassified by the adversarially trained Resnet models. For cataek 0.3, the publicly known ASR of PGD attacks on MNIST is 38.79%. However, our ASR for MNIST is 97.2%. higher than PGD. Also, with fatack 1.2, using norm Loe as reconstruction loss in Equation 6 we achieve an ASR of 97.6% against (Kolter & Wong 2017) where PGD achieves 91.6%. Finally, we achieve an ASR of 87.6% for SVHN, and 84.4% for CelebA.
7.2.1 ADVERSARIAL TEXT
Datasets. For text, we consider the SNLI (Bowman et al. 2015) dataset. SNLI consists of sentence pairs where each pair contains a premise (P) and hypothesis (H), and label indicating the relationship (entailment, neutral, contradiction) between the premise and hypothesis. For instance, the following pair is assigned the label entailment to indicate that the premise entails the hypothesis. Premise: A soccer game with multiple males playing. Hypothesis: Some men are playing sport.
Note that our results are, however, not directly comparable with (Song al. 2018) as their reported success rates for unrestricted adversarial examples, unlike ours, manually computed from Amazon MTuureers votes.
7

---

Under review as a conference paper at ICLR 2020
Table Test samples and their perturbed versions. See more examples in Appendix A.
True Input 1
P: A group of people are gathered together. H: There is group here. Label: Entailment
Adversary 1
H': There is group there. Label: Contradiction
True Input 2
female lacrosse player jumps up. H: A football player sleeps. Label: Contradiction
Adversary 2
H': A football player sits. Label: Neutral
Truc Input 3
P: A man stands in curvy corridor. H': A man runs down back alley. Label: Contradiction
Adversary 3 H': A man runs down ladder alley. Label: Neutral
Setup. We perturb the hypotheses while keeping the premises unchanged. Similar to Zhao et al. (2018b), we generate adversarial text at word level
8 1
9
2
9
2 / 7
3
2
8 1
2
9
8
2
1 1
q 3
a /
4
4
5
9
4
/
4
using a vocabulary of 11 ,,000 words. We also use ARAE (Zhao et al., (2118)) for word embedding, and CNN for sentence embedding. To generate perturbed hypotheses, we experiment with three decoders: (i.) Ps is transpose CNN, (ii.) Ps is language model, and (iii.) we use the decoder of a pre-trained ARAE (Zhao et al. (0188)) model. We detail their configurations in Appendix B.
0
4
8
3
9
4
7
9
10 6
3
3
3
/
6
3
8
B
3
5 2
6 0302526 9
050250
2 6
0
3
0
5
6
8 6
8
0
/
8
6
6
8
0
7 2
3
3
1
8
2 5
7
2
3
8
5
2 6
3
7
5
6
9
2
6
3
7
5
6
(a)
(b)
The transpose CNN generates more meaningful hypotheses than the language model and the pre- trained ARAE model although we notice some- times changes in the meaning of the original hy- potheses. We discuss these limitations more in details in Appendix A. Henceforward, we use the transpose CNN to generate perturbed hypotheses. See Table I for examples of generated hypotheses.
At Attack Success Rate (ASR). We attack an SNLI classifier that has test accuracy of 89.42% Given a pair (P. H) with label 1, its perturbed version (P, H') is adversarial if the classifier assigns the label I to (P, H), (P, H') is manually found to retain the label of (P, H), and such label differs from the one the classifier assigns to (P, H"). To compute the
(d)
2
5
5
2
5! 5
15
51
5
5 5U
5
5
50
55
5
151
50
51
ASR, we run pilot study which we detail next.
5
5
6
5.5
5
5
5
5
5
5-15
50
7.3 MANUAL EVALUATION
To validate our adversarial examples and assess their soundness vs. Song al. (2018), Zhao et al. (2018) and PGD (Madry et al. 2017) adversarial examples, we carry out a pilot study whereby we ask three yes-or-no questions: (Q1) are the adver- sarial examples semantically sound?, (Q2) are the true inputs similar perceptually or in meaning to their adversarial counterparts? and (Q3) are there any interpretable visual cues in the adversarial images that support their misclassificution?
(e)
(f)
Figure 4: Inputs (left) Adversarial exam- ples (right, inside red boxes). MNIST: (a)-(b), CelebA: (c)-(d), SVHN: (e)-(f). See Appendix A for more samples with higher resolution.
Pilot Study I. For MNIST, we pick 50 images (5 for each digit), generate their clean reconstructions, and their adversarial examples against a 40-step PGD ResNet with fattack S 0.3. We target also the certified defenses of Raghunathan et al. 22088 8) and Kolter & Wong (2017) with Cattack 0.1. For SVHN, we carry out a similar pilot study and attack a 40-step PGD ResNet. For CelebA, we pick 50 images (25 for each gender), generate adversarial examples against 40-step PGD ResNet. For all three datasets, we hand the images and the questionnaire to 10 human subjects for manual evaluation. We report in Table the results for MNIST and, in Table 4 the results for CelebA and SVHN.
We hand the same questionnaire to the subjects with 50 MNIST images, their clean reconstructions, and the adversarial examples we craft with our method. We also handed the adversarial examples
8

---

Under review as a conference paper at ICLR 2020
generated using Song al. (2018). Zhao al. (2018b) and PGD methods. We ask the subjects to assess the soundness of the adversarial examples based on the semantic features (e.g., shape, distortion, contours, class) of the real MNIST images. We report the evaluation results in Table
Table 2: Pilot Study I. Note that against the certified defenses of Raghunathan al. 2018 and Kolter Wong (2017), Song et al. (2018) achieved (manual) success rates of 86.6% and 88.6%
MNIST
QUESTIONNAIRE
40-STEP PGD
RAGHUNATHAN ET AL. 2018
KOLTER & WONG 2017
QUESTION Q1: YES
100 %
100 %
100 %
QUESTION Q2: YES
100
100 %
100 %
QUESTION Q3: No
100 %
100
Table Pilot Study The adversarial images are generated against the adversarially trained Resnets.
QUESTIONNAIRE
OUR METHOD
SONG ET AL. (2018)
ZHAD ET AL.
(0188)
PGD
QUESTION Q1: YES
100
85.9 %
97.8 %
76.7
QUESTION Q2: YES
100 %
79.3 %
89.7 %
66.8 %
QUESTION Q3: No
100
71.8 %
94.6 %
42.7
Pilot Study II SNLL Using the transpose CNN decoder Pa, generate adversarial hypotheses for the SNLI sentence pairs with the premises kept unchanged. Then, we select manually 20 pairs of clean sentences (premise, hypothesis), and adversarial hypotheses. We also pick 20 pairs of sentences and adversarial hypotheses generated this time using Zhao et al. (2018b)'s 's method against their treeLSTM classifier. We choose this classifier its accuracy (89.04%) is close to ours (89.42%) We carry out pilot study where we ask two yes-or-no questions: (Q1) are the adversarial samples semantically sound? and (Q2) are they similar the true inputs? ? We report the results in Table 4
Table 4: Pilot Studies. Some adversarial images and original ones were found blurry to evaluate.
PILOT I
PILOT II SNLI
QUESTIONNAIRE
CELEBA
SVHN
OUR METHOD
ZHAO ET AL.
2018B
QUESTION Q1: YES
100 %
951 %
83.7 %
79.6%
QUESTION Q2: YES
100 %
97 %
60.4 % N/A
56.3%
QUESTION Q3: No
100
N/A
Takeaways. As reflected in the pilot study and the attack success rates, achieve good results in the image and text classification tasks. In the image classification task, we achieve better results than PGD and Song al. (0188) both against the certified and non-certified defenses. The other takeaway is: although the targeted certified defenses are resilient to adversarial examples crafted in the input space, we can achieve manual success rates higher than the certified rates when the examples are constructed in the latent space, and the search region is unrestricted. In text classification, we achieve better results than Zhao al. (2018b) when using their treeL.TMM classifier as target model.
CONCLUSION
Many approaches in adversarial attacks fail enforce the semantic relatedness that ought to exist between original inputs and their adversarial counterparts. Motivated by this fact, we developed method tailored to ensuring that the original inputs and their adversarial examples exhibit similar semanties by conducting the search for adversarial examples in the manifold of the inputs. Our success rates against certified and non-certified defenses known to be resilient to traditional adversarial attacks illustrate the effectiveness of our method in generating sound and strong adversarial examples.
Although in the text classification task we achieved good results and generated informative adversarial sentences, the transpose CNN gets more accurate recall that it partly trained to minimize

---

Under review as a conference paper at ICLR 2020
reconstruction enror generating adversarial sentences that are different from the input sentences and yet preserve their semantic meaning becomes more challenging. the future, we intend to build upon the recent advances in text understanding to improve text generation process.
REFERENCES
Alexander A. Alemi, Ben Poole, lan Fischer, Joshua V. Dillon, Rif A. Saurous, and Kevin Murphy, An information-theoreticc analysis of deep varialee models. CoRR, abs/171 11 1.0066,, 2017. /ttpt///:/axxiiooor htsh/bbs11110 1004
David Alvarez-Melis and Tommi S. Jaakkola. causal framework for explaining the predictions of black-box sequence-to-sequeen models. CoRR, abs/1707.01943, 2017. htttt://aa/atttt
brg/abs/1707.013
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. CoRR, abs/1 707.07397, 2017. URL http:////aara.....o////////11110
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of
security: Circumventing defenses to adversarial examples. CoRR, abss/180..0040,, 2018. URL
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference. CoRR, abs/1508.05326, 2015. URL
http: org/ /abs//110 05326
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. CoRR, abs/1608. 2016. URL htt::: wwwp:iiiiworrarreooooee aba/ wwtp:///./.bbbb....ammm 04644
Liqun Chen, Shuyang Dai, Yunchen Pu, Chunyuan Li, Qinliang Su, and Lawrence Carin. Sym- metric Variational Autoencoder and Connections to Adversarial Learning. arXiv e-prints, art. arXiv:1709.01 016,, Sep 2017.
Kimberly A. Dukes. GramSchmidt Process, American Cancer Society, 2014. ISBN 9781118445112 doi: 1211002/9781111445112ooo 2t..44451112.s.00553333333 stat
ineel ibrrrr wi ley. com/
doi/abs 1Tt.11100/99991455513333
Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S. Cohen. Explorations Homeomorphic Variational Auto-Encoding, arXiv e-prints, art. arXiv:1807.04689, Jul 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2680, 2014a.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harvesting adversarial examples. CoRR abs/1412.6572, 2014b.
Jun Han and Qiang Liu. Stein variational adaptive importance sampling. Conference on Uncertainty in Artificial Intelligence (UAI), 2017.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL http::////// http://wxxvvooooraaaaa// org 2ttp.//.rr///b//112..33333
Irina Higgins, Loie Matthey, Xavier Glorot, Arka Pal, Benigno Uria, Charles Blundell, Shakir Mohamed, and Alexander Lerchner. Early Visual Concept Learning with Unsupervised Deep Learning. arXiv e-prints, art. arXiv:1606.05579, Jun 2016.
Jiri Hron, Alexander G. de G. Matthews, and Zoubin Ghahramani. Variational Gaussian Dropout not Bayesian. arXiv e-prints, art. 1iii:1711..0298 Nov 2017.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Alek- sander Madry. Adversarial Examples Are Not Bugs, They Are Features. arXiv e-prints, art. arXiv.1905.02115. May 2019.
10

---

Under review as a conference paper at ICLR 2020
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. CoRR, abs/1707.07328, 2017. htt.ttpp:/axxxi wttp:///...//////0...... 1737/
Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. arXiv e-prints, art. arXiv: 150.005770, May 2015.
Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn.
Bayesian model-agnostic meta-learning CoRR, abs/1806.03836, 2018. URL httthhh/pa/attt Erq/abs/ 1806. 03836
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. International Conference
on Learning Representations (ICLR), 2014.
J. Zico Kolter and Eric Wong. Provable defenses against adversarial examples via the convex outer adversarial polytope. CoRR, abs/171 2017. URL//tttp//aaaaa..rrr//////// 00851
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. CoRR, abs/1607.02533, 2016.
02533
02533
Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation
erasure. CoRR, abs/1612.08220, 2016. (kt.:tttp.//arxrt...r///////2222
Chen Liu, Ryota Tomioka, and Volkan Cevher. On per-iiiying bound against adversarial attacks. CoRR, abs/1903.06603 2019. URL http:///araiiv.orrr/////000
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose Bayesian inference algorithm. Neural Information Processing Systems (NIPS), 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. CoRR, abs/1706.06083 2017, URL
http:://axiia wwwp://wwwrro..ooooo.ooooo.oo.a /btprg/aab///0 06683
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. CoRR, abs/1610.08401, 2016. URL http::/aaxiiv.rrg /bs/
1610.08401
Yunchen Pu, Zhe Gan, Ricardo Henao, Chunyuan Li, Shaobo Han, and Lawrence Carin. VAE learning via Stein variational gradient descent. Neural Information Processing Systems (NIPS), 2017.
Alec Radford. Improving language understanding by generative pre-training. In arXiv, 2018.
Aditi Raghunathan, Jacob Steinhardt, and Perey Liang. Certified defenses against adversarial examples. CoRR, 0bbs/1801.0934,, 2018. URL httppp//aaar.....orrraa/////1/1/ httts//.orr/aab///00.003333
Mare R Roussel. Invariant manifolds. In Nonlinear Dynamics, 2053-2571, pp. 6-1 to 20. Morgan & Claypool Publishers, 2019. ISBN 978-11-643274464-5 doi: 10. 100000/0003-2257//00001 URL htt.httpp/dxx.....rrrg. doi w.t.:/..../....../..... 1tw.11188222332111111811
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of supervised models through robust optimization. Neurocomputing, 307:195-20 204, 2018.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifiable distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018. URL
net fortm/form?22 id= 6
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, 3-8 December 2018, Montréal, Canada,, pp. 8322 8333, 2018. htt::: /ttptt:::/ppppp...pe wwwppeerr.niss.ccoom 8052-const ruct inw.iiinnnnnnat........... htt.aa/tttrrarraaaaaaeaaaaaeeeee

---

Under review as a conference paper at ICLR 2020
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014. / .rg...ooo../.oooooo...o...ooo.rr httpg//bs/131.....
/
6199
Bing Yu, Jingfeng Wu, and Zhanxing Zhu. Tangent-normal adversarial regularization for semi- absp///bb//88
supervised learning. CoRR, abs/1806 808.00088, 2018. URL
06088
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. Adversarially regularized autoencoders. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5902-5911, Stockholmsmssan Stockholm Sweden, 10-15 Jul 2018a. PMLR. URLhtEn
prpzoceddings.ml..ppee.... prw.s/ /v80/ zhw.lbb...bb.o
wttppp/ee..//////......bbbb... html
Shengjia Zhao, Jiaming Song, and Stefano Ermon. InfoVAE: Information maximizing variational autoencoders. CoRR, abs/1 706.02262, 2017. URL
http:////aaa11...r//////766... 02262
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. In
International Conference on Learning Representations, 2018b. URL httptttp//pppeeei.....
12

---

Under review as a conference paper at ICLR 2020
APPENDIX A: DISCUSSION & ADVERSARIAL EXAMPLES
Posterior Formulation. Similar to (Kmm et al. 2018). we formalize D() for every 0 E θ as:
p(0 pD = II p( ())()) where I E D and is generated using Algorithm
(a.5)
II ((()fw (W( x-) Ga)mmaaaa Gamma(X|a (u,.5)
For every 0' E O', we compute (D) the same way. Note that (resp. 0') consists in fact of network parameters W ~ fn (resp. W fry) and scaling parameters y and A. For notational simplicity, we used before the shorthands 0 ~ fy and 0' ~ fay. The parameters y and À are initially sampled from Gamma distribution and updated as part of the learning process. In our experiments, we set the syper-parameten of the Gamma distributions a and b to 1.0 and 0.1, and a' and 8' to 1.0.
Latent Noise Level. We measure the amount of noise A we inject into the latent codes of our inputs by computing the average spectral norm of the latent codes of their adversarial counterparts. The input changes are captured by our reconstruction loss which is bounded by Catack (see Equation For MNIST, CelebA, and SVHN, the noise levels are 0.004 + 0.0003, 0.026 + 0.005, and 0.033 = 0.008.
The takeaways are: (i.) they are imperceptible, and (ii.) the distributions that O and @' follow are similar. To validate (ii.), we compute the marginals of clean and perturbed latent codes randomly sampled from θ and O/ As shown in Figure the marginal distributions overlap relatively well.
(a) MNIST
(b) CelebA
(c) SVHN
(d) SNLI
Figure 5: Marginal distributions of clean (blue) and perturbed (red) latent codes over few minibatches.
Discussion. We discuss the choices pertaining the design of our approach and their limitations. We discuss also the evaluation process of our approach against (Song et al. 2018 Zhao et al. 2018b).
Space/Time Complexity: As noted in (Jimenez Rezende & Mohamed, 2015). the Gaussian prior assumption in VAEs might be too restrictive to generate meaningful enough latent codes (Zhao al. 2017). To relax this assumption and produce informative and diverse latent codes, we used SVGD. To generate manifold preserving adversarial examples, we proposed GBSM. Both SVGD and GBSM maintain set of M model instances. As ensemble methods, both inherit the shorteomings of ensemble models most notably in space/time complexity. Thus, instead of maintaining 2 M model instances, we maintain only fn and fy from which we sample these model instances. We experimented with M set to 2, 5, 10 and 15. As M increases, notice some increase in sample quality at the expense of longer runtimes. The overhead that occurs as M takes on larger values reduces, however, drastically during inference as need only fy to sample the model instances 0', E O' in order to construct adversarial examples. One way to alleviate the overhead during training enforce weight-sharing for Om E O and O', O'. However, we did not try this out.
Preserving Textual Meaning. To construct adversarial text, we experimented with three architecture designs for the decoder ps: (i.) transpose CNN, (ii.) language model, and (iii.) the decoder of a pre-trained ARAE model (Zhao et al. 2018a). The transpose CNN generates more legible text than the other two designs although we notice sometimes some changes in meaning in the generated adversarial examples. Adversarial text generation is challenging in that small perturbations in the latent codes can go unnoticed at generation whereas high noise levels can render the outputs nonsensical. To produce adversarial sentences that faithfully preserve the meaning of the inputs, we need good sentence generators, like GPT (Radford] 2018), trained on large corpora. Training such large language models requires however time and resources. Furthermore, in our experiments, considered only a vocabulary of size 10,000 words and sentences of length no more than 10 words to align our evaluation with the experimental choices of (Zhao et al. 2018h).
13

---

Under review as a conference paper at ICLR 2020
Measuring Perceptual Quality is desirable when the method relied upon generate adversarial examples uses GANs or VAEs; both known to produce often samples of limited quality. As Song et al. (018)) perform unrestricted targeted attacks their adversarial examples might totally differ from the true imputs and /Zhao et al. (2018b) do not target certified defenses, a fair side-by-side comparison of our results and theirs using metrics like mutual information frechet inception distance, seems unachievable. Thus, to measure the quality of our adversarial examples and compare our results with (Song et al. 2018) and (Zhao et al. 2018b), carried out the pilot study.
ADVERSARIAL IMAGES: CELEBA
Table 5: CelebA samples, their clean reconstructions, and adversarial examples (in red boxes).
INPUTS
CLEAN RECON- STRUCTIONS
ADVERSARIAL. EXAMPLES
14

---

Under review as a conference paper at ICLR 2020
ADVERSARIAL IMAGES: SVHN
Here, we provide few random samples of non-targeted adversarial examples we generate with our approach the SVHN dataset well as the clean reconstructions.
Table SVHN. Images in red boxes are all adversarial.
27
47
67
7
8
8
8
8 85
8 8477
7
7
7
17-
7
47
47
8
8
8
28
8
68
88
2
7
777
27
57
8
8
8
8
88
8
38
7
7
7
37
4787
7
8
87
6
i8
INPUTS
7 61
7
1
7
7
8
8
8
8
38338
88
7
07
27.
17
175
4
8
8
8
8
8
858
7
179
7
27
7
7
18
8
8
8
8
8
8
7
72
11
7
178
8
8
8
8
8
8818
8
7
1T
7
57
7
8 6
8
18
B
T8
7 7
7 77
47
17
8
8
8
8
38
8
2 7 777
Y 27157
8
8
L
8
8
i
8
8
CLEAN RECON-
7
7 7. 7.37
178
7
7
8
K
8
8
7 61
7 TINT
T 7 7
8
5
6
20
88
STRUCTIONS
7
07 27
17S
A
4
6
6
B
8
6
8
58
7
77 7
1
16
8
8
8
8
8
8
8 8
7
7
17
7
7 178
0
6
8
8
1
0
7
7 57
7
8 6
8
10
b
6
7
7
7
7
1
17
:7
X
8
8
8
8
8
8
6
2
7
727
/
27
57
8
8
8
8
8
8
8
7
7
7:37
37
1787
77
8
8
ADVERSARIAL EXAMPLES
761
61
1
1NA7
7
70
7
8
5
7
8
20
8
7
07
27:
175
7
4
4
6
B
8
6
8
68
7
27
18
8
8
8
8
8
8
8
7
7171
17
7
7
178
8
8
1
18
8
15

---

Under review as a conference paper at ICLR 2020
ADVERSARIAL IMAGES: MNIST
Here, we provide few random samples of non-targeted adversarial examples we generate with approach the MNIST dataset well the clean reconstructions. Both the reconstructed and the adversarial images look realistic although we notice some artifacts on the latter. Basic Iterative Methods (Kurakin al., 2016). among others, also suffer from this, Because the marginal distributions of the latent codes of the ipputs and their perturbed versions overlap, we conclude that indeed the adversarial images preserve the manifold of the inputs (see Figure 5).
Table 7: MNIST. Images in red boxes are all adversarial.
8
8
2
7
4
7
6
5
9
5
4
0
2
9
5 9 2 4
6
5
5
2
4
/ 3 4
3
5
4
5
6 6 7
6 7
6
8 7
7
7
9 9
/
4
8
2 7667097
3
0
3
INPUTS
4 4 6
5 5
5
4 4 9
2 559357/ 0
2 6 0
8 7
8
5
7 1 0
0 5 9 2 4 9
9
80100
6 9
5 4
8
3 2
7 6 6 5 5 2 1/34/3 6 176 3
8 8 9
2
7
4
5 9
S
4
0
2 5
9 7
3 4
3
5
4
/ 7
7 9 9
/ 4
6 7 0
8
2 6 76670979 3
0 9 9
3
CLEAN
RECON-
4 4 4477829
6
5 5
4
STRUCTION
2 3 0
2 6 6
3 9 060390
5 8 7
8
2
3
5
q
5
7 00
7 /
09 50 9 2 4 9 7 9
8
O
/
0
6
9
5
4
8
3
2
C
8
9
2
7 9
4 /
7
6
5
9
S
4
2
9 6 6 7 1 8 0
6
5
5
2
3 7 9
4
3
1
5
4
5
6
7
6
3
2 0
8 4 2
7
7
9
9
/
4
8
2
6
6
9
3
6
9
5
/
9
3 0 5
ADVERSARIAL EXAMPLES
4
14700
4
7
1 8
6
5
3
5
/ 3
9
2
5 3
0
2
6
0
9 1
8
3 3
665 5 0
5 995566
7 0 7
8
O
1
0
9
5 84832
16

---

Under review as a conference paper at ICLR 2020
ADVERSARIAL TEXT: SNLI
Table 8: Examples of adversarially generated hypotheses with the true premises kept unchanged.
white dog running through snow. H: A CAT STALKING THROUGH THE SNOW. Label: CONTRADICTION
TRUE INPUT
ADVERSARY
H': CAT
Label: NEUTRAL
Threc dogs are scarching for something outside.
TRUE INPUT
H: THERE ARE FOUR DOGS.
Label: CONTRADICTION H:: THERE ARE FIVE DOGS. Label: NEUTRAL
ADVERSARY
P: A waterskis while attached parachute.
TRUE INPUT
H: BULLDOZER KNOCKS DOWN A HOUSE. Label: CONTRADICTION
ADVERSARY
H:: BULLDOZER KNOCKS DOWN CAGE. Label: ENTAILMENT
P: A little girl playing with flowers.
TRUE INPUT
H: LITTLE GIRL PLAYING WITH BALL. Label: CONTRADICTION
ADVERSARY
H': LITTLE GIRL IS RUNNING WITH BALL. Label: NEUTRAL
P: People stand in front of chalkboard.
TRUE INPUT
H: PEOPLE STAND OUTSIDE PHOTOGRAPHY STORE.
Label: CONTRADICTION
ADVERSARY
HI' PEOPLE STAND IN FRONT OF A WORKSHOP. Label: NEUTRAL
P: Musician entertaining his audience.
TRUE INPUT
H: THE WOMAN PLAYED TRUMPET.
Label: CONTRADICTION
H': THE WOMAN PLAYED THE DRUMS. Label: ENTAILMENT
ADVERSARY
P: A kid slip and slide.
TRUE INPUT 7
H: SMALL CHILD INSIDE EATING THEIR DINNER.
Label: CONTRADICTION
ADVERSARY
H': A SMALL CHILD IS EATING THEIR DINNER. Label: ENTAILMENT
P: A deer jumping over a fence.
TRUE INPUT
H: DEER LAYING IN THE GRASS. Label: CONTRADICTION PONY LAYING IN THE GRASS. Label: ENTAILMENT
ADVERSARY
P: Two vendors are on a curb selling halloons.
TRUE INPUT 9
H: THREE PEOPLE SELL LEMONADE THE ROAD SIDE.
Label: CONTRADICTION
H': THREE PEOPLE SELL ARTWORK BY THE ROAD SIDE. Label: MENAILMENT
ADVERSARY
17

---

Under review as a conference paper at ICLR 2020
Table 9: Some generated examples deemed adversarial by our method that are not.
P: A man is operating type of vessel.
TRUE INPUT
H: DOG KENNEL.
Label: CONTRADICTION
GENERATED
H': DOG IN DISGUISE. Label: CONTRADICTION
P: A skier.
TRUE INPUT
H: SOMEONE Label: ENTAILMENT
GENERATED
H': MAN IS SKIING. Label: NEUTRAL
P: This is bustling city street. WALKING ALONG.
TRUE INPUT
H: THERE
Label: ENTAILMENT
GENERATED
H': THERE ARE LOT GIRLS WALKING ALONG. Label: NEUTRAL
P: A soldier is looking out of window.
TRUE INPUT
H: THE PRISONER'S CELL WINDOWLESS. Label: CONTRADICTION
GENERATED
H': THE PRISONER'S HOME IS WINDOWLESS, Label: CONTRADICTION
P: Four people sitting on low cement ledge.
TRUE INPUT
H: THERE ARE FOUR PEOPLE.
Label: ENTAILMENT THERE ARE SEVERAL PEOPLE. Label: NEUTRAL
GENERATED
H::
P: Three youngsters shovel a huge pile of snow.
TRUE INPUT
H: CHILDREN WORKING TO CLEAR SNOW.
Label: ENTAILMENT
H': KIDS WORKING TO CLEAR SNOW. Label: NEUTRAL
GENERATED
P: Boys at an amphitheater.
TRUE INPUT 7
H: BOYS AT SHOW.
Label: ENTAILMENT
GENERATED
H': BOYS IN A SHOW. Label: NEUTRAL
P: Male child holding a yellow balloon.
TRUE INPUT
H: BoY HOLDING BIG BALLOON. Label: NEUTRAL
H': BOY HOLDING LARGE BALLOON.
GENERATED
Label: NEUTRAL
Women their swimsuits sunbathe on the sand. H: WOMEN UNDER THE SUN ON THE SAND. Label: ENTAILMENT FAMILY UNDER THE ON THE SAND. Label: NEUTRAL
TRUE INPUT
GENERATED
18

---

Under review as a conference paper at ICLR 2020
APPENDIX B: EXPERIMENTAL SETTINGS
Table 10: Model Configurations + SNLI Classifier + parametrra.etrrs..
NAME
CONFIGURATION
INPUT DIM: 50,
HIDDEN LAYERS: [60, 70J.
OUTPUT DIM: NUM WEIGHTS BIASES 6m
RECOGNITION NETWORKS
INPUT DIM: 50,
HIDDEN LAYERS: [60, 70], OUTPUT DIM: NUM WEIGHTS BIASES &...
INPUT DIM: 28 28 (MNIST), 64 x 64 (CELEBA), 32 32 (SVHN), 300 (SNLI) HIDDEN LAYERS: [40, 40] OUTPUT DIM (LATENT CODB): 100
PARTICLES Om
MODEL INSTANCES
INPUT DIM: 28 28 (MNIST), 64 (CELEBA). 32 x 32 (SVHN), 100 (SNLI) HIDDEN LAYERS: [40, 40] OUTPUT DIM (LATENT CODE): 100
PARAMETERS 6r,
INPUT DIM: 28 x 28 1 (MNIST), 64 x 64 3 (CELEBA), 100 (SNLI)
32 32 3 (SVHN), HIDDEN LAYERS: [40, 40] OUTPUT DIM: 28 28 (MNIST), 64 64 (CELEBA), 32 x 32 (SVHN). 100 (SNLI)
FEATURE EXTRACTOR
CELEBA SVHN: [FILTERS: 64, STRIDE: 2,
TRANSPOSE CNN
KERNEL: 5Jx 3
FOR SNLI: [FILTERS: 64, STRIDE: 1,
DECODER
KERNEL: 5] 3
LANGUAGE MODEL
VOCABULARY SIZE: 11,000 WORDS MAX SENTENCE LENGTH: 10 WORDS
SNLI CLASSIFIER
INPUT DIM: 200. HIDDEN LAYERS: [100, 100. 100), OUTPUT DIM:
LEARNING RATES
ADAM OPTIMIZER (6 = 10 00
10
MORE SETTINGS
BATCH SIZE: 64, INNER-UPDATES: 3, TRAINING EPOCHS: 1000, M =
19


Reviews:
[{"review_id": "rJx-BNkjFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach to generating adversarial examples that preserve the semantics of the input examples. To do so, the approach reconstructs the manifold where the input examples lie and then generates new examples by perturbing the elements of the manifold so as to ensure the new elements remain in the manifold to preserve the semantics of the elements.\n\nIn the presented system the manifold is learned by means of Stein Variational Gradient Descent, while the perturbation is made by applying the Gram-Schmidt process which ensures that the perturbed elements still reside in the manifold.\n\nTo generate adversarial examples the approach presented in the paper considers a scenario in which only the predictions of the classifier are known, to be able to compute and optimize the loss function.\n\nThe presented approach has been tested on toy examples regarding images (both numbers from MNIST or SVHN and images from CelebA datasets) and texts (SNLI dataset).\n\nThe performance presented in the paper is promising. The results show that the manifold shape is preserved while creating perturbed elements. The system also achieves good results in terms of adversarial success rate, which however I believe should be called \"attack success rate\", a term widely used in literature.\n\nThe paper is well written and seems to me to be mathematically sound. I have very few comments on the paper which does not present any important lack in my opinion.\n\nI suggest moving algorithm 2 to a new page in order to have the whole pseudo-code together.\nMoreover, I have found two typos, one on page 4 in the last equation where z'_m I think it is wrongly written as z^'_m (the ' is far from the z), and one on page 6 in the \"adversarial examples\" paragraph where the word \"data\" is misspelt as \"dat\".", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "Byg5fBqTKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n====== Updates ======\nI appreciate the authors' time and effort in the response. I have read the rebuttal, but I am not convinced by the authors' argument on using L2 (or L_\\infty) constraints. No matter whether L2 or L_\\infty constraint is used, the authors' method is not directly comparable to methods in Song et al. (2018), making the results in Table 2 and Table 3 meaningless and confusing. \n\n- Song et al. (2018) indeed constraints the search region of latent code to be within a small L2 ball of a randomly sampled anchor latent code. However, this anchor latent code is not directly related to any given image in the dataset, and therefore the generated adversarial examples are not close to any existing image. In contrast, the authors' attack is still basically a norm-bounded attack, which is not directly comparable to the unrestricted attack in Song et al. (2018).\n\n- Song et al. (2018) is a white box attack, while the attack in this paper is black box.\n\n====== Original review =======\n\nThis paper proposes to generate semantic preserving adversarial examples by first learning a manifold and then perturbing data along the manifold. In this way the generated adversarial examples can be semantically close to the original clean examples, and the perturbations can be hopefully more natural. For manifold learning, the authors propose to use a similar approach to that proposed in Pu et al. (2017), which uses SVGD to train a VAE. After the VAE is trained, the authors use GBSM to train a model to produce semantic adversarial examples efficiently.\n\nI have many concerns for this paper:\n\n- The approach is not well motivated. It is unclear why using a fully Bayesian framework and employing SVGD to learn the VAE model is preferred for conducting semantic adversarial attacks. Many choices in the algorithm seem to be arbitrary, and there are many approximations in the method whose accuracies have no guarantees. For example, the recognition networks  are used to approximate the updated parameters of the encoder from SVGD. Sampling from the posterior distribution of z is approximated by first doing Monte Carlo over \\Theta. For \"manifold alignment\" another recognition network is used to approximate the updates from SVGD. It is hard to predict how those approximation errors accumulate when all pieces are combined together to form a very complicated algorithm.\n\n- In Equation (6) the authors hard-constrain the generated adversarial example such that they cannot differ from the original data by some pre-specified l_2-norm. This leads to many unfair comparisons in the experiments:\n\n    1. The authors compare their approach to other attacking methods on the success rates of attacking Madry's model and Kolter & Wong's certified model. However, both Madry and Kolter & Wong's model are for attacks using the l_infinity norm. It is unfair that the authors' attack uses l_2 norm. In fact, it is known that models robust to l_infinity norm attacks are generally not robust to attacks using other norms. \n\n    2. The authors also compare their approach to methods in Song et al. (2018) and Zhao et al. (2018a). However, the two previous approaches did not directly constrain the distance between generated adversarial examples and the corresponding clean inputs. Therefore, when using human evaluation to assess the image quality of generated adversarial examples, the two previous methods are naturally at a huge disadvantage. In stark contrast, the authors' adversarial images are constrained to be close to the corresponding unperturbed images under a small l_2 norm, which naturally have higher image quality.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJehXzRNKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Major caveat: I have published in the area of adversarial attacks on NLP models, but the specifics of the methods presented in this paper are quite outside of my expertise, and I do not have time to become familiar with them for this review.  I hope there are other reviewers that are more qualified than I am to check the specifics of the methods.\n\nThis paper presents a new technique for generating adversarial examples, by first learning the data manifold in an embedding space, then finding an adversarial example that lies on the manifold.  I like this idea, it intuitively seems like a promising method for obtaining semantically meaningful adversarial examples.\n\nAs I said above, I do not feel qualified to review whether the method should _theoretically_ accomplish its goals, so my judgment of this paper is on the intuition behind the idea (which I like), and the results that I can see (which are less promising).  In order to have a \"semantics preserving\" attack, the method needs to (1) remain on the data manifold, and (2) not change the label a human would give to the input.\n\nFor (1), this appears to have been accomplished on most datasets, though it seems pretty hard to argue that the artifacts seen in the MNIST examples shown are on the data manifold - there are no such artifacts in any of the inputs, or in the clean reconstruction.  How do the authors claim that this actually did a reasonable job of staying in the data manifold?\n\nFor (2), most of the images do indeed look like they should retain their human labels, which is good (but also not hard for adversarial images).  Almost all of the textual examples, however, have correct predictions from the model after the adversarial change to the input.  You can't really argue that these are \"semantics preserving\", or even \"successful attacks\", as they change the expected input label.  This is why semantics-preserving attacks are so hard in NLP, and I don't think that this method has accomplished its goal here at all, at least for text.  The authors should consult with experts in NLP before making claims about successfully constructing semantics preserving attacks on NLP models.\n\nI'm pretty on the fence about this paper, as I like the intuition, and the method appears to work reasonably well for vision.  It does not work as claimed for text, however, and that should be fixed before this paper is published (either with softened claims or with better results).  Hopefully people from other perspectives can pipe in and give a more clear picture on this paper.\n\nEDIT: See discussion below for my justification for reducing my score from a 3 to a 1.\n\nEDIT 11/14: The authors' revisions have satisfied my concerns about how the NLP attacks are described.  I'm a little bit nervous about how the examples were changed - it seems that nothing changed about the method itself, so the authors probably cherry-picked better examples - but that's not sufficiently worrying to me to justify rejection.  The pilot study is also quite weak, as the number of inputs that were evaluated was only 20, and the questions presented don't appear to ask about changes in the label.  I don't know how you could get 100% on that given the examples that I saw in the previous version of the paper.  This is all to say that I don't think the NLP attacks are actively problematic anymore, as they were previously, now they are just weak.  The main contribution here is the technical contribution, anyway, so weaker results on one of the datasets tested is not a deal-breaker to me.  Assuming the technical contributions pass muster (which, as I said, I don't really feel qualified to judge), I'm satisfied with this paper as it is now.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a method for generating adversarial examples from images and text such that they maintain the semantics of the input.

The reviewers saw a lot of value in this work, but also some flaws.  The review process seemed to help answer many questions, but a few remain: there are some questions about the strength of the empirical results on text after the author's updates. Wether the adversarial images stay on the manifold is questioned (are blurry or otherwise noisy images "on manifold"?).  One reviewer raises good questions about the soundness of the comparison to the Song paper.

I think this review process has been very productive, and I hope the authors will agree.  I hope this feedback helps them to improve their paper.
Decision: Reject