Title: SEMANTICS PRESERVING ADVERSARIAL ATTACKS

Abstract: While progress has been made in crafting visually imperceptible adversarial examples, constructing semantically meaningful ones remains a challenge. In this paper, we propose a framework to generate semantics preserving adversarial examples. First, we present a manifold learning method to capture the semantics of the inputs. The motivating principle is to learn the low-dimensional geometric summaries of the inputs via statistical inference. Then, we perturb the elements of the learned manifold using the Gram-Schmidt process to induce the perturbed elements to remain in the manifold. To produce adversarial examples, we propose an efficient algorithm whereby we leverage the semantics of the inputs as a source of knowledge upon which we impose adversarial constraints. We apply our approach on toy data, images and text, and show its effectiveness in producing semantics preserving adversarial examples which evade existing defenses against adversarial attacks.

1. INTRODUCTION:

In response to the susceptibility of deep neural networks to small adversarial perturbations (Szegedy et al., 2014) , several defenses have been proposed (Liu et al., 2019; Sinha et al., 2018; Raghunathan et al., 2018; Madry et al., 2017; Kolter & Wong, 2017) . Recent attacks have, however, cast serious doubts on the robustness of these defenses (Athalye et al., 2018; Carlini & Wagner, 2016) . A standard way to increase robustness is to inject adversarial examples into the training inputs (Goodfellow et al., 2014a) . This method, known as adversarial training, is however sensitive to distributional shifts between the inputs and their adversarial examples (Ilyas et al., 2019) . Indeed, distortions, occlusions or changes of illumination in an image, to name a few, do not always preserve the nature of the image. In text, slight changes to a sentence often alter its readability or lead to substantial differences in meaning. Constructing semantics preserving adversarial examples would provide reliable adversarial training signals to robustify deep learning models, and make them generalize better. However, several approaches in adversarial attacks fail to enforce the semantic relatedness that ought to exist between the inputs and their adversarial counterparts. This is due to inadequate characterizations of the semantics of the inputs and the adversarial examples - Song et al. (2018) and Zhao et al. (2018b) confine the distribution of the latents of the adversarial examples to a Gaussian. Moreover, the search for adversarial examples is customarily restricted to uniformly-bounded regions or conducted along suboptimal gradient directions (Szegedy et al., 2014; Kurakin et al., 2016; Goodfellow et al., 2014b) .

In this study, we introduce a method to address the limitations of previous approaches by constructing adversarial examples that explicitly preserve the semantics of the inputs. We achieve this by characterizing and aligning the low dimensional geometric summaries of the inputs and the adversarial examples. The summaries capture the semantics of the inputs and the adversarial examples. The alignment ensures that the adversarial examples reflect the unbiased semantics of the inputs. We decompose our attack mechanism into: (i.) manifold learning, (ii.) perturbation invariance, and (iii.) adversarial attack. The motivating principle behind step (i.) is to learn the low dimensional geometric summaries of the inputs via statistical inference. Thus, we present a variational inference technique that relaxes the rigid Gaussian prior assumption typically placed on VAEs encoder networks (Kingma & Welling, 2014) to capture faithfully such summaries. In step (ii.), we develop an approach around the manifold invariance concept of (Roussel, 2019) to perturb the elements of the learned manifold while ensuring the perturbed elements remain within the manifold. Finally, in step (iii.), we propose a learning algorithm whereby we leverage the rich semantics of the inputs and the perturbations as a source of knowledge upon which we impose adversarial constraints to produce adversarial examples. Unlike (Song et al., 2018; Carlini & Wagner, 2016; Zhao et al., 2018b; Goodfellow et al., 2014b ) that resort to a costly search of adversarial examples, our algorithm is efficient and end-to-end.

The main contributions of our work are thus: (i.) a variational inference method for manifold learning in the presence of continuous latent variables with minimal assumptions about their distribution, (ii.) an intuitive perturbation strategy that encourages perturbed elements of a manifold to remain within the manifold, (iii.) an end-to-end and computationally efficient algorithm that combines (i.) and (ii.) to generate adversarial examples in a black-box setting, and (iv.) illustration on toy data, images and text, as well as empirical validation against strong certified and non-certified adversarial defenses.

2. PRELIMINARIES & ARCHITECTURE:

Notations. Let x be a sample from the input space X , with label y from a set of possible labels Y, and D = {x n } N n=1 a set of N such samples x. Also, let d be a distance measure on X capturing closeness in input space, or on Z, the embedding space of X , capturing semantics similarity.

Adversarial Examples. Given a classifier g, and its loss function , an adversarial example of x is produced by maximizing the objective below over an -radius ball around x (Athalye et al., 2017) .

x = arg max x ∈X (g(x ), y) such that x ∈ B(x; )

Above, the search region for adversarial examples is confined to a uniformly-bounded ball B(x; ). In reality, however, the shape imposed on B is quite restrictive as the optimal search region may have a different topology. It is also common practice to produce adversarial examples in the input space X -via an exhaustive and costly search procedure (Shaham et al., 2018; Song et al., 2018; Zhao et al., 2018b; Athalye et al., 2017; Carlini & Wagner, 2016; Goodfellow et al., 2014b) . Unlike these approaches, however, we wish to operate in Z, the lower dimensional embedding space of X , with minimal computational overhead. Our primary intuition is that Z captures well the semantics of D. Attack Model. Given a sample x ∈ D and its class y ∈ Y, we want to construct an adversarial example x that shares the same semantics as x. We assume the semantics of x (resp. x ) is modeled by a learned latent variable model p(z) (resp. p (z )), where z, z ∈ Z. In this setting, observing x (resp. x ) is conditioned on the observation model p(x|z) (resp. p(x |z )), that is: x ∼ p(x|z) and x ∼ p(x |z ), with z ∼ p(z) and z ∼ p (z ). We learn this model in a way that d(x, x ) is small and g(x) = y ∧ g(x ) = y while ensuring also that d(z, z ) is small. Intuitively, we get the latent z ∼ p(z) which encodes the semantics of x. Then, we perturb z in a way that its perturbed version z ∼ p (z ) lies in the manifold that supports p(z). We define a manifold as a set of points in Z where every point is locally Euclidean (Roussel, 2019) . We devise our perturbation procedure by generalizing the manifold invariance concept of (Roussel, 2019) to Z. For that, we consider two embedding maps h : X → Z and h : X → Z, parameterized by θ and θ , as surrogates for p(z) and p(z ). We assume θ and θ follow the implicit distributions p(θ) and p(θ ). In the following, we consider M such embedding maps h and h .1 If we let z = h(x; θ) and z = h (x; θ ), we ensure that z is in the manifold that supports p(z) by constraining d(z, z ) to be small. Then, given a map dec φ : Z → X , we craft x = dec φ (z ) in a way that d(x, x ) is small and g(x) = y ∧ g(x ) = y. Then, we say that x is adversarial to x and preserves its semantics.

θ m ∼ f η (ξ m ) z m ∼ p(z|x; θ m ) Decoder p φ z x x ∈ D θ m ∼ f η (ξ m ) z m ∼ p z |x; θ m E E z 1 , ..., z M

Model Architecture. To implement our attack model, we propose as a framework the architecture illustrated in Figure 1 . Our framework is essentially a variational auto-encoder with two encoders E and E' that learn the geometric summaries of D via statistical inference. We present two inference mechanisms -implicit manifold learning via Stein variational gradient descent (Liu & Wang, 2016) and Gram-Schmidt basis sign method (Dukes, 2014) -to draw instances of model parameters from the implicit distributions p(θ) and p(θ ) that we parameterize E and E' with. Both encoders optimize the uncertainty inherent to embedding D in Z while guaranteeing easy sampling via Bayesian ensembling. Finally, the decoder p φ acts as a generative model for constructing adversarial examples.

Threat Model. We consider in this paper a black-box scenario where we, as an attacker, have only access to the predictions of a classifier g. As the attacker, we want to construct adversarial examples not knowing the intricacies of g such as its loss function, nor having access to its gradient. We focus on this scenario because it is challenging and more plausible in real-life than the white-box case. This threat model serves to evaluate both certified defenses and non-certified ones under our attack model

3. IMPLICIT MANIFOLD LEARNING:

Manifold learning is based on the assumption that high dimensional data lies on or near lower dimensional manifolds in a data embedding space. In the variational auto-encoder (VAE) (Kingma & Welling, 2014) setting, the datapoints x n ∈ D are modeled via a decoder x n ∼ p(x n |z n ; φ). To learn the parameters φ, one typically maximizes a variational approximation to the empirical expected log-likelihood 1/N N n=1 log p(x n ; φ), called evidence lower bound (ELBO), defined as:

L e (φ, ψ; x) = E z|x;ψ log p(x|z; φ)p(z) q(z|x; ψ) = -KL(q(z|x; ψ) p(z|x; φ)) + log p(x; φ). (1)

The expectation E z|x;ψ can be re-expressed as a sum of a reconstruction loss, or expected negative log-likelihood of x, and a KL(q(z|x; ψ) p(z)) term. The KL term acts as a regularizer and forces the encoder q(z|x; ψ) to follow a distribution similar to p(z). In VAEs, p(z) is defined as a spherical Gaussian. That is, VAEs learn an encoding function that maps the data manifold to an isotropic Gaussian. However, Jimenez Rezende & Mohamed (2015) have shown that the Gaussian form imposed on p(z) leads to uninformative latent codes; hence to poorly learning the semantics of D (Zhao et al., 2017) . To sidestep this issue, we minimize the divergence term KL(q(z|x; ψ) p(z|x; φ)) using Stein Variational Gradient Descent (Liu & Wang, 2016) instead of explicitly optimizing the ELBO.

Stein Variational Gradient Descent (SVGD) is a nonparametric variational inference method that combines the advantages of MCMC sampling and variational inference. Unlike ELBO (Kingma & Welling, 2014), SVGD does not confine a target distribution p(z) it approximates to simple or tractable parametric distributions. It remains yet an efficient algorithm. To approximate p(z), SVGD maintains M particles z = {z m } M m=1 , initially sampled from a simple distribution, it iteratively transports via functional gradient descent. At iteration t, each particle z t ∈ z t is updated as follows:

z t+1 ← z t + α t τ (z t ) where τ (zt) = 1 M M m=1 k(z m t , zt)∇ z m t log p(z m t ) + ∇ z m t k(z m t , zt) ,

where α t is a step-size and k(., .) is a positive-definite kernel. In the equation above, each particle determines its update direction by consulting with other particles and asking their gradients. The importance of the latter particles is weighted according to the distance measure k(., .). Closer particles are given higher consideration than those lying further away. The term ∇ z m k(z m , z) is a regularizer that acts as a repulsive force between the particles to prevent them from collapsing into one particle. Upon convergence, the particles z m will be unbiased samples of the true implicit distribution p(z).

Manifold Learning via SVGD. To characterize the manifold of D, which we denote M, we learn the encoding function q(.; ψ). Similar to Pu et al. (2017) , we optimize the divergence KL(q(z|x; ψ) p(z|x; φ)) using SVGD. As an MCMC method, SVGD, however, induces inherent uncertainty we ought to capture in order to learn M efficiently. To potentially capture such uncertainty, Pu et al. (2017) use dropout. However, according to Hron et al. (2017) , dropout is not principled. Bayesian methods, on the contrary, provide a principled way to model uncertainty through the posterior distribution over model parameters. Kim et al. (2018) have shown that SVGD can be cast as a Bayesian approach for parameter estimation and uncertainty quantification. Since SVGD always maintains M particles, we introduce thus M instances of model parameters Θ = {θ m } M m=1 , where every θ m ∈ Θ is a particle that defines the weights and biases of a Bayesian neural network.

For large M , however, maintaining Θ can be computationally prohibitive because of the memory footprint. Furthermore, the need to generate the particles during inference for each test case is undesirable. To sidestep these issues, we maintain only one (recognition) network f η that takes as input ξ m ∼ N (0, I) and outputs a particle θ m . The recognition network f η learns the trajectories of the particles as they get updated via SVGD. f η serves as a proxy to SVGD sampling strategy, and is refined through a small number of gradient steps to get good generalization.

η t+1 ← arg min η M m=1 f (ξ m ; η t ) θ t m -θ t+1 m 2 with θ t+1 m ← θ t m + α t τ (θ t m ), where τ (θ t ) = 1 M M j=1 k(θ t j , θ t )∇ θ j t log p(θ t j ) + ∇ θ j t k(θ t j , θ t ) .

(2)

We use the notation SVGD τ (Θ) to denote an SVGD update of Θ using the operator τ (.). As the particles θ are Bayesian, upon observing D, we update the prior p(θ t j ) to obtain the posterior p(θ t j |D) ∝ p(D|θ t j )p(θ t j ) which captures the uncertainty. We refer the reader to Appendix A for a formulation of p(θ t j |D) and p(D|θ t j ). The data likelihood p(D|θ t j ) is evaluated over all pairs (x, z) where x ∈ D and z is a dependent variable. However, z is not given. Thus, we introduce the inversion process described in Figure 2 to generate such z using Algorithm 1. For any input x ∈ D, we sample its latent code z from p(z|x; D), which we approximate by Monte Carlo over Θ; that is:

p(z|x; D) = p(z|x; θ)p(θ|D)dz ≈ 1 M M m=1

p(z|x; θm) where θm ∼ p(θ|D).

(3)

4. PERTURBATION INVARIANCE:

Here, we focus on perturbing the elements of M. We want the perturbed elements to reside in M and exhibit the semantics of D that M captures. Formally, we seek a linear mapping h :

M → M such that for any point z ∈ M, a neighborhood U of z is invariant under h ; that is: z ∈ U ⇒ h (z ) ∈ U.

In this case, we say that M is preserved under h . Trivial examples of such mappings are linear combinations of the basis vectors of subspaces S of M called linear spans of S.

Rather than finding a linear span h directly, we introduce a new set of instances of model parameters Θ = {θ m } M m=1 . Each θ m denotes the weights and biases of a Bayesian neural network. Then, for any input x ∈ D and its latent code z ∼ p(z|x; D), a point in M, we set h (z) = z where z ∼ p(z |x; D). We approximate p(z |x; D) by Monte Carlo using Θ , as in Equation 3. We leverage the local smoothness of M to learn each θ m in a way to encourage z to reside in M in a close neighborhood of z using a technique called Gram-Schmidt Basis Sign Method.

Gram-Schmidt Basis Sign Method (GBSM). Let X be a batch of samples of D, Z m a set of latent codes z m ∼ p(z|x; θ m ) where x ∈ X, and θ m ∈ Θ. For any m ∈ {1.., M }, we learn θ m to generate perturbed versions of z m ∈ Z m along the directions of an orthonormal basis U m . As M is locally Euclidean, we compute the dimensions of the subspace Z m by applying Gram-Schmidt (Dukes, 2014) to orthogonalize the span of representative local points. We formalize GBSM as follows:

arg min δm, θ m (δ m , θ m ) := zm z m -z m + δ m sign(u im ) 2 where z m ∼ p(z |x i ; θ m ).

The intuition behind GBSM is to utilize the fact that topological spaces are closed under their basis vectors to render M invariant to the perturbations δ m . To elaborate more on GBSM, we first sample a model instance θ m . Then, we generate z m ∼ p(z |x; θ m ) for all x ∈ X. We orthogonalize Z m and find the perturbations δ m that minimizes along the directions of the basis vectors u im ∈ U m . We want the perturbations δ m to be small. With δ m fixed, we update θ m by minimizing again. We use the notation GBSM(Θ , ∆) where ∆ = {δ m } M m=1 to denote one update of Θ via GBSM. Manifold Alignment. Although GBSM confers us latent noise imperceptibility and sampling speed, Θ may deviate from Θ; in which case the manifolds they learn will mis-align. To mitigate this issue, we regularize each θ m ∈ Θ after every GBSM update. In essence, we apply one SVGD update on Θ to ensure that Θ follows the transform maps constructed by the particles Θ (Han & Liu, 2017) .

EQUATION

We use the notation SVGD π (Θ ) to refer to the gradient update rule in Equation 4. In this rule, the model instances Θ determine their own update direction by consulting only the particles Θ instead of consulting each other. Maintaining Θ = {θ m } M m=1 for large M is, however, computationally prohibitive. Thus, as in Section 3, we keep only one (recognition) network f η that takes as input ξ m ∼ N (0, I) and outputs θ m ∼ f (ξ m ; η ). Here too we refine η through a small number of gradient steps to learn the trajectories that Θ follows as it gets updated via GBSM and SVGD π .

EQUATION

5. GENERATING ADVERSARIAL EXAMPLES:

In this paper, a black-box scenario is considered. In this scenario, we have only access to the predictions of the classifier g. We produce adversarial examples by optimizing the loss below. The first term is the reconstruction loss inherent to VAEs. This loss accounts here for the dissimilarity between any input x ∈ D and its adversarial counterpart x , and is constrained to be smaller than attack so that x resides within an attack -radius ball of x. Unless otherwise specified, we shall use norm L 2 as reconstruction loss. The second term is an auxiliary log-likelihood loss (for g) of a target class y ∈ Y \ {y} where y is the class of x. This loss defines the cost incurred for failing to fool g.

EQUATION

In Algorithm 2, we show how we unify our manifold learning and perturbation strategy into one learning procedure to generate adversarial examples without resorting to an exhaustive search.

6. RELATED WORK:

Manifold Learning. VAEs are generally used to learn manifolds (Yu et al., 2018; Falorsi et al., 2018; Higgins et al., 2016) by maximizing the ELBO of the data log-likelihood (Alemi et al., 2017; Chen et al., 2017) . Optimizing the ELBO entails reparameterizing the encoder to a Gaussian distribution (Kingma & Welling, 2014) . This reparameterization is, however, restrictive (Jimenez Rezende & Mohamed, 2015) as it may lead to learning poorly the manifold of the data (Zhao et al., 2017) . To alleviate this issue, we use SVGD, similar to Pu et al. (2017) . While our approach and that of Pu et al. (2017) may look similar, ours is more principled. As discussed in (Hron et al., 2017) , dropout which Pu et al. (2017) use is not Bayesian. Since our model instances are Bayesian, we are better equipped to capture the uncertainty. Capturing the uncertainty requires, however, evaluating the data likelihood. As we are operating in latent space, this raises the interesting challenge of assigning target-dependent variables to the inputs. We overcome this challenge using our inversion process.

Adversarial Examples. Studies in adversarial deep learning (Athalye et al., 2018; Kurakin et al., 2016; Goodfellow et al., 2014b; Athalye et al., 2017) can be categorized into two groups. The first group (Carlini & Wagner, 2016; Athalye et al., 2017; Moosavi-Dezfooli et al., 2016) proposes to Algorithm 2 Generating Adversarial Examples. Lines 2 and 4 compute distances between sets keeping a one-to-one mapping between them. x is adversarial to x when L x ≤ attack and y = y .

1: function IN N E RTR A I N I N G(Θ, Θ , η, η , ∆, x) local gradient updates of f η , f η , ∆ Require: Learning rates β, β 2: η ← η -β∇ η Θ-SVGD τ (Θ) 2

apply inversion on x and update η 3:

∆, Θ ← GBSM(Θ , ∆) update ∆ and Θ using GBSM η, η , ∆ ←InnerTraining(Θ, Θ , η, η , ∆, x) 13: L x := x -x 2 ; L x := x -x 2 reconstruction losses on x and x 14:

L x := L x , if L x > attack L x + min y ∈Y 1 y=y • log (1 -P (y |x )) , otherwise 15: η ← η -α∇ η L x; η ← η -α ∇ η L x SGD update using Adam optimizer 16: φ ← φ -∇ φ (L x + L x )

SGD update using Adam optimizer generate adversarial examples directly in the input space of the original data by distorting, occluding or changing illumination in images to cause changes in classification. The second group (Song et al., 2018; Zhao et al., 2018b) , where our work belongs, uses generative models to search for adversarial examples in the dense and continuous representations of the data rather than in its input space.

Adversarial Images. Song et al. (2018) propose to construct unrestricted adversarial examples in the image domain by training a conditional GAN that constrains the search region for a latent code z in the neighborhood of a target z. Zhao et al. (2018b) use also a GAN to map input images to a latent space where they conduct their search for adversarial examples. These studies are the closest to ours. Unlike in (Song et al., 2018) and (Zhao et al., 2018b) , however, our adversarial perturbations are learned, and we do not constrain the search for adversarial examples to uniformly-bounded regions.

In stark contrast to Song et al. (2018) and Zhao et al. (2018b) approaches also, where the search for adversarial examples is exhaustive and decoupled from the training of the GANs, our approach is efficient and end-to-end. Lastly, by capturing the uncertainty induced by embedding the data, we characterize the semantics of the data better, allowing us thus to generate sound adversarial examples.

Adversarial Text. Previous studies on adversarial text generation (Zhao et al., 2018a; Jia & Liang, 2017; Alvarez-Melis & Jaakkola, 2017; Li et al., 2016) perform word erasures and replacements directly in the input space using domain-specific rules or heuristics, or they require manual curation. Similar to us, Zhao et al. (2018b) propose to search for textual adversarial examples in the latent representation of the data. However, in addition to the differences aforementioned for images, the search for adversarial examples is handled more gracefully in our case thanks to an efficient gradient-based optimization method in lieu of a computationally expensive search in the latent space.

7. EXPERIMENTS & RESULTS:

Before, we presented an attack model whereby we align the semantics of the inputs with their adversarial counterparts. As a reminder, our attack model is black-box, restricted and non-targeted.

Our adversarial examples reside within an attack -radius ball of the inputs as our reconstruction loss, which measures the amount of changes in the inputs, is bounded by attack (see Equation 6).

We validate the adversarial examples we produce based on three evaluation criteria: (i.) manifold preservation, (ii.) adversarial strength, and (iii.) soundness via manual evaluation. We provide in Appendix A examples of the adversarial images and sentences that we construct.

7.1. MANIFOLD PRESERVATION:

We experiment with a 3D non-linear Swiss Roll dataset which comprises 1600 datapoints grouped in 4 classes. We show in Figure 3 , on the left, the 2D plots of the manifold we learn. In the middle, we plot the manifold and its elements that we perturbed and whose reconstructions are adversarial. On the right, we show the manifold overlaid with the latent codes of the adversarial examples produced by PGD (Goodfellow et al., 2014b) with attack ≤ 0.3. Observe in Figure 3 , in the middle, how the latent codes of our adversarial examples espouse the Swiss Roll manifold, unlike the plot on the right. 2018) -whose attack model resembles ours2 -, for MNIST, we also target the certified defenses (Raghunathan et al., 2018; Kolter & Wong, 2017) with attack = 0.1 using norm L ∞ as reconstruction loss. For all the datasets, the accuracies of the models we target are higher than 96.3%. Next, we present our attack success rates and give examples of our adversarial images in Figure 4 .

Attack Success Rate (ASR) is the percentage of examples misclassified by the adversarially trained Resnet models. For attack = 0.3, the publicly known ASR of PGD attacks on MNIST is 88.79%. However, our ASR for MNIST is 97.2%, higher than PGD. Also, with attack ≈ 1.2, using norm L ∞ as reconstruction loss in Equation 6, we achieve an ASR of 97.6% against (Kolter & Wong, 2017) where PGD achieves 91.6%. Finally, we achieve an ASR of 87.6% for SVHN, and 84.4% for CelebA.

7.2.1. ADVERSARIAL TEXT:

Datasets. For text, we consider the SNLI (Bowman et al., 2015) dataset. SNLI consists of sentence pairs where each pair contains a premise (P) and a hypothesis (H), and a label indicating the relationship (entailment, neutral, contradiction) between the premise and hypothesis. For instance, the following pair is assigned the label entailment to indicate that the premise entails the hypothesis.

Premise: A soccer game with multiple males playing. Hypothesis: Some men are playing a sport. Setup. We perturb the hypotheses while keeping the premises unchanged. Similar to Zhao et al. (2018b) , we generate adversarial text at word level using a vocabulary of 11,000 words. We also use ARAE (Zhao et al., 2018a) for word embedding, and a CNN for sentence embedding. To generate perturbed hypotheses, we experiment with three decoders: (i.) p φ is a transpose CNN, (ii.) p φ is a language model, and (iii.) we use the decoder of a pre-trained ARAE (Zhao et al., 2018a) model. We detail their configurations in Appendix B.

The transpose CNN generates more meaningful hypotheses than the language model and the pretrained ARAE model although we notice sometimes changes in the meaning of the original hypotheses. We discuss these limitations more in details in Appendix A. Henceforward, we use the transpose CNN to generate perturbed hypotheses.

See Table 1 for examples of generated hypotheses.

Attack Success Rate (ASR). We attack an SNLI classifier that has a test accuracy of 89.42%. Given a pair (P, H) with label l, its perturbed version (P, H') is adversarial if the classifier assigns the label l to (P, H), (P, H') is manually found to retain the label of (P, H), and such label differs from the one the classifier assigns to (P, H'). To compute the ASR, we run a pilot study which we detail next. We report in Table 2 the results for MNIST and, in Table 4 , the results for CelebA and SVHN.

We hand the same questionnaire to the subjects with 50 MNIST images, their clean reconstructions, and the adversarial examples we craft with our method. We also handed the adversarial examples generated using Song et al. (2018) , Zhao et al. (2018b) and PGD methods. We ask the subjects to assess the soundness of the adversarial examples based on the semantic features (e.g., shape, distortion, contours, class) of the real MNIST images. We report the evaluation results in Table 3 . Pilot Study II -SNLI. Using the transpose CNN as decoder p φ , we generate adversarial hypotheses for the SNLI sentence pairs with the premises kept unchanged. Then, we select manually 20 pairs of clean sentences (premise, hypothesis), and adversarial hypotheses. We also pick 20 pairs of sentences and adversarial hypotheses generated this time using Zhao et al. (2018b) 's method against their treeLSTM classifier. We choose this classifier as its accuracy (89.04%) is close to ours (89.42%).

We carry out a pilot study where we ask two yes-or-no questions: (Q1) are the adversarial samples semantically sound? and (Q2) are they similar to the true inputs? We report the results in Table 4 . Takeaways. As reflected in the pilot study and the attack success rates, we achieve good results in the image and text classification tasks. In the image classification task, we achieve better results than PGD and Song et al. (2018) both against the certified and non-certified defenses. The other takeaway is: although the targeted certified defenses are resilient to adversarial examples crafted in the input space, we can achieve manual success rates higher than the certified rates when the examples are constructed in the latent space, and the search region is unrestricted. In text classification, we achieve better results than Zhao et al. (2018b) when using their treeLSTM classifier as target model.

8. CONCLUSION:

Many approaches in adversarial attacks fail to enforce the semantic relatedness that ought to exist between original inputs and their adversarial counterparts. Motivated by this fact, we developed a method tailored to ensuring that the original inputs and their adversarial examples exhibit similar semantics by conducting the search for adversarial examples in the manifold of the inputs. Our success rates against certified and non-certified defenses known to be resilient to traditional adversarial attacks illustrate the effectiveness of our method in generating sound and strong adversarial examples.

Although in the text classification task we achieved good results and generated informative adversarial sentences, as the transpose CNN gets more accurate -recall that it is partly trained to minimize a reconstruction error -, generating adversarial sentences that are different from the input sentences and yet preserve their semantic meaning becomes more challenging. In the future, we intend to build upon the recent advances in text understanding to improve our text generation process.

Measuring Perceptual Quality is desirable when the method relied upon to generate adversarial examples uses GANs or VAEs; both known to produce often samples of limited quality. As Song et al. (2018) perform unrestricted targeted attacks -their adversarial examples might totally differ from the true inputs -and Zhao et al. (2018b) do not target certified defenses, a fair side-by-side comparison of our results and theirs using metrics like mutual information or frechet inception distance, seems unachievable. Thus, to measure the quality of our adversarial examples and compare our results with (Song et al., 2018) and (Zhao et al., 2018b) , we carried out the pilot study.

ADVERSARIAL IMAGES: CELEBA 

:

The reason why we consider M instances of h and h will become apparent in Section 3.

Note that our results are, however, not directly comparable with(Song et al., 2018) as their reported success rates are for unrestricted adversarial examples, unlike ours, manually computed from Amazon MTurkers votes.

Figures and Tables:

Figure 1: Architecture. The set of model parameters Θ = {θm} M m=1 and Θ = {θ m } M m=1 are sampled from the recognition networks fη and f η . Given an input x ∈ D, we use E to sample the latent codes z1, ..., zM via Θ. These codes are passed to E to learn their perturbed versions z 1 , ..., z M using Θ . The output x ∼ p φ (x |z ) is generated via posterior sampling of a z (in red).

Figure2: Inversion. Process for computing the likelihood p(D|θ). As the decoder p φ gets accurate, the error x -x 2 becomes small (see Algorithm 2), and we get closer to sampling the optimal z.

← ηβ ∇ η Θ -SVGD π (Θ ) 2 align Θ with Θ and update η 5: return η, η , ∆ Require: Training samples (x, y) ∈ D × Y Require: Number of model instances M Require: Number of inner updates T Require: Initialize weights η, η , φ recognition nets f η , f η , decoder p φ Require: Initialize perturbations ∆ := {δ m } M m=1 latent (adversarial) perturbations Require: Learning rates , α, α , and noise margin attack 6: Sample ξ 1 , ..., ξ M from N (0, I) inputs to recognition nets f η , f η 7: for t = 1 to T do 8: Sample Θ = {θ m } M m=1 where θ m ∼ f η (ξ m ) 9: Sample Θ = {θ m } M m=1 where θ m ∼ f η (ξ m ) 10: Use Θ and Θ in Equation 3 to sample z and z 11: Sample x ∼ p(x|z, φ) and x ∼ p(x |z , φ) clean and perturbed reconstructions 12:

Figure 3: Invariance. Swiss Roll manifold learned with our encoder E (left), and after perturbing its elements with our encoder E (middle) vs. that of PGD adversarial examples (right) learned using E.

Figure 4: Inputs (left) -Adversarial examples (right, inside red boxes). MNIST: (a)-(b), CelebA: (c)-(d), SVHN: (e)-(f). See Appendix A for more samples with higher resolution.

MANUAL EVALUATIONTo validate our adversarial examples and assess their soundness vs.Song et al. (2018),Zhao et al.  (2018b)  and PGD(Madry et al., 2017) adversarial examples, we carry out a pilot study whereby we ask three yes-or-no questions: (Q1) are the adversarial examples semantically sound?, (Q2) are the true inputs similar perceptually or in meaning to their adversarial counterparts? and (Q3) are there any interpretable visual cues in the adversarial images that support their misclassification?Pilot Study I. For MNIST, we pick 50 images (5 for each digit), generate their clean reconstructions, and their adversarial examples against a 40-step PGD ResNet with attack ≤ 0.3. We target also the certified defenses ofRaghunathan et al. (2018) andKolter & Wong (2017) with attack = 0.1. For SVHN, we carry out a similar pilot study and attack a 40-step PGD ResNet. For CelebA, we pick 50 images (25 for each gender), generate adversarial examples against a 40-step PGD ResNet. For all three datasets, we hand the images and the questionnaire to 10 human subjects for manual evaluation.

Test samples and their perturbed versions. See more examples in Appendix A. True Input 1 P: A group of people are gathered together. H: There is a group here. Label: Entailment Adversary 1 H': There is a group there. Label: Contradiction True Input 2 P: A female lacrosse player jumps up. H: A football player sleeps. Label: Contradiction Adversary 2 H': A football player sits. Label: Neutral True Input 3 P: A man stands in a curvy corridor.

Table content: <table/>

Pilot Study I. Note that against the certified defenses ofRaghunathan et al. (2018) and Kolter & Wong (2017),Song et al. (2018) achieved (manual) success rates of 86.6% and 88.6%.

Table content: <table><tr><td/><td/><td>MNIST</td><td/></tr><tr><td>QUESTIONNAIRE</td><td colspan="3">40-STEP PGD RAGHUNATHAN ET AL. (2018) KOLTER &amp; WONG (2017)</td></tr><tr><td>QUESTION Q1: YES</td><td>100 %</td><td>100 %</td><td>100 %</td></tr><tr><td>QUESTION Q2: YES</td><td>100 %</td><td>100 %</td><td>100 %</td></tr><tr><td>QUESTION Q3: NO</td><td>100 %</td><td>100 %</td><td>100 %</td></tr></table>

Pilot Study I. The adversarial images are generated against the adversarially trained Resnets.

Table content: <table><tr><td>QUESTIONNAIRE</td><td>OUR METHOD</td><td>SONG ET AL. (2018)</td><td>ZHAO ET AL. (2018B)</td><td>PGD</td></tr><tr><td>QUESTION Q1: YES</td><td>100 %</td><td>85.9 %</td><td>97.8 %</td><td>76.7 %</td></tr><tr><td>QUESTION Q2: YES</td><td>100 %</td><td>79.3 %</td><td>89.7 %</td><td>66.8 %</td></tr><tr><td>QUESTION Q3: NO</td><td>100 %</td><td>71.8 %</td><td>94.6 %</td><td>42.7 %</td></tr></table>

Pilot Studies. † Some adversarial images and original ones were found blurry to evaluate.

Table content: <table><tr><td/><td>PILOT I</td><td/><td colspan="2">PILOT II -SNLI</td></tr><tr><td>QUESTIONNAIRE</td><td>CELEBA</td><td>SVHN</td><td>OUR METHOD</td><td>ZHAO ET AL. (2018B)</td></tr><tr><td>QUESTION Q1: YES</td><td>100 %</td><td>95  † %</td><td>83.7 %</td><td>79.6%</td></tr><tr><td>QUESTION Q2: YES</td><td>100 %</td><td>97 %</td><td>60.4 %</td><td>56.3%</td></tr><tr><td>QUESTION Q3: NO</td><td>100 %</td><td>100 %</td><td>N/A</td><td>N/A</td></tr></table>

CelebA samples, their clean reconstructions, and adversarial examples (in red boxes).

Table content: <table><tr><td>INPUTS</td></tr><tr><td>CLEAN</td></tr><tr><td>RECON-</td></tr><tr><td>STRUCTIONS</td></tr></table>

Peer review (iclr2020):
Major caveat: I have published in the area of adversarial attacks on NLP models, but the specifics of the methods presented in this paper are quite outside of my expertise, and I do not have time to become familiar with them for this review. I hope there are other reviewers that are more qualified than I am to check the specifics of the methods. This paper presents a new technique for generating adversarial examples, by first learning the data manifold in an embedding space, then finding an adversarial example that lies on the manifold. I like this idea, it intuitively seems like a promising method for obtaining semantically meaningful adversarial examples. As I said above, I do not feel qualified to review whether the method should _theoretically_ accomplish its goals, so my judgment of this paper is on the intuition behind the idea (which I like), and the results that I can see (which are less promising). In order to have a *semantics preserving* attack, the method needs to (1) remain on the data manifold, and (2) not change the label a human would give to the input. For (1), this appears to have been accomplished on most datasets, though it seems pretty hard to argue that the artifacts seen in the MNIST examples shown are on the data manifold - there are no such artifacts in any of the inputs, or in the clean reconstruction. How do the authors claim that this actually did a reasonable job of staying in the data manifold? For (2), most of the images do indeed look like they should retain their human labels, which is good (but also not hard for adversarial images). Almost all of the textual examples, however, have correct predictions from the model after the adversarial change to the input. You can*t really argue that these are *semantics preserving*, or even *successful attacks*, as they change the expected input label. This is why semantics-preserving attacks are so hard in NLP, and I don*t think that this method has accomplished its goal here at all, at least for text. The authors should consult with experts in NLP before making claims about successfully constructing semantics preserving attacks on NLP models. I*m pretty on the fence about this paper, as I like the intuition, and the method appears to work reasonably well for vision. It does not work as claimed for text, however, and that should be fixed before this paper is published (either with softened claims or with better results). Hopefully people from other perspectives can pipe in and give a more clear picture on this paper. EDIT: See discussion below for my justification for reducing my score from a 3 to a 1. EDIT 11/14: The authors* revisions have satisfied my concerns about how the NLP attacks are described. I*m a little bit nervous about how the examples were changed - it seems that nothing changed about the method itself, so the authors probably cherry-picked better examples - but that*s not sufficiently worrying to me to justify rejection. The pilot study is also quite weak, as the number of inputs that were evaluated was only 20, and the questions presented don*t appear to ask about changes in the label. I don*t know how you could get 100% on that given the examples that I saw in the previous version of the paper. This is all to say that I don*t think the NLP attacks are actively problematic anymore, as they were previously, now they are just weak. The main contribution here is the technical contribution, anyway, so weaker results on one of the datasets tested is not a deal-breaker to me. Assuming the technical contributions pass muster (which, as I said, I don*t really feel qualified to judge), I*m satisfied with this paper as it is now.