TOWARDS FEATURE SPACE ADVERSARIAL ATTACK
Authors: Qiuling Xu
Guanhong Tao
Siyuan Cheng
Lin Tan
Xiangyu Zhang
Venue:  (2020)

Abstract: We propose a new type of adversarial attack to Deep Neural Networks (DNNs) for image classification. Different from most existing attacks that directly perturb input pixels. Our attack focuses on perturbing abstract features, more specifically, features that denote styles, including interpretable styles such as vivid colors and sharp outlines, and uninterpretable ones. It induces model misclassfication by injecting style changes insensitive for humans, through an optimization procedure. We show that state-of-the-art pixel space adversarial attack detection and defense techniques are ineffective in guarding against feature space attacks. 

Under review as a conference paper at ICLR 2020
TOWARDS FEATURE SPACE ADVERSARIAL ATTACK
Anonymous authors
Paper under double-bling review
ABSTRACT
We propose new type of adversarial attack to Deep Neural Networks (DNNs) for image classification. Different from most existing attacks that directly perturb input pixels. Our attack focuses on perturbing abstract features, more specifically, features that denote styles, including interpretable styles such as vivid colors and sharp outlines, and uninterpretable ones. It induces model misclassfication by injecting style changes insensitive for humans, through an optimization procedure. We show that state-of-the-art pixel space adversarial attack detection and defense techniques are ineffective in guarding against feature space attacks.
INTRODUCTION
Adversarial attacks prominent threat the broad application of Deep Neural Networks (DNNs). In the context of classification applications, given a pre-trained model M and a benign input æ of some output label y, adversarial attack perturbs æ such that M misclassifies the perturbed x. The perturbed input is called adversarial example. Such perturbations are usually bounded by some distance norm such that they are not perceptible by humans. Ever since it was proposed in (Szegedy et al. 2014). there has been large body of research that develops various methods to construct adversarial examples, e.g., (Carlini & Wagner |017]| Madry et al., 2018), with different modalities such as images (Carlini & Wagner, (017), audio (Qin et al. 2019)), text (Ebrahimi al. 2018), and video (Li et al. 2019), detect adversarial examples (Tao et al. 2018 Ma et al. 2019). and use adversarial examples to harden the models (Madry et al. 2018 Zhang et al. 2019)..
However, most existing attacks are in the pixel space, that is, the perturbations occur directly in the pixel space, and the pixel level differences between the adversarial example and the original input are bounded. In this paper, we illustrate that adversarial attack can be conducted in the feature space. The underlying assumption (in the context of image classification) is that during training, DNN may extract a large number of abstract features. While many of them denote critical characteristics of the object, some of them are secondary, for example, the different styles of an image (c.g., vivid colors versus pale colors, sharp outlines versus blur outlines). These secondary features may play an improperly important role in model prediction. As a result, feature space attack can inject such secondary features, which are not simple pixel perturbation, but rather functions over the given benign input, to induce model misclassification. Since humans are not sensitive these features, the resulted adversarial examples look very natural from humans' perspective. As many of these features are pervasive, the resulted pixel space perturbation may be relatively much more substantial than existing pixel space attacks. As such, pixel space detection and hardening techniques are ineffective for feature space attacks (see K4). Figure shows a number of adversarial examples generated by technique, their comparison with the original examples, and the pixel space distances. Observe that while the distances are much larger compared to those in pixel space attacks, the adversarial examples are completely natural, or even indistinguishable from the original inputs in humans' eyes. The contrast of the benign-adversan co-locate with the primary content features, denoting imperceptible tweaking of these features.
pairs illustrates that the malicious perturbations largely
Under the hood, we consider that the activations of an inner layer represent a set of abstract fea- tures, including those primary and secondary. To avoid generating adversarial examples that are unnatural, we refrain from tampering with the primary features (or content features) and focus on perturbing the secondary style features. Inspired by the recent advance in style transfer (Huang & Belongie, (017), the mean and variance of activations are considered the style. As such, we fo- cus on perturbing the means and variances while preserving the shape of the activation values (i.e., the up-and-downs of these values and the relative scale of such up-and-downs). We use gradient driven optimization to search for the style perturbations that can induce misclassification. Since our threat model is the same existing pixel space attacks, that is, the attack is launched by providing the adversarial example to the model. An important step is to translate the activations with style changes back to naturally looking pixel space example. We address the problem by considering

---

Under review as a conference paper at ICLR 2020
(a) Spaniel
(b) Espresso (c) Balloon
(d) Llama
(e) Printer
(l) Lizard
(g) Guitar
(h) Race
==2-/21/255
192/255
149/255
183/255
252/255
225/255
216/255
248/255
l =25.92
24.47
20.75
28.55
21.80
40.88
25.60
29.67
Figure I: Examples by feature space attack. The top row presents original images. The middle row
denotes adversarial samples. The third row shows the pixel-wise difference (x3) between original
images and adversarial samples. The fo and Eg norms shown the bottom.
the differences of any pair of training inputs of the same class the possible style differences, and pre- pre-training a decoder that can automatically impose styles in the pixel space based on the style fea- ture perturbation happening in inner layer. We propose two concrete feature space attacks, one to enhance styles and the other to impose styles constituted from a set of pre-defined style prototypes.
We evaluate attacks 3 datasets and 7 models. We show that feature space attacks ef- fectively generate adversarial samples that evade 5 state-of-the-ee detection/defense approaches. Particularly, our proposed attack can reduce the detection rate of state-of-the-ar approach (Roth) et al, 2019) to 0.04% on the CIFAR-10 dataset, and the prediction accuracy of a model hardened by of adversarial training technique (Xie al., 2019) to 1.25% on ImageNet. Moreover, we observe that despite the large distance introduced in the pixel space, the distances in feature space are similar or even smaller than those in ε-norm based attacks. The generated adversarial examples have only natural, and in many cases, human imperceptible style differences compared with the original inputs, demonstrating the practicality of the attacks.
This work only demonstrates the feasibility of feature space attacks. The features we are attacking are relatively simple. In the future, we expect more research on complicated feature space attacks.
BACKGROUND AND RELATED WORK
Style Transfer. Huang & Belongie (017)) proposed transfer the style from a (source) image to another (target) that may have different content such that the content of the target image largely retains while features that are not essential to the content align with those the source image. Specifically, given an input image, say the portrait of actor Brad Pitt, and style picture, e.g., drawing painter Vincent van Gogh, the goal of style transfer is to produce a portrait of Brad Pitt that looks like picture painted by Vincent van Gogh. Existing approaches leverage various techniques to achieve this purpose. Gatys et al. (2016) utilized the feature representations in con- volutional layers of DNN extract content features and style features of input images. Given random white noise image, the algorithm feeds the image to the DNN to obtain the corresponding content and style features. The content features from the white noise image are compared with those from a content image, and the style features are contrasted with those from a style image. It then minimizes the above two differences to transform the noise image to a content image with style. Due to the inefficiency of this optimization process, researchers replace it with neural network that is trained minimize the same objective (Li & Wand 2016) Johnson al. 2016). Further study ex- tends these approaches to synthesize more than just one fixed style (Dumoulin et al.) [017]] Li et al.) 2017). Huang & Belongie (0177) introduced a simple yet effective approach, which can efficiently enable arbitrary style transfer. This approach proposes an adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Adversarial Attacks. In the context of image classification, given input æ, and DNN model (.), an adversary produces a sample such that M(x') + (x). If the output label y is chosen by the adversary in advance, then it is called a targeted attack, i.e., M(x') ) = yf M M(:);; Otherwise,
2

---

Under review as a conference paper at ICLR 2020
AC of
Encoder
Mor 6
Decoder
%
6
Decoder training phase
Target M
Encoder
Decoder
Encoder
(b) Feature space attack
Figure 2: Procedure of feature space adversarial attack. Two phases are involved during the attack generation process: (a) decoder training phase and (b) feature space attack phase.
it is an untargeted attack. Existing attacks, targeted untargeted, adopt ε-norm based metrics to measure the magnitude of introduced perturbation (small values correspond to little perceptibility to humans). Three (-norm metrics commonly used: fo. l2, and (x.. Specifically, to metric gauges the number of pixels modified when introducing the perturbation. Let ô x' - z be the perturbation, to can be calculated using | Hi]ō, + 0}/. The & metrie is the Euclidean distance which mea- sures the aggregated pixel changes. is computed with the following equation: ||81|2 = =VE, 87 The loo metric gauges the maximum change among all the modified pixels. The measurement of foo is processed with ||0]] max; |6g]. All three metrics are widely used as the standard mea- surement for evaluating stealthiness of adversarial attacks (Carlini & Wagner 2017 Madry et al.) 2018) and robustmess of DNN models (Xu et al.) 2018 Ma et al. (219)). In this paper, however, we find that these traditional pixel-spuce f-norm metrics do not fully reflect the stealthiness of feature space attacks. Figure I showcases adversarial samples generated using our feature space attack. All the generated samples have very large &o bound (> (>120/255), which least 7.5 times larger than the commonly used bounds in the literature of adversarial attacks (Kurakin et al. 201 Roth al. 2019) <c16/255 on ImageNet dataset). The &g bound is more than 20, substantially larger than those in the literature well, i.e., less than 0.5 in (Rony et al. 2019; Lecuyer al. 2019 Cohen al., 2019). However, observe that the adversarial samples have very natural style differences compared the original inputs.
FEATURE SPACE ATTACK
Overview. We aim to demonstrate that perturbation in the feature space can lead to model mis- behavior, which existing pixel space hardening techniques cannot effectively defend against. The hypothesis is that during training, the model picks up numerous features, many of which do not describe the key characteristics (or content) the object, but rather human imperceptible features such styles. These subtle features may play improperly important role in model prediction. As a result, injecting such features to a benign image can lead to misclassification. However, the feature space is not exposed attackers such that they cannot directly perturb features. Therefore, prominent challenge is to derive the corresponding pixel space mutation that appears natural to hu- mans while leading the intended feature space perturbation, and eventually the misclassification. In particular, the attack comprises two phases: (1) training a decoder that can translate feature space perturbation pixel level changes that look natural for humans; (2) launching the attack by first using gradient based optimization to identify feature space perturbation that can cause misclassifica- tion and then using the decoder to generate the corresponding adversarial example. Inspired by style transfer techniques, we consider much confined feature perturbation space style perturbation. Specifically, as in (Huang & Belongie 2017), we consider the mean and variance of the activations of inner layer denote the style of the features in that layer whereas the activations themselves denote the content features. We hence perturb the mean and variance of content features by per- forming predefined transformation that largely preserves the shape of the features while changing
3

---

Under review as a conference paper at ICLR 2020
the mean and variance. The decoder then decodes the perturbed feature values to an image closely resembles the original image with only style differences that appear natural to humans but causing model misclassification.
Figure 2 illustrates the workflow of the proposed attack. In the Decoder training phase (a), a set of image pairs with each pair from the same class (and hence their differences can be intuitively considered as style differences) are fed to a fixed Encoder that essentially consists of the first a few layers of a pre-trained model (e.g., VGG-19) (step ()), The Encoder produces the internal embeddings of the two respectively images, which correspond to the activation values of some inner layer in the pre-trained model, e.g., conv4_1 (step 2)). Each internal embedding consists of a number of matrices, one for every channel. For each embedding matrix, the mean and variance are computed. We use these values from the two input images to produce the integrated embedding A (step 3), which will be discussed in details later in this section. Intuitively, it is generated by performing shape-preet prepe-ppes transformation of the upper matrix so that it retains the content features denoted by the upper matrix while having the mean and variance of the lower matrix (i.e., the style denoted by the lower matrix). We employ a Decoder to reconstruct a raw image from A at step ④. which is supposed to have the content of the upper image (called the content image) and the style of the lower image (called the style image). To enable good reconstruction performance, two losses are utilized for optimizing the Decoder. The first one is the content loss. Specifically, at step (5) the reconstructed image is passed to the Encoder to acquire the reconstructed embedding (B, and then the difference between the integrated embedding A and the reconstructed embedding (B) minimized. The second one is the style loss. Particularly, the means and variances of few selected internal layers of the Encoder are computed for both the generated image and the original style image. The difference of these values of the two images is minimized. The Decoder optimization process is conducted on the original training dataset of target model M (under attack). Intuitively, the decoder is trained to understand the style differences so that it can decode feature style differences to realistic pixel space style differences, by observing the possible style differences.
When launching the attack ((b) in Figure 2). through the same process as in the Decoder training phase. The key differences are that only one input image is required and the Decoder is fixed in this phase. Given a target model M (under attack), the reconstructed image is fed to M at step c to yield prediction @ As the attack goal is to induce M to misclassify, the difference between prediction L and target output label (different from (8) is considered the adversarial loss for launching the attack. In addition, the content loss between A and (B) is also included. The attack updates the means and variances of embedding matrices at step ( with respect to the adversarial loss and style loss. The final reconstructed image that induces the target model M to misclassify is a successful adversarial sample.
test input image is fed to the Encoder and goes
3.1 DEFINITIONS
In this section, we formally define feature space attack. Considering a typical classification problem, where the samples æ E Ed and corresponding label y E {0, ..]...n] jointly obey a distribution D(æ, y). Given a classifier M : Rd + 10.1....,, mn with parameter 0. The goal of training to find the best parameter arg max, P(mw)-o/M(:: 8) y). Empirically, people associate a continuous loss function LM.e (xr, y/), e.g. cross-entropy, prediction and the true label. And the goal is rewritten as arg min, B(a.,))- (CM, ( v)]. We use Ca in short for Lare e in the following discussion. In adversarial learning, the adversary can introduce perturbation 8 S Rd into the natural samples (æ, v) ~ D. For a given sample x with label y. an adversary chooses the most malicious perturbation arg maxses Lss (x + 8, y) to make the classifier M predict incorrectly. Normally $ is confined as an bp-ball centered on O. In this case, the Ép norm of pixel space differences measures the distance between adversarial samples (i.e., x + 8 that causes misclassification) and the original samples. Thus we refer to this attack model as the pixel space attack. Most existing adversarial attacks fall into this category. Different from adding bounded perturbation in the pixel space, feature space attack applies perturbation in the feature space such that encoder (to extract the feature representation of the benign input) and decoder function (that translates perturbed feature values to a naturally looking image that closely resembles the original input in humans' perspective).
measure the difference between the
Formally, consider an encoder function f Rd - R and a decoder function f :R' + Rd. The former encodes sample an embedding b E TR and the latter restores embedding back to
4

---

Under review as a conference paper at ICLR 2020
sample. A perturbation function A : + R transforms given embedding to another. For given sample æ. the adversary chooses the best perturbation function to make the model M predict incorrectly.
max L [f f(z), aER
(1)
Functions f and /-1 need to satisfy additional properties to ensure the attack is meaningful. We call them the wellness properties of encoder and decoder.
Wellness of Encoder f. In order to get meaningful embedding. there ought to exist well- functioning classifier g based on the embedding. with a prediction error rate less than ó1.
y) 2 61. for a given 51. In practice, this property can be easily satisfied as one can construct g from a well-functioning classifier M, by decomposing M = M2 M1 and take M1 as f and M2 as g.
3g : IR 10, .,...,,),, n)
(2)
Wellness of Decoder f-1. Function f-1 is essentially a translator that translates what the adversary has done on the embedding back to sample in Rd We hence require that for all possible adversarial transformation a E A., f ought to retain what the adversary has applied to the embedding in the restored sample.
Va E A, let B a [(x), F(a.y)~DlSS (="( B S 82, for a given 82.
This ensures decoded (adversarial) sample induce the intended perturbation in the feature space. Note that f can always restore a benign sample back to itself. This equivalent to requiring the identity function in the perturbation function set A.
Given (f.f-1,A) satisfying the aforementioned properties, we define Equation as a feature space attack. Under this definition, pixel space attack is special case of feature space attack. For an lp- norm e-bounded pixel space attack, i.e., = {||8||, S e). we can rewrite it as a feature-space attack. Let encoder and decoder f-1 be an identity function and let U181
a(m) = m + 8).
pixel space allack def
Las + 6, = maax Lm (a(æ),,)) v] 16
max
aeA
max LaM [f oa f
Equation I
agA
One can easily verify the wellness of and f-1 Note that the stealthiness of feature space attack depends on the selection of A, analogous to that the stealthiness of pixel space attack depending on the lp norm. Next, we demonstrate two stealthy feature space attacks.
3.2
ATTACK DESIGN
3.2.1
DECODER TRAINING
Our decoder design is illustrated in Figure 2a It is inspired by style transfer in (Huang & Belongie 2017]. To train the decoder, we enumerate all the possible pairs of images each class in the original training set and use these pairs as a new training set. We consider each pair has the same content features (as they belong to the same class) and hence their differences essentially denote style differences. By training the decoder on all possible style differences (in the training set) re- gardless the output classes, we have a general decoder that can recognize and translate arbitrary style perturbation. Formally, given normal image 2, and another image æ., from the same class as Zp, the training process first passes them through pre-trained Encoder f (e.g.. VGG-19) to obtain embeddings 3" = f(x,), B ( E R II-CC, where C' is the channel size, and and the height and width of each channel. For each channel c, the mean and variance are computed across the spatial dimensions (step 2 in Figure 2ah. That is,
H
W
H Σ Σ
1
PB.
Blue
OB,
B MBB)2
(5)
HW
HW h= w=1
==1
We combine the embeddings BP, B from the two input images using the following equation:
B png
E 21. CJ, Bº = =0
+ MB
(6)
Dg
5

---

Under review as a conference paper at ICLR 2020
where B is the result embedding of channel c. Intuitively, the transformation retains the shape of B" while enforcing the mean and variance of B B is then fed to the Decoder f for recon- structing the image with the content of zp and the style of x (steps 3 & 4 in Figuur 2ap. In order to generate a realistic image, the reconstructed image is passed to Encoder f to acquire the recon- structed embedding B" = f f (BB) (step 5) The difference between the combined embedding 8a and the reconstructed embedding B", called the content loss, is minimized using the following equation during the Decoder training:
Coontent = || - B
(7)
In addition, some internal layers of Encoder f are selected, whose means and variances (computed by [qquatin 5) are used for representing the style of input images. The difference of these values between the style image By and the reconstructed image Lr, called the style loss, is minimized when training the Decoder. defined as follows:
Cstyle Σ EIIx(6((EE)) - ((o((x,)))2 + VEL
Σ ( (0.(22)))) İEL
(8)
where (bx-)) denotes layer of Encoder and L the set of layers considered. In this paper, consists of convl_1, conv2.1, conv3_1 I and conv4_1 for the ImageNet dataset, and co and conv2.1 for the CIFAR-10 and SVHN datasets. pu(-) and o(.) denote the mean and the variance, respectively. The Decoder training is to minimize Ccontent Latyle-
3.2.2 TWo FEATURE SPACE ATTACKS
Recall in the attack phase Figure 2b), the encoder and decoder are fixed. The style features of benign image are perturbed while the content features are retained, aiming trigger misclassi- fication. The pre-trained decoder then translates the perturbed embedding back to an adversarial sample. During perturbation, we focus on minimizing two loss functions. The first one is the adver- sarial loss Lis whose goal is to induce misclassification. The second one is similar to the content loss in the Decoder training (Equation 7). Intuitively, although the decoder is trained in a way that it is supposed to decode with minimal loss, arbitrary style perturbation may still cause substantial loss. Hence, such loss has to be considered and minimized during style perturbation.
With two different sets of transformations A, we devise two respective kinds of feature space attacks, feature augmentation attack and feature interpolation attack. For feature augmentation attack, at- tacker can change both the mean and standard deviation of each channel of the benign embedding independently. The boundary of increments or decrements are set by -nom under logarithm scale (to achieve stealthiness). Specifically, given two perturbation vectors T for the mean and 7" for the variance, both have the same dimension C as the embedding (denoting the C channels) and are bounded by the list of possible transformations A is defined as follows.
A U|| 11x Se and 1= T" TRERC { : a(B) (,,,, = =² (Br,w.e-uap (BB))+++ + [BBe] (9)
Note that μB denotes the means of embedding B for the C channels. The subscript denotes a specific channel. The transformation essentially enlarges the variance of the embedding at channel by factor of e and the mean by a factor of T.
For the feature interpolation attack, the attacker provides images as the style feature prototypes. Let S,,, Sy be the simplex determined by Uke(1.2....,)](((.r and (e]1.......!)(((( respectively. The attacker can modify the vectors of μa and Fg to be any point on the simplex.
BR - MBc
A Un, e aa a( B),, o;
(10)
BB
Intuitively, it enforces a style constructed from an interpolation of the k style prototypes.
Optimization. In pixel level attacks, two kinds of optimization techniques are widely used: Gra- dient Sign Method, e.g., PGD (Madry et al.] 2018). and using continuous function, e.g. tanh, to approximate and bound lec. e.g., C&W (Carlini & Wagner 2017). However in our context, we found these two techniques do not perform well. Using gradient sign tends to induce large con- tent loss while using tanh function inside the feature space empirically causes numerical instability. Instead, we use the iterative gradient method with gradient clipping. Specifically, We first calculate the gradient of loss L with respect to variables (c.g., T and T²). The gradient is then clipped by a constant related to the dimension of variables. || < 10/ 10///Dimes Diieension of variable. Then an Adam optimizer iteratively optimizes the variables using the clipped gradients.

---

Under review as a conference paper at ICLR 2020
4 EVALUATION
We evaluate feature space attack on 3 widely used image classification datasets and 5 state-of-the- art detection/defense approaches. All the target models (under attack) used in the experiments are pre-trained or trained using the code provided by the detection/defense approaches. We discuss how the detection/defense approaches perform under E-norm based pixel space attack in comparison with our feature space attack. Details are elaborated in the remainder of the section.
4.1 SETUP
Datasets. Three datasets are employed in the experiments: CIFAR-10 (Krizhevsky et al., 2009). ImageNet (Russakovsky et al. 2015) and SVHN (Netzer et al. 2011). CIFAR-10 is an object clas- sification dataset, which consists of 10 classes. ImageNet is one of the largest image classification datasets and comprises 1,000 categories of objects. SVHN is real-world digital recognition dataset with 10 classes of digits from 0 to 9.
Detection and Defense Approaches. We use 5 state-of-the-are detection and defense approaches to demonstrate the effectiveness of proposed feature space attack. Detection approaches aim to iden- tify adversarial samples while they provided to DNN. They often work as add-on to the model and do not aim to harden the model. We use a state-of-the-art adversarial example detection approach proposed by Roth et al. (20T9) to test our attack. Defense approaches, the other hand, harden models such that they are robust against ad adversarial example attacks. Existing state-of-the- art defense mechanisms either use adversarial training or certify a bound for each input image. We adopt 4 state-of-the-ee defense approaches in the literature (Madry et al. 2018: Zhang et al. 2019
Xie et al. 2019 Song et al., 2019) for evaluation.
Attack Settings. The two proposed feature space attacks have similar performance on various ex- perimental settings. Unless otherwise stated, we use feature augmentation attack as the default method. For the Encoder, we use VGG-19 from the input layer up to the relu4_1 for ImageNet, and up to relu2.1 for CIFAR-10 and SVHN To launch attacks, we set the loo - of embedding, in [Equation 9 to ln(1.5) for all the untargeted attacks and l2(2) for all the targeted attacks. We randomly select 1,000 images to perform the attacks on ImageNet. For CIFAR-10 and SVHN, we use all the inputs in the validation set.
4.2 ATTACK AGAINST DETECTION APPROACH
We use a state-of-the-arl adversarial sample detection approach "The Odds are Odd" (02) (Roth) et al. 2019) to demonstrate how feature space attack can evade it. 02 detects adversarial samples by adding random noise to input images and observing activation changing at a certain layer of a DNN. Specifically, 02 uses the penultimate layer (before the logits layer) as the representation of input images. It then defines a statistical variable that measures pairwise differences between two classes computed from the penultimate layer. The authors observed that adversarial samples differ significantly from benign samples regarding this variable when random noise is added. By performing statistical test on this variable, 02 is able detect PGD attacks (Madry et al. 2018) with over 99% detection rate on CIFAR-10 with bound foo = 8/ 8/255 and on ImageNet with foo 2/255. It also has over 90% detection rate against PGD and C&W (Carlini & Wagner 2017) attacks under &g metric on CIFAR-10. The & bounds were not given in the original paper (Roth et al. 2019).
Table shows the results of 02 on detecting different input samples. The first two columns are the datasets and models used for evaluation. The third column denotes the prediction accuracy of models on normal inputs. The following three columns present the detection rate of 02 on normal inputs, PGD adversarial samples, and feature space adversarial samples, respectively. The detection rate on normal inputs indicates that 02 falsely recognizes normal inputs as adversarial, which are essentially false positives. We can observe that 02 can effectively detect PGD attack on both datasets, but fails to detect feature space attack. Particularly, O2 has only 0.04% detection rate on CIFAR-10, which indicates that 02 is almost completely evaded by feature space attack. As for ImageNet. 02 can detect 25.30% of feature space adversarial samples but at the cost of a 19.20% false positive ratd !! The results show that 02 is ineffective against feature space attack.
'The parameters used for ImageNet are not given in the original paper. We can only reduce to this false positive rate after parameter tuning.
7

---

Under review as a conference paper at ICLR 2020
Table 1: O2 detection rate on normal inputs and adversarial samples.
Detection Rate
Dataset
Model
Accuracy
Normal
PGD
Feature Space
CIFAR-10 ResNet-18
91.95 75.20
0.95
99.61
0.04
ImageNet
ResNet-50
19.20
99.40
25.30
Table Evaluation of adversarial attacks against various defense approaches.
SVHN
CIFAR-10 10
ImageNet
Attack
Adaption
Madry
TRADES
Denoise (t,1)
Denoise (u,1)
Denoise (u,5)
None
84.84
77.84
84.97
61.25
61.25
78.12
PGD
52.84
41.43
54.02
42.60
12.50
27.15
Decoder
84.81
77.35
84.01
64.68
64.00
82.37
Feature Space
2.56
7.05
8.64
11.41
1.25
1.25
4.3
ATTACK AGAINST DEFENSE APPROACHES
We evaluate our feature space attack on 4 state-of-the-art adversarial training approaches: Madry (Madry et al. 2018), TRADES (Zhang et al. 2019), Denoise (Xie et al. 2019), and Adaption (Song al. 2019). For Denoise, the original paper only evaluated targeted attacks. We conduct ex- periments on both targeted and untargeted attacks. We hence use Denoise (t.1) to denote the top-1 accuracy of hardened model targeted attack and Denoise (u,5) the top-5 accuracy untargeted attack. We launch the PGD &oo attack as well as our feature space attack on the four defense ap- proaches. The experimental results demonstrate the performance of proposed feature space attack compared to existing f-norm based pixel space attack. Table 2 demonstrates the performance of adversarial attacks against various defense approaches. The first column denotes attack methods, where "None" presents the model accuracy on benign inputs and "Decoder" denotes the samples directly generated from the decoder without any feature space perturbation. The latter is show that the Decoder can generate faithful and natural images from embeddings. The following columns show different defense approaches (second row) applied on various datasets (first row). We can see that the PGD attack can reduce model accuracy to some extent when defense mechanisms are considered. Feature space attack, on the other hand, can effectively reduce model accuracy down to less than 12%, and most results are order of magnitude smaller than PGD. Especially, model accuracy on ImageNet only 1.25% when using untargeted attack, even in the presence of the defense technique. Interestingly, if images generated directly from the Decoder without any fea- ture space perturbation, the model accuracy improves on ImageNet. This indicates that the trained Decoder indeed captures the content feature of input images.
We study the f-norm distances in both the pixel space and the feature space for both pixel space attacks and feature space attacks. We observe that in the pixel space, the introduced perturbation by feature space attack is much larger than that of the PGD attack. However in the feature space, our attack has very similar distances as PGD, Figure and Figure (in Appendix 3A.I show that the adversarial samples have only style differences that are natural or even human imperceptible. Details can be found in Appendix A..I
We conducted an experiment to show that adversarial training in the pixel space does not improve model robustness against feature space attack and vice versa. Details can be found in Appendix A.A.] We have also studied the characteristics of the adversarial samples generated by different feature space attacks and attack settings. Please see Appendix A.2
CONCLUSIONS
We propose feature space adversarial attack on DNNs. It is based on perturbing style features and retaining content features. Such attacks inject natural style changes to input images to cause model misclassification. Since they usually cause substantial pixel space perturbations and existing detection/defense techniques are mostly for bounded pixel space attacks, these techniques are not effective for feature space attacks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Proceedings of 38th IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning (ICML), pp. 1310-1320, 2019.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In Proceedings of 5th International Conference Learning Representations (ICLR), 2017.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial exam- ples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 31-36, 2018.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Puttern Recog- nition (CVPR), pp. 2414-2423, 2016.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp- 1501-1510, 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference Computer Vision (ECCV), pp. 694-711, 2016.
Alex Krizhevsky et Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In Proceedings of 5th International Conference Learning Representations (ICLR), 2017. 2017.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In Proceedings of 40th IEEE Sym- posium on Security and Privacy (SP), 2019.
Chuan Li and Michael Wand. Precomputed real-time texture synthesis with markovian generative adversarial networks. In European Conference on Computer Vision (ECCV), pp. 702-716, 2016.
Shasha Li, Ajaya Neupane, Sujoy Paul, Chengyu Song, Srikanth Krishnamurthy, Amit K Roy Chowdhury, and Ananthram Swami. Adversarial perturbations against real-time video classifica- tion systems. In Proceedings of 26th Anmual Network and Distributed System Security Symposium (NDSS), 2019.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Diversified tex- ture synthesis with feed-forward networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3920-3928, 2017.
Shiqing Ma, Yingqi Liu, Guanhong Tao, Wen- Chuan Lee, and Xiangyu Zhang. Nic: Detecting ad- versarial samples with neural network invariant checking. In Proceedings of 26th Annual Network and Distributed System Security Symposium (NDSS), 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In Proceedings of 6th International Conference on Learning Representations (ICLR), 2018
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Yao Qin, Nicholas Carlini, Ian Goodfellow, Garrison Cottrell, and Colin Raffel. Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition. In Proceedings of the 36th International Conference on Machine Learning (ICMI., pp. 5231-5240, 2019.

---

Under review as a conference paper at ICLR 2020
Jérôme Rony, Luiz G Hafemann, Luiz S Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and norn for efficient gradient- based adversarial attacks and defenses. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 3322-4330,, 2019.
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting adversarial examples. In International Conference on Machine Learning (ICML), pp. 5498-5507, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252, 2015.
Chuanbiao Song, Kun He, Liwei Wang, and John E. Hoperoft. Improving the generalization of adversarial training with domain adaptation. In International Conference on Learning Represen- tations (ICLR), 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing Properties of Neural Networks. In International Conference on Learning Representations (ICLR), 2014.
Guanhong Tao, Shiqing Ma, Yingqi Liu, and Xiangyu Zhang. Attacks meet interpretability: Attribute-teered detection of adversarial samples. In Advances Neural Information Processing Systems (NeurIPS), pp. 7717-7728, 2018
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 501-509, 2019.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In Proceedings of 25th Annual Network and Distributed System Security Symposium (NDSS), 2018.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning (ICML), pp. 7472-7482, 2019.
A APPENDIX
A.1 MEASUREMENT PERTURBATION PIXEL AND FEATURE SPACES
To measure the magnitude of perturbation introduced by adversarial attacks, we use both foo and & distances. In addition, as we aim understand how the different attacks perturb the pixel space and the feature space, we compute the distances for both spaces. For the pixel space, the calculation is discussed in 22 For the feature space, we normalize the embeddings before distance calculation. For
each channel, we use h(())= f())-1/()) to normalize the embedding produced by the Encoder f())
f())
given input œ. The feature space difference hence can be computed as [|/(x)) - -+<=")) llp Table 3 Table 5) and Table 6) illustrate the magnitude of perturbation introduced by adversarial attacks on pixel and feature spaces for different hardened models. It can be observed that in the pixel space, the introduced perturbation by feature space attack is much larger than that of the PGD attack with l and & distances. In the feature space, however, our feature space attack does not induce large difference between normal inputs and adversarial samples. Particularly, the difference is similar or even smaller than that by the PGD attack. We further investigate the feature space distance between feature space adversarial samples and the corresponding normal images of their target labels. The to and & distances are 24.24 and 890 on the hardened (targeted) ImageNet model respectively, which are much larger than those (9.99 and 283) between the adversarial samples and the original (attacked) images. This indicates that feature space attack indeed leverages abstract features for generating adversarial samples. As shown in FFguure I and Figure the introduced perturbation is either insensitive to humans imperceptible.
10

---

Under review as a conference paper at ICLR 2020
We further explore the space targeted by different adversarial attacks using adversarial training. The assumption is that models hardened using an attack method are resilient to adversarial samples generated by the same attack method. Specifically, the base model. We then employ different attack approaches to generate adversarial samples against this base model both the training and validation sets. For each attack approach, we further retrain the base model using the corresponding adversarial samples. Finally, we test the performance of the retrained model different adversarial samples from the validation set. Table 4 presents the results. Each row represents the model retrained using the corresponding adversarial samples on the training set, where FA is feature augmentation attack and FI feature interpolation attack. Each column denotes the test adversarial samples generated on the validation set. We observe that retraining on the PGD attack does not improve model robustness against feature space attacks and vice versa. It indicates that the PGD attack and feature space attack exploit different spaces in generating adversarial samples.
a CIFAR- 10 model hardened by PGD as
Table 3: Magnitude of perturbation on hard- ened SVHN models.
Table 4: Model accuracy of adversarially trained models in different attack spaces.
Pixel Space
Feature Space
Model
PGD
FA
FI
Attack
1oo
100
l2
PGD
72.18
55.31
57.96
FA
54.68
84.06
75.31
PGD 0.02
1.04
11.50
53.78
46.56
60.93
87.81
Decoder
0.08
1.05
10.12
36.52
Feature Space
0.12
2.01
10.51
41.83
Table 5: Magnitude of perturbation hardened CIFAR-10 models.
Madry
TRADES
Attack
Pixle Space
Feature Space
Pixle Space
Feature Space
to
82
to
L2
Loo
L2
too
Eg
PGD
0.03
1.51
6.86
41.71
0.03
1.47
6.49
37.89
Decoder
0.19
1.85
4.29
25.13
0.18
1.85
4.30
25.15
Feature Space
0.27
4.08
6.88
43.38
0.28
4.72
7.43
46.43
Table Magnitude of perturbation on hardened ImageNet models.
Targeted
Untargeted
Attack
Pixle Space
Feature Space
Pixle Space
Feature Space
too
l2
too
fz
loo
lz
too
8
PGD
0.06
19.48
12.08
375
0.03
9.24
16.91
227
Decoder
0.87
44.98
8.26
193
0.88
44.67
15.24
214
Feature Space
0.89
69.07
9.99
283
0.86
54.89
16.09
236
A.2 TWO FEATURE SPACE ATTACKS
We generate and visually analyze the two feature space attacks. The ImageNet model hardened by feature denoising is used for generating adversarial samples. Columns (a), (d), and (g) in Figure 3) present the original images. Columns (b), (c), and (d) present the adversarial samples generate by the Encoder and the corresponding Decoder, with different encoder depths. Specifically, column (b) uses conv2.1 layers, column (c) uses conv3_1 layers and column (d) uses conv4_1 layers of a pre-trained VGG-19 19 as the Encoder, and the Decoders are of neural network structure similar to the corresponding Encoders but in a reverse order. Observe that as the Encoder becomes deeper, object outlines and textures changed in addition to colors. Columns (e) and (f) adversarial samples

---

Under review as a conference paper at ICLR 2020
generated by the feature argumentation attack (FA) and feature interpolation attack (FI). We observe that they both generate realistic images.
In column (h), only perturb the mean of embedding whereas in column (i) we only perturb the standard deviation of embedding. The results indicate that mean values tend to represent the background and the overall color tone. In contrast, the standard deviations tend to represent the object shape and relative color.
Decoder
Original
Original
FA
Original
Mean
Variance
Shallow
Deep
(b)
(c)
(d)
(e)
(h)
Figure 3: The adversarial samples from different feature space attack methods.
12


Reviews:
[{"review_id": "ryxidrATYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors have introduced a new type of adversarial attacks that perturb abstract features of the image. They have shown that pixel space adversarial attack detection and defense techniques are ineffective in guarding against feature space attacks.\n\nI have some concerns about the novelty of the attack and the appropriateness of defenses that have been tested.\n\n- Since the attack is done in the feature space, the defense should also be done in the feature space. For example, adversarial training or smoothing can be done in the feature space. See: https://arxiv.org/abs/1802.03471\n\n- There are attacks that perturb colors or other interpretable features of the image that have not been mentioned in the paper. For example, see https://arxiv.org/abs/1804.00499 and https://arxiv.org/pdf/1906.00001\n\n- If the decoder has a high Lipschitz constant, a small perturbation in the feature scape 'can' lead to a large and visible perturbation in the pixel space. It was not clear to me how this is being controlled in the current method. \n ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SklK3X3ptB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an adversarial attack method, which conducts perturbations in the feature spaces, instead of the raw image space. Specifically, the proposed method firstly learns an encoder that encodes features into the latent space, where style features are learned. At the same time, a decoder is learned to reconstruct the images with the encoded features. To conduct attacks, perturbations are added into the encoded features and attack images are generated with the decoder given the perturbated features. The experiment results look promising, showing that the proposed method achieves better attack performance with realistic adversarial images.\n\nThe general idea of perturbating the feature (latent) space is not a novel one, which has been studied in [1]. However, the proposed one is with an autoencoder framework instead of GAN used in [1]. Therefore, the proposed approach is able to construct adversarial examples for specific images. In addition, the training of the encoder is adapted from a style transfer method, which seems to learn good features that capture style features. \n\nIt is a bit unclear on the intuition of the constructions of Eq. (5) and (6). The details may be in Huang & Belongie, 2017. But it is better to provide more intuitive explanation and discussion on why these constructions capture style variation.\n\nThe results shown in the paper look promising. But it would be more comprehensive to compare with other pixel attacks in addition to PGD. Moreover, it is unclear whether it is a fair comparison between the proposed approach and pixel attacks, even under the same amount of perturbations. It would be good if the code will be released.\n\nMinor:\n\nLast sentence in the first paragraph of page 3: a missing reference.\n\n[1] Song, Yang, Rui Shu, Nate Kushman, and Stefano Ermon. \"Constructing unrestricted adversarial examples with generative models.\" In Advances in Neural Information Processing Systems, pp. 8312-8323. 2018.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlmoHxaKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting new adversarial attack technique that attempts to perturb abstract features learned by the target neural net. It is well written and easy to follow.  Its main contribution is the joining of ideas developed in the style transfer literature with those from the adversarial literature.  \n\nThe authors establish that they are able to create adversarial attacks that look similar to the original image but are miss classified. These images are not bounded by small epsilons, but are said to be indistinguishable by people. They illustrate a sample of these attacks, but no human study is employed to back up this claim. A simple human evaluation to prove that the attacks are indistinguishable from unperturbed images would strengthen the work (This can be done easily and at low cost by  employing mechanical Turk or an equivalent system for example).  \n\nThey make use of a detection mechanism (The Odds are Odd, Rothet al.) to verify that their adversarial attacks are hard to detect, but this particular detection mechanism has already  been broken (https://arxiv.org/pdf/1907.12138.pdf). If there is an as of yet unbroken detection mechanism that could be tested, that would improve the work. Alternatively the authors should acknowledge that there are simpler ways of evading this detection method.    \n\nThe attack that they propose targets the feature space, but no feature space detection methods are tested. The work would be improved by testing on a feature based detection methods such as dKNN (https://arxiv.org/pdf/1902.01889.pdf)\n\nOverall the work is interesting and novel, and creatively joins together two otherwise distinct areas of machine learning research to make a modest but novel contribution to the field.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an improved feature space adversarial attack.

However, the contribution is unclear in its significance, in part due to an important prior reference was omitted (song et al.) 

Unfortunately the paper is borderline, and not above the bar for acceptance in the current pool.
Decision: Reject