TOWARDS FEATURE SPACE ADVERSARIAL ATTACK

1 INTRODUCTION . Adversarial attacks are a prominent threat to the broad application of Deep Neural Networks ( DNNs ) . In the context of classification applications , given a pre-trained model M and a benign input x of some output label y , adversarial attack perturbs x such thatM misclassifies the perturbed x . The perturbed input is called adversarial example . Such perturbations are usually bounded by some distance norm such that they are not perceptible by humans . Ever since it was proposed in ( Szegedy et al. , 2014 ) , there has been a large body of research that develops various methods to construct adversarial examples , e.g. , ( Carlini & Wagner , 2017 ; Madry et al. , 2018 ) , with different modalities such as images ( Carlini & Wagner , 2017 ) , audio ( Qin et al. , 2019 ) , text ( Ebrahimi et al. , 2018 ) , and video ( Li et al. , 2019 ) , detect adversarial examples ( Tao et al. , 2018 ; Ma et al. , 2019 ) , and use adversarial examples to harden the models ( Madry et al. , 2018 ; Zhang et al. , 2019 ) . However , most existing attacks are in the pixel space , that is , the perturbations occur directly in the pixel space , and the pixel level differences between the adversarial example and the original input are bounded . In this paper , we illustrate that adversarial attack can be conducted in the feature space . The underlying assumption ( in the context of image classification ) is that during training , a DNN may extract a large number of abstract features . While many of them denote critical characteristics of the object , some of them are secondary , for example , the different styles of an image ( e.g. , vivid colors versus pale colors , sharp outlines versus blur outlines ) . These secondary features may play an improperly important role in model prediction . As a result , feature space attack can inject such secondary features , which are not simple pixel perturbation , but rather functions over the given benign input , to induce model misclassification . Since humans are not sensitive to these features , the resulted adversarial examples look very natural from humans ’ perspective . As many of these features are pervasive , the resulted pixel space perturbation may be relatively much more substantial than existing pixel space attacks . As such , pixel space detection and hardening techniques are ineffective for feature space attacks ( see §4 ) . Figure 1 shows a number of adversarial examples generated by our technique , their comparison with the original examples , and the pixel space distances . Observe that while the distances are much larger compared to those in pixel space attacks , the adversarial examples are completely natural , or even indistinguishable from the original inputs in humans ’ eyes . The contrast of the benign-adversarial pairs illustrates that the malicious perturbations largely co-locate with the primary content features , denoting imperceptible tweaking of these features . Under the hood , we consider that the activations of an inner layer represent a set of abstract features , including those primary and secondary . To avoid generating adversarial examples that are unnatural , we refrain from tampering with the primary features ( or content features ) and focus on perturbing the secondary style features . Inspired by the recent advance in style transfer ( Huang & Belongie , 2017 ) , the mean and variance of activations are considered the style . As such , we focus on perturbing the means and variances while preserving the shape of the activation values ( i.e. , the up-and-downs of these values and the relative scale of such up-and-downs ) . We use gradient driven optimization to search for the style perturbations that can induce misclassification . Since our threat model is the same as existing pixel space attacks , that is , the attack is launched by providing the adversarial example to the model . An important step is to translate the activations with style changes back to a naturally looking pixel space example . We address the problem by considering the differences of any pair of training inputs of the same class as the possible style differences , and pre-training a decoder that can automatically impose styles in the pixel space based on the style feature perturbation happening in an inner layer . We propose two concrete feature space attacks , one to enhance styles and the other to impose styles constituted from a set of pre-defined style prototypes . We evaluate our attacks on 3 datasets and 7 models . We show that feature space attacks can effectively generate adversarial samples that evade 5 state-of-the-art detection/defense approaches . Particularly , our proposed attack can reduce the detection rate of a state-of-the-art approach ( Roth et al. , 2019 ) to 0.04 % on the CIFAR-10 dataset , and the prediction accuracy of a model hardened by a state-of-art adversarial training technique ( Xie et al. , 2019 ) to 1.25 % on ImageNet . Moreover , we observe that despite the large distance introduced in the pixel space , the distances in feature space are similar or even smaller than those in ` -norm based attacks . The generated adversarial examples have only natural , and in many cases , human imperceptible style differences compared with the original inputs , demonstrating the practicality of the attacks . This work only demonstrates the feasibility of feature space attacks . The features we are attacking are relatively simple . In the future , we expect more research on complicated feature space attacks . 2 BACKGROUND AND RELATED WORK . Style Transfer . Huang & Belongie ( 2017 ) proposed to transfer the style from a ( source ) image to another ( target ) that may have different content such that the content of the target image largely retains while features that are not essential to the content align with those of the source image . Specifically , given an input image , say the portrait of actor Brad Pitt , and a style picture , e.g. , a drawing of painter Vincent van Gogh , the goal of style transfer is to produce a portrait of Brad Pitt that looks like a picture painted by Vincent van Gogh . Existing approaches leverage various techniques to achieve this purpose . Gatys et al . ( 2016 ) utilized the feature representations in convolutional layers of a DNN to extract content features and style features of input images . Given a random white noise image , the algorithm feeds the image to the DNN to obtain the corresponding content and style features . The content features from the white noise image are compared with those from a content image , and the style features are contrasted with those from a style image . It then minimizes the above two differences to transform the noise image to a content image with style . Due to the inefficiency of this optimization process , researchers replace it with a neural network that is trained to minimize the same objective ( Li & Wand , 2016 ; Johnson et al. , 2016 ) . Further study extends these approaches to synthesize more than just one fixed style ( Dumoulin et al. , 2017 ; Li et al. , 2017 ) . Huang & Belongie ( 2017 ) introduced a simple yet effective approach , which can efficiently enable arbitrary style transfer . This approach proposes an adaptive instance normalization ( AdaIN ) layer that aligns the mean and variance of the content features with those of the style features . Adversarial Attacks . In the context of image classification , given an input x , and a DNN model M ( · ) , an adversary produces a sample x′ such that M ( x′ ) 6=M ( x ) . If the output label y is chosen by the adversary in advance , then it is called a targeted attack , i.e. , M ( x′ ) =y 6=M ( x ) ; Otherwise , it is an untargeted attack . Existing attacks , targeted or untargeted , adopt ` -norm based metrics to measure the magnitude of introduced perturbation ( small values correspond to little perceptibility to humans ) . Three ` -norm metrics are commonly used : ` 0 , ` 2 , and ` ∞ . Specifically , ` 0 metric gauges the number of pixels modified when introducing the perturbation . Let δ=x′−x be the perturbation , ` 0 can be calculated using ‖δ‖0 = ∣∣ { i|δi 6= 0 } ∣∣ . The ` 2 metric is the Euclidean distance which measures the aggregated pixel changes . It is computed with the following equation : ‖δ‖2 = √∑ i δ 2 i . The ` ∞ metric gauges the maximum change among all the modified pixels . The measurement of ` ∞ is processed with ‖δ‖∞ = maxi |δi| . All three metrics are widely used as the standard measurement for evaluating stealthiness of adversarial attacks ( Carlini & Wagner , 2017 ; Madry et al. , 2018 ) and robustness of DNN models ( Xu et al. , 2018 ; Ma et al. , 2019 ) . In this paper , however , we find that these traditional pixel-space ` -norm metrics do not fully reflect the stealthiness of feature space attacks . Figure 1 showcases adversarial samples generated using our feature space attack . All the generated samples have very large ` ∞ bound ( > 120/255 ) , which is at least 7.5 times larger than the commonly used bounds in the literature of adversarial attacks ( Kurakin et al. , 2017 ; Roth et al. , 2019 ) ( < 16/255 on ImageNet dataset ) . The ` 2 bound is more than 20 , substantially larger than those in the literature as well , i.e. , less than 0.5 in ( Rony et al. , 2019 ; Lecuyer et al. , 2019 ; Cohen et al. , 2019 ) . However , observe that the adversarial samples have very natural style differences compared to the original inputs . 3 FEATURE SPACE ATTACK . Overview . We aim to demonstrate that perturbation in the feature space can lead to model misbehavior , which existing pixel space hardening techniques can not effectively defend against . The hypothesis is that during training , the model picks up numerous features , many of which do not describe the key characteristics ( or content ) of the object , but rather human imperceptible features such as styles . These subtle features may play an improperly important role in model prediction . As a result , injecting such features to a benign image can lead to misclassification . However , the feature space is not exposed to attackers such that they can not directly perturb features . Therefore , a prominent challenge is to derive the corresponding pixel space mutation that appears natural to humans while leading to the intended feature space perturbation , and eventually the misclassification . In particular , the attack comprises two phases : ( 1 ) training a decoder that can translate feature space perturbation to pixel level changes that look natural for humans ; ( 2 ) launching the attack by first using gradient based optimization to identify feature space perturbation that can cause misclassification and then using the decoder to generate the corresponding adversarial example . Inspired by style transfer techniques , we consider a much confined feature perturbation space – style perturbation . Specifically , as in ( Huang & Belongie , 2017 ) , we consider the mean and variance of the activations of an inner layer denote the style of the features in that layer whereas the activations themselves denote the content features . We hence perturb the mean and variance of content features by performing a predefined transformation that largely preserves the shape of the features while changing the mean and variance . The decoder then decodes the perturbed feature values to an image closely resembles the original image with only style differences that appear natural to humans but causing model misclassification . Figure 2 illustrates the workflow of the proposed attack . In the Decoder training phase ( a ) , a set of image pairs with each pair from the same class ( and hence their differences can be intuitively considered as style differences ) are fed to a fixed Encoder that essentially consists of the first a few layers of a pre-trained model ( e.g. , VGG-19 ) ( step 1© ) . The Encoder produces the internal embeddings of the two respectively images , which correspond to the activation values of some inner layer in the pre-trained model , e.g. , conv4 1 ( step 2© ) . Each internal embedding consists of a number of matrices , one for every channel . For each embedding matrix , the mean and variance are computed . We use these values from the two input images to produce the integrated embedding A© ( step 3© ) , which will be discussed in details later in this section . Intuitively , it is generated by performing a shape-preserving transformation of the upper matrix so that it retains the content features denoted by the upper matrix while having the mean and variance of the lower matrix ( i.e. , the style denoted by the lower matrix ) . We employ a Decoder to reconstruct a raw image from A© at step 4© , which is supposed to have the content of the upper image ( called the content image ) and the style of the lower image ( called the style image ) . To enable good reconstruction performance , two losses are utilized for optimizing the Decoder . The first one is the content loss . Specifically , at step 5© the reconstructed image is passed to the Encoder to acquire the reconstructed embedding B© , and then the difference between the integrated embedding A© and the reconstructed embedding B© is minimized . The second one is the style loss . Particularly , the means and variances of a few selected internal layers of the Encoder are computed for both the generated image and the original style image . The difference of these values of the two images is minimized . The Decoder optimization process is conducted on the original training dataset of target model M ( under attack ) . Intuitively , the decoder is trained to understand the style differences so that it can decode feature style differences to realistic pixel space style differences , by observing the possible style differences . When launching the attack ( ( b ) in Figure 2 ) , a test input image is fed to the Encoder and goes through the same process as in the Decoder training phase . The key differences are that only one input image is required and the Decoder is fixed in this phase . Given a target model M ( under attack ) , the reconstructed image is fed to M at step 6© to yield prediction E© . As the attack goal is to induce M to misclassify , the difference between prediction E© and a target output label ( different from E© ) is considered the adversarial loss for launching the attack . In addition , the content loss between A© and B© is also included . The attack updates the means and variances of embedding matrices at step 7© with respect to the adversarial loss and style loss . The final reconstructed image that induces the target model M to misclassify is a successful adversarial sample .

Summary: This paper presents an adversarial attack method, which conducts perturbations in the feature spaces, instead of the raw image space. Specifically, the proposed method firstly learns an encoder that encodes features into the latent space, where style features are learned. At the same time, a decoder is learned to reconstruct the images with the encoded features. To conduct attacks, perturbations are added into the encoded features and attack images are generated with the decoder given the perturbated features. The experiment results look promising, showing that the proposed method achieves better attack performance with realistic adversarial images.