Unrestricted Adversarial Examples via Semantic Manipulation

Machine learning models , especially deep neural networks ( DNNs ) , have been shown to be vulnerable against adversarial examples which are carefully crafted samples with a small magnitude of the perturbation . Such adversarial perturbations are usually restricted by bounding their Lp norm such that they are imperceptible , and thus many current defenses can exploit this property to reduce their adversarial impact . In this paper , we instead introduce “ unrestricted ” perturbations that manipulate semantically meaningful image-based visual descriptors – color and texture – in order to generate effective and photorealistic adversarial examples . We show that these semantically aware perturbations are effective against JPEG compression , feature squeezing and adversarially trained model . We also show that the proposed methods can effectively be applied to both image classification and image captioning tasks on complex datasets such as ImageNet and MSCOCO . In addition , we conduct comprehensive user studies to show that our generated semantic adversarial examples are photorealistic to humans despite large magnitude perturbations when compared to other attacks . 1 INTRODUCTION . Machine learning ( ML ) , especially deep neural networks ( DNNs ) have achieved great success in various tasks , including image recognition ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) , speech processing ( Hinton et al. , 2012 ) and robotics training ( Levine et al. , 2016 ) . However , recent literature has shown that these widely deployed ML models are vulnerable to adversarial examples – carefully crafted perturbations aiming to mislead learning models ( Carlini & Wagner , 2017 ; Kurakin et al. , 2016 ; Xiao et al. , 2018b ) . The fast growth of DNNs based solutions demands in-depth studies on adversarial examples to help better understand potential vulnerabilities of ML models and therefore improve their robustness . To date , a variety of different approaches has been proposed to generate adversarial examples ( Goodfellow et al. , 2014b ; Carlini & Wagner , 2017 ; Kurakin et al. , 2016 ; Xiao et al. , 2018a ) ; and many of these attacks search for perturbation within a bounded Lp norm in order to preserve their photorealism . However , it is known that the Lp norm distance as a perceptual similarity metric is not ideal ( Johnson et al. , 2016 ; Isola et al. , 2017 ) . In addition , recent work show that defenses trained on Lp bounded perturbation are not robust at all against new types of unseen attacks ( Kang et al. , 2019 ) . Therefore , exploring diverse adversarial examples , especially those with “ unrestricted '' magnitude of perturbation has acquired a lot of attention in both academia and industries ( Brown et al. , 2018 ) . Recent work based on generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014a ) have introduced unrestricted attacks ( Song et al , 2018 ) . However , these attacks are limited to datasets like MNIST , CIFAR and CelebA , and are usually unable to scale up to bigger and more complex datasets such as ImageNet . Xiao et al . ( 2018b ) directly manipulated spatial pixel flow of an image to produce adversarial examples without Lp bounded constraints on the perturbation . However , the attack does not explicitly control visual semantic representation . More recently , Hosseini & Poovendran ( 2018 ) manipulated hue and saturation of an image to create adversarial perturbations . However , these examples are easily distinguishable by human and are also not scalable to complex datasets . ∗ indicates equal contributions . In this work , we propose unrestricted attack strategies that explicitly manipulate semantic visual representations to generate natural-looking adversarial examples that are “ far ” from the original image in tems of the Lp norm distance . In particular , we manipulate color ( cAdv ) and texture ( tAdv ) to create realistic adversarial examples ( see Fig 1 ) . cAdv adaptively chooses locations in an image to change their colors , producing adversarial perturbation that is usually fairly substantial , while tAdv utilizes the texture from other images and adjusts the instance ’ s texture field using style transfer . These semantic transformation-based adversarial perturbations shed light upon the understanding of what information is important for DNNs to make predictions . For instance , in one of our case studies , when the road is recolored from gray to blue , the image gets misclassified to tench ( a fish ) although a car remains evidently visible ( Fig . 2b ) . This indicates that deep learning models can easily be fooled by certain large scale patterns . In addition to image classifiers , the proposed attack methods can be generalized to different machine learning tasks such as image captioning ( Karpathy & Fei-Fei ( 2015 ) ) . Our attacks can either change the entire caption to the target ( Chen et al. , 2017 ; Xu et al. , 2019 ) or take on more challenging tasks like changing one or two specific target words from the caption to a target . For example , in Fig . 1 , “ stop sign ” of the original image caption is changed to “ cat sitting ” and “ umbrella is ” for cAdv and tAdv respectively . To ensure our “ unrestricted '' semantically manipulated images are natural , we conducted extensive user studies with Amazon Mechanical Turk . We also tested our proposed attacks on several state of the art defenses . Rather than just showing the attacks break these defenses ( better defenses will come up ) , we aim to show that cAdv and tAdv are able to produce new types of adversarial examples . Experiments also show that our proposed attacks are more transferable given their large and structured perturbations ( Papernot et al. , 2016 ) . Our semantic adversarial attacks provide further insights about the vulnerabilities of ML models and therefore encourage new solutions to improve their robustness . In summary , our contributions are : 1 ) We propose two novel approaches to generate “ unrestricted '' adversarial examples via semantic transformation ; 2 ) We conduct extensive experiments to attack both image classification and image captioning models on large scale datasets ( ImageNet and MSCOCO ) ; 3 ) We show that our attacks are equipped with unique properties such as smooth cAdv perturbations and structured tAdv perturbations . 4 ) We perform comprehensive user studies to show that when compared to other attacks , our generated adversarial examples appear more natural to humans despite their large perturbations ; 5 ) We test different adversarial examples against several state of the art defenses and show that the proposed attacks are more transferable and harder to defend . 2 COLORIZATION ATTACK ( cADV ) Background . Image Colorization is the task of giving natural colorization to a grayscale image . This is an ill-posed problem as there are multiple viable natural colorizations given a single grayscale image . Deshpande et al . ( 2017 ) showed that diverse image colorization can be achieved by using an architecture that combines VAE ( Kingma & Welling ( 2013 ) ) and Mixture Density Network ; while Zhang et al . ( 2017 ) demonstrated an improved and diverse image colorization by using input hints from users guided colorization process . Our goal is to adversarially color an image by leveraging a pretrained colorization model . We hypothesize that it is possible to find a natural colorization that is adversarial for a target model ( e.g. , classifier or captioner ) by searching in the color space . Since a colorization network learns to color natural colors that conform to boundaries and respect short-range color consistency , we can use it to introduce smooth and consistent adversarial noise with a large magnitude that looks natural to humans . This attack differs from common adversarial attacks which tend to introduce short-scale high-frequency artifacts that are minimized to be invisible for human observers . We leverage Zhang et al . ( 2016 ; 2017 ) colorization model for our attack . In their work , they produce natural colorizations on ImageNet with input hints from the user . The inputs to their network consist of the L channel of the image in CIELAB color space XL ∈ RH×W×1 , the sparse colored input hints Xab ∈ RH×W×2 , and the binary mask M ∈ BH×W×1 , indicating the location of the hints . cAdv Objectives . There are a few ways to leverage the colorization model to achieve adversarial objectives . We experimented with two main methods and achieved varied results . Network weights . The straightforward approach of producing adversarial colors is to modify Zhang et al . ( 2017 ) colorization network , C directly . To do so , we simply update C by minimizing the adversarial loss objective Jadv , which in our case , is the cross entropy loss . t represents target class and F represents the victim network . θ∗ = argmin θ Jadv ( F ( C ( XL , Xab , M ; θ ) ) , t ) ( 1 ) Hints and mask . We can also vary input hints Xab and mask M to produce adversarial colorizations . Hints provides the network with ground truth color patches that guides the colorization , while the mask provides its spatial location . By jointly varying both hints and mask , we are able to manipulate the output colorization . We can update the hints and mask as follows : M∗ , X∗ab = argmin M , Xab Jadv ( F ( C ( XL , Xab , M ; θ ) ) , t ) ( 2 ) cAdvAttack Methods . Attacking network weights allows the network to search the color space with no constraints for adversarial colors . This attack is the easiest to optimize , but the output colors are not realistic as shown in Fig . 2 . Our various strategies outlined below are ineffective as the model learns to generate the adversarial colors without taking into account color realism . However , colorizations produced often correlate with colors often observed in the target class . This suggests that classifiers associate certain colors with certain classes which we will discuss more in our case study . Attacking input hints and mask jointly gives us natural results as the pretrained network will not be affected by our optimization . Attacking hints and mask separately also works but takes a long optimization time and give slightly worse results . For our experiments , we use Adam Optimizer ( Kingma & Ba ( 2014 ) ) with a learning rate of 10−4 in cAdv . We iteratively update hints and mask until our adversarial image reaches the target class and the confidence change of consecutive iterations does not exceed a threshold of 0.05 . Control over colorization . Current attack methods lack control over where the attack occurs , opting to attack all pixels indiscriminately . This lack of control is not important for most attacks where the is small but is concerning in cAdv where making unstructured large changes can be jarring . To produce realistic colorization , we need to avoid making large color changes at locations where colors are unambiguous ( e.g . roads in general are gray ) and focus on those where colors are ambiguous ( e.g . an umbrella can have different colors ) . To do so , we need to segment an image and determine which segments should be attacked or preserved . To segment the image into meaningful areas , we cluster the image ’ s ground truth AB space using K-Means . We first use a Gaussian filter of σ = 3 to smooth the AB channels and then cluster them into 8 clusters . Then , we have to determine which cluster ’ s colors should be preserved . Fortunately , Zhang et al . ( 2017 ) network output a per-pixel color distribution for a given image which we used to calculate the entropy of each pixel . The entropy represents how confident the network is at assigning a color at that location . The average entropy of each cluster represents how ambiguous their color is . We want to avoid making large changes to clusters with low-entropy while allowing our attack to change clusters with high entropy . One way to enforce this behavior is through hints , which are sampled from the ground truth at locations belonging to clusters of low-entropy . We sample hints from the k clusters with the lowest entropy which we refer as cAdvk ( e.g . cAdv2 samples hints from the 2 lowest entropy clusters ) . Number of input hints . Network hints constrain our output to have similar colors as the ground truth , avoiding the possibility of unnatural colorization at the cost of color diversity . This trade-off is controlled by the number of hints given to the network as initialization ( Fig . 4 ) . Generally , providing more hints gives us similar colors that are observed in original image . However , having too many hints is also problematic . Too many hints makes the optimization between drawing adversarial colors and matching local color hints difficult . Since the search space for adversarial colors is constrained because of more hints , we may instead generate unrealistic examples . Number of Clusters . The trade-off between the color diversity and the color realism is also controlled by the number of clusters we sample hints from as shown in Fig . 3 . Sampling from multiple clusters gives us realistic colors closer to the ground truth image at the expense of color diversity . Empirically , from our experiments we find that in terms of color diversity , realism , and robustness of attacks , using k = 4 and 50 hints gives us better adversarial examples . For the rest of this paper , we fix 50 hints for all cAdvk methods . 3 TEXTURE ATTACK ( tADV ) Background . Texture transfer extracts texture from one image and adds it to another . Transferring texture from one source image to another target image has been widely studied in computer vision ( Efros & Freeman ( 2001 ) ; Gatys et al . ( 2015 ) ) . The Convolutional Neural Network ( CNN ) based texture transfer from Gatys et al . ( 2015 ) led to a series of new ideas in the domain of artistic style transfer ( Gatys et al . ( 2016 ) ; Huang & Belongie ( 2017 ) ; Li et al . ( 2017 ) ; Yeh et al . ( 2019 ) ) . More recently , Geirhos et al . ( 2018 ) showed that DNNs trained on ImageNet are biased towards texture for making predictions . Our goal is to generate adversarial examples by infusing texture from another image without explicit constraints on Lp norm of the perturbation . For generating our tAdv examples , we used a pretrained VGG19 network ( Simonyan & Zisserman , 2014 ) to extract textural features . We directly optimize our victim image ( Iv ) by adding texture from a target image ( It ) . A natural strategy to transfer texture is by minimizing within-layer feature correlation statistics ( gram matrices ) between two images Gatys et al . ( 2015 ; 2016 ) . Based on Yeh et al . ( 2019 ) , we find that optimizing cross-layer gram matrices instead of within-layer gram matrices helps produce more natural looking adversarial examples . The difference between the within-layer and the cross-layer gram matrices is that for a within-layer , the feature ’ s statistics are computed between the same layer . For a cross-layer , the statistics are computed between two adjacent layers . tAdv Objectives . tAdv directly attacks the image to create adversarial examples without modifying network parameters . Moreover , there is no additional content loss that is used in style transfer methods ( Gatys et al . ( 2016 ) ; Yeh et al . ( 2019 ) ) . Our overall objective function for the texture attack contains a texture transfer loss ( LAt ) and an cross-entropy loss ( Jadv ) . LAtAdv = αL A t ( Iv , It ) + βJadv ( F ( Iv ) , t ) ( 3 ) Unlike style transfer methods , we do not want the adversarial examples to be artistically pleasing . Our goal is to infuse a reasonable texture from a target class image to the victim image and fool a classifier or captioning network . To ensure a reasonable texture is added without overly perturbing the victim image too much , we introduce an additional constraint on the variation in the gram matrices of the victim image . This constraint helps us to control the image transformation procedure and prevents it from producing artistic images . Let m and n denote two layers of a pretrained VGG-19 with a decreasing spatial resolution and C for number of filter maps in layer n , our texture transfer loss is then given by LAt ( Iv , It ) = ∑ ( m , n ) ∈L 1 C2 ∑ ij ∥∥Gm , nij ( Iv ) −Gm , nij ( It ) ∥∥2 std ( Gm , nij ( Iv ) ) ( 4 ) Let f be feature maps , Ufn be an upsampled fn that matches the spatial resolution of layer m. The cross layer gram matrices G between the victim image ( Iv ) and a target image ( It ) is given as Gm , nij ( I ) = ∑ p [ fmi , p ( I ) ] [ Ufnj , p ( I ) ] T ( 5 ) Texture Transfer . To create tAdv adversarial examples , we need to find images to extract the texture from , which we call “ texture source ” ( Ts ) . A naive strategy is to randomly select an image from the data bank as Ts . Though this strategy is successful , their perturbations are clearly perceptible . Alternatively , we can randomly select Ts from the adversarial target class . This strategy produces less perceptible perturbations compared to the random Ts method as we are extracting a texture from the known target class . A better strategy to select Ts is to find a target class image that lies closest to the victim image in the feature space using nearest neighbors . This strategy is sensible as we assure our victim image has similar feature statistics as our target image . Consequently , minimizing gram matrices is easier and our attack generates more natural looking images ( see Fig . 5 ) . For texture transfer , we extract cross-layer statistics in Eq . 4 from the R11 , R21 , R31 , R41 , and R51 of a pretrained VGG19 . We optimize our objective ( Eq . 3 ) using an L-BFGS ( Liu & Nocedal ( 1989 ) ) optimizer . tAdv attacks are sensitive and if not controlled well , images get transformed into artistic images . Since we do not have any constraints over the perturbation norm , it is necessary to decide when to stop the texture transfer procedure . For a successful attack ( images look realistic ) , we limit our L-BFGS to fixed number of small steps and perform two set of experiments : one with only one iteration or round of L-BFGS for 14 steps and another with three iterations of 14 steps . For the three iterations setup , after every iteration , we look at the confidence of our target class and stop if the confidence is greater than 0.9 . Texture and Cross-Entropy Weights . Empirically , we found setting α to be in the range [ 150 , 1000 ] and β in the range [ 10−4 , 10−3 ] to be successful and also produce less perceptible tAdv examples . The additional cross-entropy based adversarial objective Jadv helps our optimization . We ensure large flow of gradients is from the texture loss and they are sufficiently larger than the adversarial crossentropy objective . The adversarial objective also helps in transforming victim image to adversarial without stylizing the image . All our tabulated results are shown for one iteration , α = 250 and β = 10−3 , unless otherwise stated . We use the annotation tAdviterα for the rest of the paper to denote the texture method that we are using . Control over Texture . The amount of texture that gets added to our victim image is controlled by the texture weight coefficient ( α ) . Increasing texture weights improves attack success rate at the cost of noticeable perturbation . When compared to within-layer statistics , the cross-layer statistics that we use are not only better at extracting texture , it is also easier to control the texture weight .

Summary: This paper introduces two new adversarial attacks: one is generating adversarial examples by colouring the original images and the other is by changing textures of the original images. Specifically, the former one minimises the cross-entropy between the output of the classifier and the target label with the network weights of a pre-trained colourisation network. While the latter minimises the cross-entropy as well as the loss that defines the texture differences.