Title: SEMANTICADV: GENERATING ADVERSARIAL EXAM-PLES VIA ATTRIBUTE-CONDITIONAL IMAGE EDITING

Abstract: Deep neural networks (DNNs) have achieved great success in various applications due to their strong expressive power. However, recent studies have shown that DNNs are vulnerable to adversarial examples which are manipulated instances targeting to mislead DNNs to make incorrect predictions. Currently, most such adversarial examples try to guarantee "subtle perturbation" by limiting the L p norm of the perturbation. In this paper, we aim to explore the impact of semantic manipulation on DNNs predictions by manipulating the semantic attributes of images and generate "unrestricted adversarial examples". In particular, we propose an algorithm SemanticAdv which leverages disentangled semantic factors to generate adversarial perturbation by altering controlled semantic attributes to fool the learner towards various "adversarial" targets. We conduct extensive experiments to show that the semantic based adversarial examples can not only fool different learning tasks such as face verification and landmark detection, but also achieve high targeted attack success rate against real-world black-box services such as Azure face verification service based on transferability. To further demonstrate the applicability of SemanticAdv beyond face recognition domain, we also generate semantic perturbations on street-view images. Such adversarial examples with controlled semantic manipulation can shed light on further understanding about vulnerabilities of DNNs as well as potential defensive approaches.

1. INTRODUCTION:

Deep neural networks (DNNs) have demonstrated great successes in advancing the state-of-the-art performance of discriminative tasks (Krizhevsky et al., 2012; Goodfellow et al., 2016; He et al., 2016; Collobert & Weston, 2008; Deng et al., 2013; Silver et al., 2016) . However, recent research found that DNNs are vulnerable to adversarial examples which are carefully crafted instances aiming to induce arbitrary prediction errors for learning systems. Such adversarial examples containing small magnitude of perturbation have shed light on understanding and discovering potential vulnerabilities of DNNs (Szegedy et al., 2013; Goodfellow et al., 2014b; Moosavi-Dezfooli et al., 2016; Papernot et al., 2016; Carlini & Wagner, 2017; Xiao et al., 2018b; c; a; 2019) . Most existing work focused on constructing adversarial examples by adding L p bounded pixel-wise perturbations (Goodfellow et al., 2014b) or spatially transforming the image (Xiao et al., 2018c; Engstrom et al., 2017 ) (e.g., in-plane rotation or out-of-plane rotation). Generating unrestricted perturbations with semantically meaningful patterns is an important yet under-explored field.

At the same time, deep generative models have demonstrated impressive performance in learning disentangled semantic factors through data generation in an unsupervised (Radford et al., 2015; Karras et al., 2018; Brock et al., 2019) or weakly-supervised manner based on semantic attributes (Yan et al., 2016; Choi et al., 2018) . Empirical findings in (Yan et al., 2016; Zhu et al., 2016a; Radford et al., 2015) demonstrated that a simple linear interpolation on the learned image manifold can produce smooth visual transitions between a pair of input images.

In this paper, we introduce a novel attack SemanticAdv which generates unrestricted perturbations with semantically meaningful patterns. Motivated by the findings mentioned above, we leverage an attribute-conditional image editing model (Choi et al., 2018) to synthesize adversarial examples by interpolating between source and target images in the feature-map space. Here, we focus on changing a single attribute dimension to achieve adversarial goals while keeping the generated adversarial image reasonably-looking (e.g., see Figure 1 ). To validate the effectiveness of the proposed attack method, we consider two tasks, namely, face verification and landmark detection, as face recognition field has been extensively explored and the commercially used face models are relatively robust since they require a low false positive rate. We conduct both qualitative and quantitative evaluations on CelebA dataset (Liu et al., 2015) . To demonstrate the applicability of SemanticAdv beyond face domain, we further extend SemanticAdv to generate adversarial street-view images. We treat semantic layouts as input attributes and use the image editing model (Hong et al., 2018) pre-trained on Cityscape dataset (Cordts et al., 2016) . Please find more visualization results on the anonymous website: https://sites.google.com/view/generate-semantic-adv-example.

The contributions of the proposed SemanticAdv are three-folds. First, we propose a novel semanticbased attack method to generate unrestricted adversarial examples by feature-space interpolation. Second, the proposed method is able to generate semantically-controllable perturbations due to the attribute-conditioned modeling. This allows us to analyze the robustness of a recognition system against different types of semantic attacks. Third, as a side benefit, the proposed attack exhibits high transferability and leads to a 65% query-free black-box attack success rate on a real-world face verification platform, which outperforms the pixel-wise perturbations in attacking existing defense methods.

2. RELATED WORK:

Semantic image editing. Semantic image synthesis and manipulation is a popular research topic in machine learning, graphics and vision. Thanks to recent advances in deep generative models (Kingma & Welling, 2014; Goodfellow et al., 2014a; Oord et al., 2016) and the empirical analysis of deep classification networks (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015) , past few years have witnessed tremendous breakthroughs towards high-fidelity pure image generation (Radford et al., 2015; Karras et al., 2018; Brock et al., 2019) , attribute-to-image generation (Yan et al., 2016; Choi et al., 2018) , text-to-image generation (Mansimov et al., 2015; Reed et al., 2016; Van den Oord et al., 2016; Odena et al., 2017; Zhang et al., 2017; Johnson et al., 2018) , and imageto-image translation (Isola et al., 2017; Zhu et al., 2017; Liu et al., 2017; Wang et al., 2018b; Hong et al., 2018) .

Adversarial examples. Generating L p bounded adversarial perturbation has been extensively studied recently (Szegedy et al., 2013; Goodfellow et al., 2014b; Moosavi-Dezfooli et al., 2016; Papernot et al., 2016; Carlini & Wagner, 2017; Xiao et al., 2018b Joshi et al. (2019) proposed to generate semantic-based attacks against a restricted binary classifier, while our attack is able to mislead the model towards arbitrary adversarial targets. They conduct the manipulation within the attribution space which is less flexible and effective than our proposed feature-space interpolation. In this work, we introduce the concept of semantic attacker that aims at generating adversarial examples by adding semantically meaningful perturbation with a conditional generative model G.

3. SEMANTIC ADVERSARIAL EXAMPLES:

Compared to traditional attacker that usually produces pixel-wise perturbations, the proposed method is able to produce semantically meaningful perturbations.

Semantic image editing. For simplicity, we start with the formulation where the input attribute is represented as a compact vector. This formulation can be directly extended to other input attribute formats including semantic layouts. Let c ∈ R D C be an attribute representation reflecting the semantic factors (e.g., expression or hair color of a portrait image) of image x, where D C indicates the attribute dimension and c i ∈ {0, 1} indicates the appearance of i-th attribute. Here, our goal is to use the conditional generator for semantic image editing. For example, given a portrait image of a girl with black hair and blonde hair as the new attribute, our generator is supposed to synthesize a new image that turns the girl's hair from black to blonde. More specifically, we denote the new attribute as c new ∈ R D C such that the synthesized image is given by x new = G(x, c new ). In the special case when there is no attribute change (c = c new ), the generator simply reconstructs the input: x = G(x, c). Supported by the findings mentioned in (Bengio et al., 2013; Reed et al., 2014) , our synthesized image x new should fall close to the data manifold if we constrain the change of attribute values to be sufficiently small (e.g., we only update one semantic attribute at a time). In addition, we can potentially generate many such images by linearly interpolating between the semantic embeddings of the conditional generator G using original image x and the synthesized image x new with the augmented attribute.

Attribute-space interpolation. We start with a simple solution (detailed in Eq. 1) assuming the adversarial example can be found by directly interpolating in the attribute-space. Given a pair of attributes c and c new , we introduce an interpolation parameter α ∈ (0, 1) to generate the augmented attribute vector c * ∈ R D C (see Eq. 1). Given augmented attribute c * and original image x, we produce the synthesized image by the generator G. For our notation purpose, we also introduce a delegated function T G as a re-parametrization for the generator G. Our formulation is also supported by the empirical results on attribute-conditioned image progression (Yan et al., 2016; Radford et al., 2015) that a well-trained generative model has the capability to synthesize a sequence of images with smooth attribute transitions.

x adv = argmin α L(T G (α; x, c, c new )) where T G (α; x, c, c new ) = G(x, c * ) and c * = α • c + (1 -α) • c new (1)

Feature-map interpolation. Alternatively, we propose to interpolate using the feature map produced by the generator G = G dec • G enc . Here, G enc is the encoder module that takes the image as input and outputs the feature map. Similarly, G dec is the decoder module that takes the feature map as input and outputs the synthesized image. Let f * = G enc (x, c) ∈ R H F ×W F ×C F be the feature map of an intermediate layer in the generator, where H F , W F and C F indicate the height, width, and number of channels in the feature map.

x adv = argmin α L(T G (α; x, c, c new )) where T G (α; x, c, c new ) = G dec (f * ), f * = α G enc (x, c) + (1 -α) G enc (x, c new ) (2)

Compared to attribute-space interpolation which is parameterized by a scalar, we parameterize feature-map interpolation by a tensor α ∈ R H F ×W F ×C F (α h,w,k ∈ (0, 1), where 1 ≤ h ≤ H F , 1 ≤ w ≤ W F , and 1 ≤ k ≤ C F ) with the same shape as the feature map. Compared to linear interpolation over attribute-space, such design introduces more flexibility when interpolating between the original image and the synthesized image. Empirical results in Section 4.2 show our design is critical to the adversarial attack success rate.

3.2. ADVERSARIAL OPTIMIZATION OBJECTIVES:

As we see in Eq. 3, we obtain the adversarial image x adv by minimizing the objective L(•) with respect to the synthesized image T G (α; x, c, c new ), which is defined in Eq.( 1) and Eq.( 2) respectively.

Here, each synthesized image T G (α; x, c, c new ) is produced by the interpolation using the conditional generator G. In our objective function, the first term is the adversarial metric, the second term is a smoothness constraint, and λ is used to control the balance between the two terms. The adversarial metric is minimized once the model M has been successfully attacked towards the target image-label pair (x tgt , y tgt ). For identify verification, y tgt is the identity representation of the target image; For structured prediction tasks in our paper, y tgt either represents certain coordinates (landmark detection) or semantic label maps (semantic segmentation).

EQUATION

Identity verification. In the identity verification task, two images are considered to be the same identity if the corresponding identity embeddings from the verification model M are reasonably close.

EQUATION

As we see in Eq. 4, Φ id M (•, •) measures the distance between two identity embeddings from the model M, where the normalized L 2 distance is used in our setting. In addition, we introduce the parameter κ representing the constant related to the false positive rate (FPR) threshold computed from the development set.

Structured prediction. For structured prediction tasks such as landmark detection and semantic segmentation, we use Houdini objective proposed in Cisse et al. (2017) as our adversarial metric and select the target landmark (semantic segmentation) target as y tgt . In addition, Φ M (•, •) is a scoring function for each image-label pair and γ is the threshold.

EQUATION

where y * = M(T G (α; x, c, c new )) and l(y * , y tgt ) is task loss decided by the specific adversarial target.

Interpolation smoothness L smooth . As the tensor to be interpolated in the feature-map space has far more parameters compared to the attribute itself, we propose to enforce a smoothness constraint on the tensor α used in feature-map interpolation. As we see in Eq. 6, the smoothness loss encourages the interpolation tensors to consist of piece-wise constant patches spatially, which has been widely used as a pixel-wise de-noising objective for natural image processing (Mahendran & Vedaldi, 2015; Johnson et al., 2016) .

L smooth (α) = H F -1 h=1 W F w=1 α h+1,w -α h,w 2 2 + H F h=1 W F -1 w=1 α h,w+1 -α h,w 2 2 (6)

4. EXPERIMENTS:

In the experimental section, we mainly focus on analyzing the proposed SemanticAdv in attacking state-of-the-art face recognition systems on CelebA (Liu et al., 2015) due to its wide applicability (e.g., identification for mobile payment) in the real world. In addition, we extend our attack to urban street scenes with semantic label maps as the condition. We attack the semantic segmentation model DRN-D-22 (Yu et al., 2017) previously trained on Cityscape (Cordts et al., 2016) by generating adversarial examples with dynamic objects manipulated (e.g., insert a car into the scene).

The experimental section is organized as follows. First, we analyze the quality of generated adversarial examples and qualitatively compare our method with L p bounded pixel-wise optimization-based method (Carlini & Wagner, 2017; Dong et al., 2018; Xie et al., 2019) . Second, we provide both qualitative and quantitative results by controlling each of the semantic attributes at a time. In terms of attack transferability, we evaluate our proposed SemanticAdv on various settings and further demonstrate the effectiveness of our method via query-free black-box attacks against online face verification platforms. Third, we compare our method with the baseline against different defense methods on the face verification task. Fourth, we demonstrate that the proposed SemanticAdv also applies to the face landmark detection and street-view semantic segmentation.

4.1 EXPERIMENTAL SETUP Face identity verification. We select ResNet-50 and ResNet-101 (He et al., 2016) trained on MS-Celeb-1M (Guo et al., 2016) as our face verification models. The models are trained using two different objectives, namely, softmax loss (Sun et al., 2014; Zhang et al., 2018) and cosine loss (Wang et al., 2018a) . For simplicity, we use the notation "R-N-S" to indicate the model with N -layer residual blocks as backbone trained using softmax loss, while "R-N-C" indicates the same backbone trained using cosine loss. The distance between face features is measured by normalized L2 distance. For R-101-S model, we decide the parameter κ based on the false positive rate (FPR) for the identity verification task. Three different FPRs have been used: 10 -3 (with κ = 1.24), 3 × 10 -4 (with κ = 1.05), and 10 -4 (with κ = 0.60). The distance metrics and selected thresholds are commonly used when evaluating the performance of face recognition model Klare et al. (2015) ; Kemelmacher-Shlizerman et al. (2016) . Please check the Appendix (see Table B ) for more details.

To distinguish between the FPR we used in generating adversarial examples and the other FPR used in evaluation, we introduce two notations "Generating FPR (G-FPR)" and "Test FPR (T-FPR)". For the experiment with query-free black-box API attacks, we use the online face verification services provided by Face++ (fac) and AliYun (ali).

Face landmark detection. We select Face Alignment Network (FAN) (Bulat & Tzimiropoulos, 2017b) trained on 300W-LP (Zhu et al., 2016b ) and fine-tuned on 300-W (Sagonas et al., 2013) for 2D landmark detection. The network is constructed by stacking Hour-Glass network (Newell et al., 2016) with hierarchical block (Bulat & Tzimiropoulos, 2017a) . Given a portrait image as input, FAN outputs 2D heatmaps which can be subsequently leveraged to yield 68 2D landmarks.

Semantic attacks on face images. In our experiments, we randomly sample 1, 280 distinct identities form CelebA (Liu et al., 2015) . To reduce the reconstruction error brought by the generator (e.g., x = G(x, c)) in practice, we take one more step to obtain the updated feature map f = G enc (x , c), where x = argmin x G(x , c) -x in feature-map interpolation. In our experiments, we use the last conv layer before upsampling in the generator as our as feature-map f given by the attack effectiveness. We also fix the parameter λ (e.g., balances the adversarial loss and smoothness constraint in Eq. 3) to be 0.01 for both face verification and landmark detection.

We used the StarGAN (Choi et al., 2018) for attribute-conditional image editing. In particular, we re-trained model on CelebA dataset (Liu et al., 2015) by aligning the face landmarks and then resizing images to resolution 112 × 112. In addition, we select 17 identity-preserving attributes as our input condition, as such attributes related to facial expression and hair color.

For each distinct identity pair (x, x tgt ), we perform semanticAdv guided by each of the 17 attributes (e.g., we intentionally add or remove one specific attribute while keeping the rest unchanged). In total, for each image x, we generate 17 adversarial images with different augmented attributes. In the experiments, we select a pixel-wise adversarial attack method (Carlini & Wagner, 2017) (referred as CW) as our baseline for comparison. Compared to our proposed method, CW does not require visual attributes as part of the system, as it only generates one adversarial example for each instance. We refer the corresponding attack success rate as the instance-wise success rate in which the attack success rate is calculated for each instance. For each instance with 17 adversarial images using different augmented attributes, if one of the 17 resulting images can attack successfully, we count the attack of this instance as one success, vice verse.

Semantic attacks on street-view images. We select DRN-D-22 (Yu et al., 2017) as our semantic segmentation model and fine-tune the model on image regions with resolution 256 × 256. To synthesize semantic adversarial perturbations, we consider semantic label maps as the input attribute and leverage a generative image manipulation model (Hong et al., 2018) pre-trained on CityScape (Cordts et al., 2016) dataset. Given an input semantic label map at resolution 256 × 256, we select a target object instance (e.g., a pedestrian) to attack. Then, we create a manipulated semantic label map by inserting another object instance (e.g., a car) in the vicinity of the target object. Similar to the experiments in the face domain, for both semantic label maps, we used the image manipulation encoder to extract features (with 1, 024 channels at spatial resolution 16 × 16) and conducted featurespace interpolation. We synthesized the final image by feeding the interpolated features to the image manipulation decoder. By searching the interpolation coefficient that maximizes the attack rate, we are able to fool the segmentation model with the synthesized final image. 100% success rate on all those models (see Figure 3 ). We conjecture that this is because the high dimensional feature space can provide more manipulation freedom. Overall analysis. Figure 2 shows the generated adversarial images and corresponding perturbations against R-101-S of SemanticAdv and CW respectively. The text below each figure is the name of augmented attribute, the sign before the name represents "adding" (in red) or "removing" (in blue) the corresponding attribute from the original image. We see that SemanticAdv is able to generate perceptually reasonable examples guided by the corresponding attribute. In particular, SemanticAdv is able to generate perturbations on the corresponding regions correlated with the augmented attribute, while the perturbations of CW have no specific pattern and are evenly distributed across the image.

Semantic Adversarial Examples:

Analysis: controlling single attribute. One of the key advantages of SemanticAdv is that we can generate adversarial perturbations in a more controllable fashion guided by the semantic attributes. This allows analyzing the robustness of a recognition system against different types of semantic attacks. We group the adversarial examples by augmented attributes in various settings. In Figure 3 , we present the attack success rate against two face verification models, namely, R-101-S and R-101-C, guided by different attributes. We highlight the bar with light blue for G-FPR equals to 10 -3 and blue for G-FPR equals to 10 -4 , respectively. As we see in this figure, with a larger T-FPR 10 -3 , our SemanticAdv can achieve almost 100% attack success rate across different attributes. With a smaller T-FPR 10 -4 , we find that SemanticAdv guided by some attributes such as Mouth Slightly Open and Arched Eyebrows achieve less than 50% attack success rate, while the other attributes such as Pale Skin and Eyeglasses are relatively less affected. In summary, we found that SemanticAdv guided by attributes describing the local shape (e.g., mouth, earrings) achieve a relatively lower attack success rate compared to attributes relevant to the color (e.g., hair color) or entire face region (e.g., skin). This suggests that the face verification models used in our experiments are more robustly trained in terms of detecting local shapes compared to colors. Please note that in practice we have the flexibility to select attributes for attacking an image based on the perceptual quality and attack success rate. 1a illustrates the transferability of SemanticAdv among different models by using the same FPRs (G-FPR = T-FPR = 10 -3 ). Table 1b illustrates the result with different FPRs (G-FPR = 10 -4 and T-FPR = 10 -3 ) for generation and evaluation. As shown in Table 1a , adversarial examples generated against models trained with softmax loss exhibit certain transferability compared to models trained with cosine loss. We conduct the same experiment by generating adversarial examples with CW and found it has weaker transferability compared to our SemanticAdv (results in brackets of Table 1 ).

As Table 1b illustrates, the adversarial examples generated against the model with smaller G-FPR = 10 -4 exhibit strong attack success rate when evaluating on the model with larger T-FPR = 10 -3 . Especially, we found the adversarial examples generated against R-101-S have the best attack performance on other models. These findings motivate the analysis of query-free black-box API attack detailed in the following paragraph.

Under review as a conference paper at ICLR 2020 User study. To measure the perceptual quality of the adversarial images generated by SemanticAdv, we conduct a user study on Amazon Mechanical Turk (AMT). We use the adversarial examples generated with G-FPR < 10 -4 , which is the most strict setting in our experiment, to conduct the user study for both CW and SemanticAdv. In total, we collect 2, 620 annotations from 77 participants. In 39.14±1.96% (close to random guess 50%) of trials the adversarial images generated by SemanticAdv are selected as reasonably-looking images and in 30.27 ± 1.96% of trials, the adversarial images generated by CW are selected as reasonably-looking. et al., 2016) , Blurring (Li & Li, 2017) , and Feature Squeezing (Xu et al., 2017) .

M test / M opt R-50-S R-101-S R-50-C R-101-C R-50

SemanticAdv against defense methods. We evaluate the strength of the proposed attack by testing against four existing defense methods, namely, Feature squeezing (Xu et al., 2017) , Blurring (Li & Li, 2017 ), JPEG (Dziugaite et al., 2016) and AMI (Tao et al., 2018) . For AMI (Tao et al., 2018) , we first extract attribute witnesses with our aligned face images and then leverage them Figure 5 illustrates SemanticAdv is more robust against these defense methods comparing with CW. The same G-FPR and T-FPR are used for evaluation. Under the condition that T-FPR is 10 -3 , both SemanticAdv and CW achieve high attack success rate, while SemanticAdv marginally outperforms CW when FPR goes down to 10 -4 . While defense methods have proven to be effective against CW attacks on classifiers trained with ImageNet (Krizhevsky et al., 2012) , our results indicate that these methods are still vulnerable in face verification system with small T-FPR.

4.3. SemanticAdv ON FACE LANDMARK DETECTION:

We also evaluate the effectiveness of SemanticAdv on face landmark detection. We select two attack tasks, namely, "Rotating Eyes" and "Out of Region". For the "Rotating Eyes" task, we rotate the coordinates of the eyes in the image counter-clockwise by 90 • . For the "Out of Region" task, we set a target bounding box and attempt to push all points out of the box. We summarize the experimental setup and quantitative results in the Appendix (see Table D ). As we see in Figure 6 , our method is applicable to attack landmark detection models.

4.4. SemanticAdv ON STREET-VIEW SEMANTIC SEGMENTATION:

We further demonstrate the applicability of our SemanticAdv beyond face domain by generating adversarial perturbations on street-view images. Figure 7 illustrates the adversarial examples on semantic segmentation. In the first example, we select the leftmost the pedestrian as the target object instance and insert another car into the scene to attack it. The segmentation model has been successfully attacked to neglect the pedestrian (see last column), while it does exist in the scene (see second-to-last column). In the second example, we insert an adversarial car in the scene by SemanticAdv and the cyclist has been recognized as a pedestrian by the segmentation model.

5. CONCLUSIONS:

Overall we presented a novel attack method SemanticAdv, which is capable of generating unrestricted adversarial perturbations guided by semantic attributes edition. Compared to existing methods, SemanticAdv works in a more controllable fashion. Experimental evaluations on face verification and landmark detection demonstrate several unique properties including attack transferability. We believe this work would open up new research opportunities and challenges in the field of adversarial learning. For instance, how to leverage semantic information to defend against such attacks will lead to potential new discussion. Defense methods. Feature squeezing (Xu et al., 2017 ) is a simple but effective method by reducing color bit depth to remove the adversarial effects. We compress the image represented by 8 bits for each channel to 4 bits for each channel to evaluate the effectiveness. For Blurring (Li & Li, 2017) , we use a 3 × 3 Gaussian kernel with standard deviation 1 to smooth the adversarial perturbations. For JPEG Dziugaite et al. ( 2016), it leverages the compression and decompression to remove the adversarial perturbation. We set the compression ratio as 0.75 in our experiment.

B FACE LANDMARK DETECTION:

Implementation details. To perform attack on the face landmark detection model, we run optimization for up to 2, 000 steps with a fixed learning rate 0.05. We set the balancing factor λ (see Eq.

3) to value 0.01 for this experiment.

Evaluation Metrics. We apply two different metrics for two adversarial attack tasks respectively.

For "Rotating Eyes" task, we use a well-adopted metric Normalized Mean Error (NME) (Bulat & Tzimiropoulos, 2017b) for experimental evaluation.

EQUATION

where p k denotes the k-th ground-truth landmark, pk denotes the k-th predicted landmark and √ W B * H B is the square-root area of ground-truth bounding box, where W B and H B represents the width and height of the box.

For "Out of Region" task, we consider the attack is successful if the landmark predictions fall outside a pre-defined centering region on the portrait image. Thus, we introduce a metric that reflects the portion of landmarks outside of the pre-defined centering region: r OUT = Nout Ntotal , where N out denotes the number of predicted landmarks outside the pre-defined bounding box and N total denotes the total number of landmarks. Table D : Quantitative results on face landmark detection (%) The two row shows the measured ratios (lower is better) for "Rotating Eyes" and "Out Of Region" task, respectively.

We present the quantitative results of SemanticAdv on face landmark detection model in Table D including two adversarial tasks, namely, "Out of Region" and "Rotating Eyes". We observe that our method is efficient to perform attacking on landmark detection models. For certain attributes such as "Eyeglasses", "Plae Skin", SemanticAdv is able to achieve reasonably-good performance.

C USER STUDY:

We conducted a user study on the adversarial images of SemanticAdv and CW used in the experiment of API-attack and the original images. The adversarial images are generated with G-FPR< 10 -4 for both methods. We present a pair of original image and adversarial image to participants and ask them to rank the two options. The order of these two images is randomized and the images are displayed for 2 seconds in the screen during each trial. After the images disappear, the participants have unlimited time to select the more reasonably-looking image according to their perception. For each participant, one could only conduct at most 50 trials, and each adversarial image was shown to 5 different participants. Some qualitative results are shown in Figure H . In total, we collect 2, 620 annotations from 77 participants. In 39.14 ± 1.96% of trials the adversarial images generated by SemanticAdv are selected as reasonably-looking images and in 30.27±1.96% of trails, the adversarial images generated by CW are selected as reasonably-looking images. It indicates that our semantic adversarial examples are more perceptual reasonably-looking than CW. Additionally, we also conduct the user study with larger G-FPR= 10 -3 . In 45.42±1.96% of trials, the adversarial images generated by SemanticAdv are selected as reasonably-looking images, which is very close to the random guess (50%). 

D ABLATION STUDY: FEATURE-SPACE INTERPOLATION:

We conduct an ablation study on feature-space interpolation by analyzing attack success rates with different feature-maps in the StarGAN network. Table E shows the attack success rate on R-101-S. Here, we use f i to represent the feature-map after i-th up-sampling operation. f 0 denotes the featuremap before applying up-sampling operation. The result demonstrates that samples generated by interpolating on f 0 achieve the highest success rate. Since f 0 is the feature-map before decoder, it still well embeds semantic information in the feature space. We adopt f 0 for interpolation in our experiments.

We also conduct a qualitative comparison between attribute-space and feature-space interpolation.

As shown in Figure I , images generated by attribute-space and feature-space interpolation are both reasonably-looking.

T-FPR (G-FPR) 10 -3 (10 -3 ) 3 × 10 -4 (3 × 10 -4 ) 10 -4 (10 -4 ) Layer (f ) 99.29 98.32 75.62 97.35 94.10 57.15 76.64 67.40 19.63 Table E: Attack success rate by selecting different layer's feature-map for interpolation on R-101-S(%). f i indicates the feature-map after i-th up-sampling operation.

f 0 f 1 f 2 f 0 f 1 f 2 f 0 f 1 f 2 Attack Success Rate

T-FPR (G-FPR):

10 -3 (10 -3 ) 3 × 10 -4 (3 × 10 -4 ) 10 -4 (10 -4 ) Layer (f ) 

f -2 f -1 f -2 f -1 f -2 f -

E SEMANTIC ATTACK TRANSFERABILITY:

In Table G , we present the quantitative results of the transferability with G-FPR = 10 -4 , T-FPR = 10 -4 . We observe that with the strict criterion (Lower T-FPR) of the verification model, the transferability becomes lower cross different models.

To explore whether the improvement of transferability in SemanticAdv is introduced by semantic editing rather than optimized semantic perturbation in feature space. We add a StrawMan [CW +

x new ] baseline which uses a controllable semantic-attribute-based generator to generate semantically different images without any notion of an adversarial attack, and then applies standard L p CW attacks on that generated image. The results are shown in Table H . The performance of the StrawMan [CW +

x new ] is worse than SemanticAdv. This result justifies that our SemanticAdv is able to produce novel adversarial examples which cannot be simply achieved by combining attribute-conditional image editing model with L p bounded perturbation. 

:



Figures and Tables:

Figure 1: Left: Overview of the proposed SemanticAdv. Right: Illustration of our SemanticAdv in the real world face verification platform. Note that the confidence denotes the likelihood that two faces belong to the same person.

PROBLEM DEFINITIONLet M be a machine learning model trained on a dataset D = {(x, y)} consisting of image-label pairs, where x ∈ R H×W ×D I and y ∈ R D L denote the image and the ground-truth label, respectively.Here, H, W , D I , and D L denote the image height, image width, number of image channels, and label dimensions, respectively. For each image x, our model M makes a prediction ŷ = M(x) ∈ R D L . Given a target image-label pair (x tgt , y tgt ) and y = y tgt , a traditional attacker aims to synthesize adversarial examples {x adv } by adding pixel-wise perturbations to or spatially transforming the original image x such that M(x adv ) = y tgt .

SemanticAdv ON FACE IDENTITY VERIFICATION Attribute-space vs. feature-space interpolation. First, we found that both attribute-space and feature-space interpolation could generate reasonable samples (see Figure I in Appendix). Compared to attribute-space interpolation, generating adversarial examples with feature-space interpolation produced much better quantitative results (see Table E in Appendix). We measured the attack success rate of attribute-space interpolation (with G-FPR = T-FPR = 10 -3 ): 0.08% on R-101-S, 0.31% on R-101-C, and 0.16% on both R-50-S and R-50-C. While feature-space interpolation achieves almost

Figure 2: Qualitative comparisons between our proposed SemanticAdv and pixel-wise adversarial examples generated by CW (Carlini & Wagner, 2017). Along with the adversarial examples, we also provide the corresponding perturbations (residual) on the right. Perturbations generated by our SemanticAdv (G-FPR = 10 -3 ) are unrestricted with semantically meaningful patterns. More results are shown in Appendix (see Figure N).

Figure 4 shows the adversarial examples with augmented semantic attributes against R-101-S model. The attribute names are shown in the bottom. The upper images are G(x, c new ) generated by StarGAN with augmented attribute c new where the lower images are the corresponding adversarial images with the same augmented attribute.

Figure 3: Quantitative analysis on the attack success rate with different single-attribute attacks. In each figure, we show the results correspond to a larger FPR (G-FPR = T-FPR = 10 -3 ) in skyblue and the results correspond to a smaller FPR (G-FPR = T-FPR = 10 -4 ) in blue, respectively. Source Image Target Image

Figure 4: Qualitative analysis on single-attribute adversarial attack (G-FPR = 10 -3 ). More results are shown in Appendix (see Figure K, Figure L and Figure M).

Quantitative analysis on query-free black-box attack. We use ResNet-101 optimized with softmax loss for evaluation and report the attack success rate(%) on two online face verification platforms. Note that for PGD-based attacks, we adopt MI-FGSM ( = 8) inDong et al. (2018) and M-DI 2 -FGSM ( = 8) inXie et al. (2019), respectively.

Figure 5: Quantitative analysis on attacking several defense methods including JPEG (Dziugaiteet al., 2016), Blurring(Li & Li, 2017), and Feature Squeezing(Xu et al., 2017).

Figure 6: Qualitative results on attacking face landmark detection model.

removal attack Task: cyclist → pedestrian attack Adversarial Image by Optimization (ɑ=argmin ɑ L(.))

Figure 7: Qualitative results on attacking street-view semantic segmentation model.

Figure H: Qualitative comparisons among ground truth, pixel-wise adversarial examples generated by CW, and our proposed SemanticAdv. Here, we present the results from G-FPR < 10 -4 so that perturbations are visible.

Attack success rate by selecting different layer's feature-map for interpolation on R-101-S(%). f -2 indicates the feature-map after the last down-sampling operation and f -2 indicates the feature-map after f -2 . interpolation (with α)Synthesized image by interpolation (with α)

Figure I: Qualitative comparisons between attribute-space and feature-space interpolation.

Figure J: Illustration of our SemanticAdv in the real world face verification platform (editing on pale skin). Note that the confidence denotes the likelihood that two faces belong to the same person.

We further analyze the robustness of the recognition system by generating adversarial examples guided by different visual attributes. Concurrent to our work,

Table content: <table><tr><td>). To further explore diverse</td></tr><tr><td>adversarial attacks and potentially help inspire defense mechanisms, it is important to generate the so-</td></tr><tr><td>called "unrestricted" adversarial examples which contain unrestricted magnitude of perturbation while</td></tr><tr><td>still preserve perceptual realism Brown et al. (2018). Recently, Xiao et al. (2018c); Engstrom et al.</td></tr><tr><td>(2017) propose to spatially transform the image patches instead of adding pixel-wise perturbation,</td></tr><tr><td>while such spatial transformation does not consider semantic information. Our proposed semanticAdv</td></tr><tr><td>focuses on generating unrestricted perturbation with semantically meaningful patterns guided by</td></tr><tr><td>visual attributes.</td></tr></table>

Transferability of SemanticAdv: cell (i, j) shows attack success rate of adversarial examples generated against j-th model and evaluate on i-th model. Results of CW are listed in brackets. Left: Results generated with G-FPR = 10 -3 and T-FPR = 10 -3 ; Right: Results generated with G-FPR = 10 -4 and T-FPR = 10 -3 .Query-free black-box API attack. In this experiment, we generate adversarial examples against R-101-S with G-FPR = 10 -3 (κ = 1.24), G-FPR = 10 -4 (κ = 0.60), and G-FPR < 10 -4 (κ = 0.20), respectively. We evaluate our algorithm on two industry level APIs, namely, Face++ and AliYun face verification platform. Since attack transferability has never been explored in concurrent work that generates semantic adversarial examples, we use L p bounded pixel-wise methods(Carlini & Wagner, 2017;Dong et al., 2018;Xie et al., 2019) as our baselines. As we see Table2, which shows the best performance of each method, our SemanticAdv achieves much higher attack success rate than CW in both APIs with all FPR thresholds (e.g., our adversarial examples generated with G-FPR < 10 -4 achieves 64.63% attack success rate on Face++ platform with T-FPR = 10 -3 ). In addition, we found that lower G-FPR can achieve higher attack success rate on APIs within the same T-FPR (see TableIin Appendix).

Table content: <table><tr><td/><td/><td/><td/><td>M test / M opt</td><td>R-50-S</td><td>R-101-S</td></tr><tr><td>-S</td><td colspan="3">1.000 (1.000) 0.108 (0.007) 0.023 (0.002) 0.018 (0.002)</td><td>R-50-S</td><td>1.000 (1.000) 0.862 (0.530)</td></tr><tr><td>R-101-S</td><td colspan="3">0.169 (0.006) 1.000 (1.000) 0.030 (0.002) 0.032 (0.003)</td><td>R-101-S</td><td>0.874 (0.422) 1.000 (1.000)</td></tr><tr><td>R-50-C</td><td colspan="3">0.166 (0.019) 0.202 (0.025) 1.000 (1.000) 0.048 (0.007)</td><td>R-50-C</td><td>0.693 (0.347) 0.837 (0.579)</td></tr><tr><td>R-101-C</td><td colspan="3">0.120 (0.015) 0.236 (0.029) 0.040 (0.006) 1.000 (1.000)</td><td>R-101-C</td><td>0.617 (0.218) 0.888 (0.617)</td></tr><tr><td/><td>(a)</td><td/><td/><td/><td>(b)</td></tr><tr><td>API name</td><td/><td>Face++</td><td/><td/><td>AliYun</td></tr><tr><td colspan="2">Attacker / Evaluation Metric</td><td/><td/><td/></tr><tr><td colspan="2">Dong et al. (2018)</td><td>30.77</td><td>21.03</td><td colspan="2">18.00</td><td>6.50</td></tr><tr><td colspan="2">Xie et al. (2019)</td><td>37.95</td><td>25.64</td><td colspan="2">21.50</td><td>11.00</td></tr><tr><td colspan="2">CW (G-FPR &lt; 10 -4 )</td><td>41.62</td><td>24.37</td><td colspan="2">19.00</td><td>12.00</td></tr><tr><td colspan="2">SemanticAdv (G-FPR &lt; 10 -4 )</td><td>64.63</td><td>42.69</td><td colspan="2">35.50</td><td>22.17</td></tr></table>

It indicates that SemanticAdv can generate more reasonable-looking adversarial examples compared with CW under the most strict setting with G-FPR < 10 -4 . Qualitative comparisons are shown in Appendix (see Figure H).

Table content: <table/>

Quantitative result of identity verification (%). It shows accuracy of face verification model and attack success rate of SemanticAdv and CW. x , G(x , c) and G(x , c new ) are the intermediate results of our method before adversarial perturbation.

Table content: <table><tr><td/><td>Verification Accuracy</td><td>98.36</td><td>98.78</td><td>98.63</td><td>98.84</td></tr><tr><td/><td>x</td><td>0.00</td><td>0.00</td><td>0.08</td><td>0.00</td></tr><tr><td/><td>G(x , c)</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.23</td></tr><tr><td/><td>G(x , c new )(Best)</td><td>0.16</td><td>0.08</td><td>0.16</td><td>0.31</td></tr><tr><td/><td>SemanticAdv (Best)</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr><tr><td/><td>SemanticAdv (Worst)</td><td>91.95</td><td>93.98</td><td>99.53</td><td>99.77</td></tr><tr><td/><td>SemanticAdv (Average)</td><td>98.98</td><td>99.29</td><td>99.97</td><td>99.99</td></tr><tr><td/><td>CW</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr><tr><td/><td>Verification Accuracy</td><td>97.73</td><td>97.97</td><td>97.91</td><td>97.85</td></tr><tr><td/><td>x</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>G(x , c)</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td>3 × 10 -4</td><td>G(x , c new )(Best) SemanticAdv (Best)</td><td>0.00 100.00</td><td>0.00 100.00</td><td>0.00 100.00</td><td>0.00 100.00</td></tr><tr><td/><td>SemanticAdv (Worst)</td><td>83.75</td><td>79.06</td><td>98.98</td><td>96.64</td></tr><tr><td/><td>SemanticAdv (Average)</td><td>97.72</td><td>97.35</td><td>99.92</td><td>99.72</td></tr><tr><td/><td>CW</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr><tr><td/><td>Verification Accuracy</td><td>93.25</td><td>92.80</td><td>93.43</td><td>92.98</td></tr><tr><td/><td>x</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>G(x , c)</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td>10 -4</td><td>G(x , c new )(Best) SemanticAdv (Best)</td><td>0.00 100.00</td><td>0.00 100.00</td><td>0.00 100.00</td><td>0.00 100.00</td></tr><tr><td/><td>SemanticAdv (Worst)</td><td>33.59</td><td>19.84</td><td>67.03</td><td>48.67</td></tr><tr><td/><td>SemanticAdv (Average)</td><td>83.53</td><td>76.64</td><td>95.57</td><td>91.13</td></tr><tr><td/><td>CW</td><td>100.00</td><td>100.00</td><td>100.00</td><td>100.00</td></tr></table>

Peer review (iclr2020):
The authors describe a method for adversarially modifying a given (test) example that 1) still retains the correct label on the example, but 2) causes a model to make an incorrect prediction on it. The novelty of their proposed method is that their adversarial modifications are along a provided semantic axis (e.g., changing the color of someone*s skin in a face recognition task) instead of the standard perturbations that the existing literature has focused on (e.g., making a very small change to each individual pixel). The adversarial examples that the authors construct, experimentally, are impressive and striking. I*d especially like to acknowledge the work that the authors put in to construct an anonymous link where they showcase results from their experiments. Thank you! Overall, I think that this is interesting work that can help to broaden the study of adversarial examples and make them more applicable even in non-adversarial settings (e.g., by making models more robust to the changes in semantic attributes that the authors consider). There has been quite a bit of interest in the community in adversarial examples that are not just perturbations, and I believe that the authors* approach will encourage a good deal of follow-up research. However, my main concern with the paper is that in my opinion, it does not sufficiently address why it is important to generate adversarial examples in the way they do. For example: 1) Is the argument that this is a more powerful attack surface, so adversaries should take note (and defenders should figure out how to defend against this)? If that is the case, what is the attack model under which these attacks are realistic? For example, the original attacks are motivated in the sense that the adversarial examples are visually imperceptible, so they might not be noticed by the end-user. What is the equivalent argument for these semantic attacks? 2) Is the argument that these semantic attacks somehow capture a more realistic part of the data distribution over all natural images, and therefore it is good to have models that perform well on these semantic adversarial examples even if we*re not concerned about an adversary (e.g., because the model might generalize better to other tasks or be more causally correct)? If that*s the case, then I think this needs to be explored more. For example, what about the following straw man baseline: use a controllable semantic-attribute-based generator to generate semantically different images without any notion of an adversarial attack, and then do standard attacks on that generated image? How would that be better or worse than the proposed method? 3) Or is the argument that it is just good to be able to generate examples that models get wrong? If so, why, and why is this method better than other methods? I think the paper would be significantly stronger if the importance and implications of their work were explicated along the above lines. For this reason, my current assessment is a weak reject, though I*d be open to changing this assessment. === Less critical comments, no need to respond or fix right away === While the overall concept and approach was clear, I generally found the notation and mathematical exposition difficult to follow. Please be more precise. Here is a non-exhaustive list of examples from section 3: a) I*m not sure what*s the difference between and , or between and . These seem to be used somewhat interchangeably? b) Equation 3 is the central optimization problem in the paper, and should be written out explicitly using as the optimization variable, instead of referring to equations 1 and 2 (in which doesn*t even appear). c) I didn*t understand equation 4. What does assuming mean? What happens when that is not true? d) Equation 5: Why is in the right hand side by not in the left? e) Equation 6: is missing an argument.