Fine-grained Synthesis of Unrestricted Adversarial Examples
Authors: Omid Poursaeed
Tianxing Jiang
Yordanos Abraham Goshu
Harry Yang
Serge Belongie
Ser-Nam Lim
Venue:  (2021)
Keywords: adversarial examples
unrestricted attacks
generative models
adversarial training
generative adversarial networks

Abstract: We propose a novel approach for generating unrestricted adversarial examples by manipulating fine-grained aspects of image generation. Unlike existing unrestricted attacks that typically hand-craft geometric transformations, we learn stylistic and stochastic modifications leveraging state-of-the-art generative models. This allows us to manipulate an image in a controlled, fine-grained manner without being bounded by a norm threshold. Our approach can be used for targeted and non-targeted unrestricted attacks on classification, semantic segmentation and object detection models. Our attacks can bypass certified defenses, yet our adversarial images look indistinguishable from natural images as verified by human evaluation. Moreover, we demonstrate that adversarial training with our examples improves performance of the model on clean images without requiring any modifications to the architecture. We perform experiments on LSUN, CelebA-HQ and COCO-Stuff as high resolution datasets to validate efficacy of our proposed approach. 

Under review as a conference paper at ICLR 2021
FINE-GRAINED SYNTHESIS OF UNRESTRICTED ADVERSARIAL EXAMPLES
Anonymous authors Paper under double-blind review
ABSTRACT
We propose novel approach for generating unrestricted adversarial examples by manipulat- ing fine-grained aspects of image generation. Unlike existing unrestricted attacks that typically hand-craft geometric transformations, we learn stylistic and stochastic modifications leverag- ing state-of-the-as generative models. This allows us to manipulate an image in a controlled, fine-graime gra-ggained manner without being bounded by norm threshold. Our approach can be used for targeted and non-targeted unrestricted attacks on classification, semantic segmentation and ob- ject detection models. Our attacks can bypass certified defenses, yet our adversarial images look indistinguishable from natural images as verified by human evaluation. Moreover, we demon- strate that adversarial training with our examples improves performance of the model on clean images without requiring any modifications to the architecture. We perform experiments LSUN, CelebA-HQ and COCO-Sture as high resolution datasets to validate efficacy of our pro- posed approach.
INTRODUCTION
Adversarial examples, inputs resembling real samples but maliciously crafted to mislead machine learning mod- els, have been studied extensively the last few years. Most of the existing papers, however, focus on norm- constrained attacks and defenses, in which the adversarial input lies in e-neighborhood of real sample using the Lp distance metric (commonly with p = 0, 2, oo). For small e, the adversarial input is quasi-indissinguisaalle from the natural sample. For adversarial image fool the human visual system, it is sufficient to be norm- constrained; but this condition not necessary. Moreover, defenses tailored for norm-constraing attacks can fail on other subtle input modifications. This has led to recent surge of interest in which the adversary is not bounded by a norm threshold. These methods typically hand-craft transformations to capture visual similarity. Spatial transformations (Engstrom et al. (2017); Xiao et al. (2018); Alaifari al.) ]. viewpoint or pose changes [Alcorn et al. (2018)], inserting small patches [Brown et al. (2017)), among other methods, have been proposed for unrestricted adversarial attacks.
unrestricted adversarial attacks
In this paper, we focus on fine-grained manipulation of images for unrestricted adversarial attacks. We build upon thate-of-the-art generative models which disentangle factors of variation in images. We create fine and coarse- grained adversarial changes by manipulating various latent variables at different resolutions. Loss of the target network is used to guide the generation process. The pre-trained generative model constrains the search space for our adversarial examples to realistic images, thereby revealing the target model's 's vulnerability in the natural image space. We verify that we do not deviate from the space realistic images with user study as well as a t-SNE plot comparing distributions of real and adversarial images (see Fig, 7 in the appendix). As a result, we observe that including these examples in training the model enhances its accuracy on clean images.
Our contributions can be summarized as follows:
We present the first method for fine- treeggraiedd generation of high-resolution unrestricted adversarial exam- ples which the attacker controls which aspects of the image to manipulate, resulting diverse set of realistic, on-the-manifold adversarial examples.
We demonstrate that adversarial training with our examples improves performance of the model on clean images. This is in contrast to training with norm-bounded perturbations which degrades the model's accuracy. Unlike recent approaches such as Xie et al. (2020) which use a separate auxiliary batch norm for adversarial examples, our method does not require any modifications to the architecture.
We propose the first method for generating unrestricted adversarial examples for semantic segmentation and object detection. Training with our examples improves segmentation results on clean images.

---

Under review as a conference paper at ICLR 2021
We demonstrate that our proposed attack can break certified defenses on norm-bounded perturbations.
2 RELATED WORK
2.1
NORM-CONSTRAINED ADVERSARIAL EXAMPLES
Most of the existing works on adversarial attacks and defenses focus on norm-constrained adversarial examples: for a given classifier F: RT [l....,K' K} and an image x E IR", the adversarial image x' E IR is created such that || z'I1p < c and F(x) & F(z'). Common values for are 0, 2, oo, and e is chosen small enough so that the perturbation is imperceptible. Various algorithms have been proposed for creating x' from z. Optimization-base methods solve a surrogate optimization problem based on the classifier's loss and the perturbation norm. In their pioneering paper on adversarial examples. SSzeedy et al. (2013) use box-constrained L-BFGS [Fletcher] (2013)] to minimize the surrogate loss function. Carlini & Wagner (2017) propose stronger optimization- for Lu. Ly and Ls norms using better objective functions and the Adam optimizer. Gradient- based methods use gradient of the classifier's ' loss with respect to the input image. Fast Gradient Sign Method (FGSM) [Goodfellow al. [014g] uses a first-order approximation of the function for faster generation and is optimized for the T. norm. Projected Gradient Descent (PGD) (Madry et al. (2017)) is an iterative variant of FGSM which provides strong first-order attack by using multiple steps of gradient ascent and projecting perturbed images an e-ball centered at the input. Other variants of FGSM are proposed by Dong et al. (2018) and Kurakin et al. (2016).
attacks
Several methods have been proposed for defending against adversarial attacks. These approaches can be broadly categorized to empirical defenses which are empirically robust to adversarial examples, and certified defenses which are provably robust to a certain class of attacks. One of the most successful empirical defenses adversarial training [Goodiellow et al. (2014); Kurakin et al. (2016); Madry et al. (2017)) which augments training data with adversarial examples generated as the training progresses. Many empirical defenses attempt to combat adversaries using a form of input pro-psocessiig or by manipulating intermediate features or gradients [Guo et al. (2017): Xiie jet al. (2017); Samangouei et al.] (). Few approaches have been able to scale up to high-resolution datasets such as ImageNet [Liao et al. (2018); Xie et al. 2018); Kannan et al. (0018). Athalye et al. (2018) show that many of these defenses fail due to obfuscated gradients, which occurs when the defense method is designed to mask information about the model's gradients. Vulnerabilities of empirical defenses have led increased interest in certified defenses, which provide a guarantee that the classifier's prediction is constant within a neighborhood of the input. Several certified defenses have been proposed (Wong & Kolter (2017); RRghunathann et al., (2018); Tsuzuku et al. [2018)] which typically do not scale to ImageNet. [Cohen et al. (2019) use randomized smoothing with Gaussian noise to obtain provably L2-robust classifiers on ImageNet. Lecuyer et al. (2019) propose an alternative certified defense at ImageNet scale leveraging a connection between robustness against adversarial
examples and differential privacy theory.
2.2 UNRESTRICTED ADVERSARIAL EXAMPLES
For an image to be adversarial, it needs to be visually indistinguishable from real images. One way to achieve this is by applying subtle geometric transformations to the input image. Spatially transformed adversarial examples are introduced by [iiao et al. (2018) in which a flow field is learned to displace pixels of the image. Similarly, Alaifari let al. (2018) iteratively apply small deformations the input in order obtain the adversarial image. [Engstrom et al. 2017) show that simple translations and rotations are enough for fooling deep neural networks. Alcorn et al. (2018) manipulate pose of an object to fool deep neural networks. They estimate parameters of 3D renderer that cause the target model to misbehave in response to the rendered image. Another approach for evading the norm constraint is to insert new objects in the image. Adversarial Patch [Brown al.) (2017) creates an adversarial image by completely replacing part of an image with a synthetic patch, which is image-agnostic and robust to transformationn Existence of on-the-manifold adversarial examples is also shown by Gilmer et al.] (2018), that
consider the task of classifying between two concentric n-dimensional spheres. [Stutz et al. (2019) demonstrate that both robust and accurate models are possible by using on-the-manifold adversarial examples.
challenge
for creating unrestricted adversarial examples and defending against them is introduced by [rown et al. 2018) using the simple task of classifying between birds and bicycles. The recent work by Gowal et al. (2020) show that adversarial training with examples generated by StyleGAN can improve performance of the model clean images. They consider the classification task on low-resolution datasets such as ColorMNIST and CelebA, and only use fine changes in their adversarial training. Our approach is effective on high-resolution datasets such as CelebA-HQ and LSUN, uses a range of low-level to high-level changes for adversarial training and encompasses several tasks including classification, segmentation and detection. In addition, we demonstrate that our adversarial examples can break certified defenses on norm-comee comm-constaained perturbations and are realistic as verified by human evaluation. Song
2

---

Under review as a conference paper at ICLR 2021
Lutent
Normllzee
A
Noise
Synthesis network g
PC
Mapping network
PC
Y Yodr
Inde
f
g(y. m))
g(Yabro Mudeo)
Forward path (clean)
Forward puth (adversarial)
Classifier
Gradient path (advensarial)
JK.
target T
Figure I: Classification architecture. Style (y) and noise (17) variables are used to generate images g(y; 17) which fed the classifier F. Adversarial style and noise tensors are initialized with y and v and iteratively updated using gradients of the loss function J.
al. (2018) search in the latent (z) space of AC-GAN (Odena et al. (017)) to find generated images that can fool target classifier but yield correct predictions on AC-GAN's s auxiliary classifier. They constrain the search region of that it is close randomly sampled noise vector, and show results MNIST, SVHN and CelebA datasets. Requiring two classifiers to have inconsistent predictions degrades sample quality of the model. As we show in the appendix, training with these adversarial examples hurts the model's performance clean images. Moreover, this approach has no control over the generation process since small changes in the space can lead to large changes in generated images and even create unrealistic samples. On the other hand, our method manipulates high-resolution real or synthesized images in generates samples which improve the model's accuracy on clean images both in classification and segmentation tasks. To further illustrate difference of our approach with Song et al. (2018), we plot t-SNE embeddings of real images from CelebA-HQ as well as adversarial examples from our method and Song et al.'s approach in the appendix and show that our adversarial images stay closer to the manifold of real images.
manner owing to the interpretable disentangled latent space. It also
APPROACH
Most of the existing works on unrestricted adversarial attacks rely on geometric transformations and deformations which are oblivious to latent factors of variation. In this paper, we leverage disentangled latent representations of images for unrestricted adversarial attacks. We build upon state-of-the-a generative models and consider various target tasks: classification, semantic segmentation and object detection.
3.1 CLASSIFICATION
Style-GAN Karras et al. 2018) is state-of-the-art generative model which disentangles high-level attributes and stochastic variations an unsupervised manner. Stylistic variations are represented by style variables and stochas- tic details are captured by noise variables. Changing the noise only affects low-level details, leaving the overall composition and high-level aspects intact. This allows us to manipulate the noise variables such that variations barely noticeable by the human eye. The style variables affect higher level aspects of image generation. For instance, when the model is trained on bedrooms, style variables from the top layers control viewpoint of the cam- era, middle layers select the particular furniture, and bottom layers deal with colors and details of materials. This allows us to manipulate images in a controlled manner, providing an avenue for fine-grained unrestricted attacks.
represent Style-GAN with mapping function f and a synthesis network g- The mapping function is an 8-layer MLP which takes a latent code z, and produces an intermediate latent vector w = fz)). This vector then specialized by learned affine transformations A style variables y, which control adaptive instance normalization operations after each convolutional layer of the synthesis network Noise inputs are single-channel images consisting of un-correlated Gaussian noise that are fed to each layer of the synthesis network. Learned per- feature scaling factors B are used to generate noise variables 77 which are added to the output of convolutional
Formally,

---

Under review as a conference paper at ICLR 2021
8
Bab
3x3 Damy
SPADE
ReL.U
SPADE
RaLU
7
Nor
SPADE
SPADE
BRADE
SPADEI RecSl
JPPDDE
Figure 2: Semantic segmentation architecture. Adversarial parameters "lado and Badv are initialized with y and 3, and iteratively updated to fool the segmentation model G.
layers. The synthesis network takes style y and noise input, and generates an image x = g(y. 1)). We pass the generated image to a pre-trained classifier F. We seek to slightly modify x so that F can no longer classify it correctly. We achieve this through perturbing the style and noise tensors. We initialize adversarial style and (0) noise variables as Yady = y and 7adv 77, and iteratively update them in order fool the classifier, Loss of the classifier determines the update rule, which in turn depends on the type of attack. As common in the literature, we
(0)
consider two types of attacks: non-targeted and targeted.
In order to generate non-targeted adversarial examples, we need to change the model's 's original prediction. Starting iteratively perform gradient ascent in the style and noise
from initial values (o) y and 1/0 ") spaces of the generator to find values that maximize the classifier's loss. Alternatively, as proposed by Kurakin as our target. We found this approach
ct al. (0116), we can use the least-likely predicted class Uxx arg more effective in practice. At time step the update rule for the style and noise variables is:
y adv = y adv sign (V
[t++)
(t)) [(F(g(y (t) adv Jady )). lxx))
(1)
Madv
(t+1) Madv 8- sign(V y
J(F adv: m)d))), (1x))
(2)
in which J(.,.) is the classifier's loss function, F(.) gives the probability distribution over classes, x = g(y,n), and 6, E IR are step sizes. We use (c. 6) (0.004, 0.2) and (0.004, 0.1) for LSSUN and CelebA-HQ respectively. We perform multiple steps of gradient descent (usually 2 10) until the classifier fooled.
Generating targeted adversarial examples more challenging as we need to change the prediction to specific class T. In this case, we perform gradient descent to minimize the classifier's loss with respect to the target:
(t-1) y adv e sign(V V ((F (t) (L)
adv: Fadv (), T))
(3)
adv
y adv
7adv nadv 8 sign( (V [(F(gly adv ().)))
(++1)
(t)
(4)
Tadv We use 5c (0.005, 0.2) and (0.004, 0.1) the experiments on LSUN and CelebA-HQ respectively, practice to 15 updates suffice to fool the classifier. Note that we only control deviation from the initial latent variables, and do not impose any norm constraint on generated images.
3.1.1 INPUT-CONDITIONED GENERATION
Generation also be conditioned real input images by embedding them into the latent space of Style-GAN. We first synthesize images similar to the given input image 7 by optimizing values of y and v such that g(y, n)) is close More specifically, we minimize the perceptual distance [Johnson al. (2016)) between g 1)) and I. We can then proceed similar to equations 1-4 to perturb these tensors and generate the adversarial image. Realism of synthesized images depends on inference properties of the generative model. In practice, generated images resemble input images with high fidelity especially for CelebA-HO images.

---

Under review as a conference paper at ICLR 2021
3.2 SEMANTIC SEGMENTATION AND OBJECT DETECTION
We also consider the task of semantic segmentation and leverage the generative model proposed by Park et al. (2019). The model is conditioned on input semantic layouts and uses OPatially-Adaptivy (om)aoaraiinntionn (SPADE) modules to better preserve semantic information against common normalization layers. The layout is first projected onto an embedding space and then convolved to produce the modulation parameters 7 and B3. We adversarially modify these parameters with the goal of fooling segmentation model. We consider non-targeted attacks using per per-pixel predictions and compute gradient of the loss function with respect to the modulation param- Figure iillstraate the architecture. Note that manipulating
eters with an update rule similar to equations] I and - variables at smaller resolutions lead to coarser changes. We consider a similar architecture for the object detection task except that we pass the generated image to the detection model and try to increase its loss. Results for this task are shown in the appendix.
4 RESULTS AND DISCUSSION
We provide qualitative and quantitative results using experiments on LSUN (Yu al. (015 and CelebA-HQ Karras et al. (2017)) LSUN contains 10 scene categories and 20 object categories. We use all the scene classes as well as two object classes: cars and cats. We consider this dataset since it is used in Style-GAN, and is well suited for a classification task. For the scene categories, a 10-way classifier is trained based on Inception-v3 (Szegedy al.) [2016)] which achieves an accuracy of 87. 87.7% on LSUN's test set. The two object classes also appear in ImageNet (Deng et al. (2009)1, a richer dataset containing 1000 categories. Therefore, for experiments on cars and cats we use an Inception-v3 model trained on ImageNet. This allows us to explore a broader set of categories in our attacks, and is particularly helpful for targeted adversarial examples. CelebA-HQ consists of 30,000 face images at 1024 x 1024 resolution. We consider the gender classification task, and use the classifier provided by Karras et al. (2018). This is binary task for which targeted and non-targeted attacks are similar.
In order to synthesize a variety of adversarial examples, we use different random seeds in Style-GAN to obtain various values for z, w, y and 17. Style-based adversarial examples are generated by initializing Yadv with the value of y, and iteratively updating it as in equation [ (or 3) until the resulting image g(yadv. 17) fools the classifier F, Noise-based adversarial examples are created similarly using Madv and the update rule in equation 2 (or 4.. While using different step sizes makes a fair comparison difficult, we generally found casier to fool the model by manipulating the noise variables. We also combine the effect of style and noise by simultaneously updating Yadv and Madv in each iteration, and feeding g(yadv, Madv) to the classifier. In this case, the effect of style usually dominates since it creates coarser changes.
Figure 3 illustrates generated adversarial examples on LSUN. Original image g(y, '7), noise-based image gly, Madv) ) and style-based image g(nav, 77) are shown. Adversarial images look almost indistinguishable from natural images. Manipulating the noise variable results in subtle, imperceptible changes. Varying the style leads to coarser changes such as different colorization, pose changes, and even removing or inserting objects in the scene. We can also control granularity of changes by selecting specific layers of the model. Manipulating top layers, corresponding to coarse spatial resolutions, results in high-level changes. Lower layers, on the other hand, modify finer details. In the first two columns of Figure 33 we only modify top 6 layers (out of 18) to generate adversarial images. The middle two columns change layers to 12, and the last column uses the bottom 6 layers.
Figure 4 depicts adversarial examples CelebA-HQ gender classification. Males are classified as females and vice versa. As we observe, various facial features are altered by the model yet the identity is preserved. Similar to LSUN images, noise-based changes are more subtle than style-based ones, and observe spectrum of high- level, mid-level and low-level changes. Figure 55 illustrates adversarial examples conditioned on real input images using the procedure described in Section 3.1.1 Synthesized images resemble inputs with high fidelity, and set the initial values in our optimization process. In some cases, we can notice how the model is altering masculine or feminine features. For instance, women's 's faces become more masculine in columns 2 and 4, and men's 's beard is removed in column 3 of Figure f and column of Figure 5
We also show results on semantic segmentation in Figure Glin which we consider non-targeted attacks on DeepLab- v2 Chen et al. (2017) with a generator trained on the COCO-stuff dataset [Caesar et al.) (2018)]. We iteratively modify modulation parameters at all layers, using a step size of 0.001, to maximize the segmentation loss with respect to the given label map. As we observe, subtle modifications to images lead to large drops in accuracy.
Unlike perturbation-based attacks, Lp distances between original and adversarial images are large, yet they are visually similar. Moreover, we do not observe high-freeuency perturbations in the generated images. The model
5

---

Under review as a conference paper at ICLR 2021
High-level changes
Mid-level
Low-level
Original
Adversarial (noise)
Binder
Minivan
Toikt Paper
Trolley
Living mom
Adversarial (style)
White Fox
Minivan
Boa
Projector
Living room
(a) Non-targeted
Original
Adversarial (noise)
Tower
Mountain
Tower
Adversarial (style)
Tower
Mountain Tent
Toy Poodle
Catamount
Tower
Targeted
Figure 3: Unrestricted adversarial examples on LSUN for a) non-targeted and b) targeted attacks. Predicted classes shown under each image.
learns to modify the initial input without leaving the manifold of realistic images. Additional examples and higher- resolution images are provided in the appendix.
4.1 ADVERSARIAL TRAINING
Adversarial training increases robustness of models by injecting adversarial examples into training data. Adver- sarial training with norm-bounded examples degrades performance of the classifier on clean images as they have different underlying distributions. We show that adversarial training with unrestricted examples improves the model's ' accuracy on clean images. To ensure that the model maximally benefits from these additional samples, we need avoid unrealistic examples which do not resemble natural images. Therefore, we only include samples that can fool the model in less than specific number of iterations. We use a threshold of 10 as the maximum number of iterations, and demonstrate results on classification and semantic segmentation. We use the first 10 generated examples for each starting image in the segmentation task. Table shows accuracy of the strengthened and orig-

---

Under review as a conference paper at ICLR 2021
Figure 4: Unrestricted adversarial examples on CelebA-HQ gender classification. From top bottom: original,
noise-based and style-based adversarial images. Males are classified as females and vice versa.
Figure 5: Input-conditioned adversarial examples on CelebA-HQ gender classification. From top to bottom: input, generated and style-based images. Males are classified as females and vice versa.
Iteration
Generated Image
Prediction
36.4% Figure 6: Unrestricted adversarial examples for semantic segmentation. Generated images, corresponding predic- tions and their accuracy (ratio of correctly predicted pixels) are shown for different number of iterations.
Accuracy 80.2%
62.2%
39.4%
37.5%
0.1%

---

Under review as a conference paper at ICLR 2021
inal classifiers on clean and adversarial test images. For the segmentation task we report the average accuracy of adversarial images at iteration 10. Similar to norm-constrained perturbations, adversarial training is an effective defense against our unrestricted attacks. Note that accuracy of the model on clean test. images improved after ad- versarial training. This is in contrast to training with mom--ouunee adversarial inputs which hurts the classifier's s performance on clean images, and is due to the fact that unlike perturbation-base based inputs, our generated images live the manifold of realistic images as constrained by the generative model.
Classification (LSUN)
Classification (CelebA-HIQ)
Segmentation
Clean
Adversarial
Clean
Adversarial
Clean
Adversarial
Adv. Trained
89.5%
78,4%
%
83.6%
69.3%
60.2%
Original 88.9%
0.0%
95.7%
0.0%
67.9%
2.7%
Table 1: Accuracy of adversarially trained and original models clean and adversarial test images.
4.2 USER STUDY
Norm-constrained attacks provide visual realism by Lp proximity to a real input. To verify that our unrestricted adversarial examples are realistic and correctly classified by oracle, we perform human evaluation using Amazon Mechanical Turk. In the first experiment, each adversarial image is assigned to three workers, and their majority vote is considered as the label. The user interface for each worker contains nine images, and shows possible labels to choose from. We use 2400 noise-based and 2400 style-based adversarial images from the LSUN dataset, containing 200 samples from each class (10 scene classes and object classes). The results indicate that 99 99.2% of workers' majority votes match the ground-truth labels. This number is 98.7% for style-based adversarial examples and 99.7% for noise-based ones. As we observe in Figure 3 noise-based examples do not deviate much from the original image, resulting in easier prediction by human observer. On the other hand, style-based images show coarser changes, which a few cases result in unrecognizable images or false predictions by the workers.
We use similar setup in the second experiment but for classifying real versus fake (generated). We also include 2400 real images as well as 2400 unperturbed images generated by Style-GAN. 74. 74.7% of unperturbed images are labeled by workers as real. This number is 74.3% for noise-based adversarial examples and 70.8% for style-based ones, indicating less than 4% drop compared with unperturbed images generated by Style-GAN.
4.3 EVALUATION ON CERTIFIED DEFENSES
Cohen et al. 2019) propose the first certified defense on norm-bounded perturbations at the scale of ImageNet. Using randomized smoothing with Gaussian noise, their defense guarantees a certain top-1 accuracy for pertur- bations with La norm less than a specific threshold. We demonstrate that our unrestricted attacks can break this certified defenses on ImageNet. We use 400 noise-based and 400 style-based adversarial images from the object categories of LSUN, and group all relevant ImageNet classes as the ground-truth Our adversarial examples are evaluated against a randomized smoothing classifier based on ResNet-50 using Gaussian noise with standard de- viation of 0.5. shows accuracy of the model clean and adversarial images. As observe, the accuracy drops on adversarial inputs, and the certified defense is not effective against our attack. Note that we stop updating adversarial images soon as the model is fooled. If we keep updating for more iterations afterwards, can achieve even stronger attacks.
Accuracy
Clean
63.1%
Adversarial (style)
21.7%
Adversarial (noise)
37.8%
Table 2: Accuracy of a certified classifier equipped with randomized smoothing on our adversarial images.
5 CONCLUSION AND FUTURE WORK
The area of unrestricted adversarial examples is relatively under-explored. Not being bounded by a norm threshold provides its own pros and cons. It allows us to create diverse set of attack mechanisms; however, fair comparison of relative strength of these attacks is challenging. It is also unclear how to even define provable defenses. While several papers have attempted interpret norm-constrained attacks in terms of decision boundaries, there has been less effort in understanding the underlying reasons for models' vulnerabilities to unrestricted attacks. We believe these can be promising directions for future research. We also plan further explore transferability of our approach for black-box box attacks in the future.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rima Alaifari, Giovanni S Alberti, and Tandri Gauksson. Adef: An iterative algorithm to construct adversarial deformations, arXiv preprint arXiv: 2018.
Michael A Alcom, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. arXiv preprint arXiv: 2018.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circum- venting defenses to adversarial examples. arXiv preprint arXiv:1882..00420, 2018.
Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint 1rXiv:1712.09665, 2017.
Tom B Brown, Nicholas Carlini, Chiyuan Zhang, Catherine Olsson, Paul Christiano, and lan Goodfellow. Unre- stricted adversarial examples. arXiv preprint arXiv: 2018.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1209-1218 1218, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Sympo- sium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Liang-Chied Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transac- tions on pattern analysis and machine intelligence, 40(4):834-84, 2017.
Jeremy M Cohen, Elan Rosenfeld, and J Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint arXiv: (9t.::::9900911990 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9185-9193, 2018.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. A rotation and translation suffice: Fooling cnns with simple transformations. arXiv preprint arXiv:1712.02779, 2017.
Roger Fletcher. Practical methods of optimization. John Wiley & Sons, 2013.
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow. Adversarial spheres. arXiv preprint S.Ki::1001.0774,, 2018.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1412.651112.657... 2014.
Sven Gowal, Chongli Qin, Po-Sen Huang, Taylan Cemgil, Krishnamurthy Dvijotham, Timothy Mann, and Push-
meet Kohli. Achieving robustness in the wild via adversarial mixing with disentangled representations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 121 1211-1220, 2020.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. arXiv preprint arXiv:1711.00111. 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv:1803.000 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 1710.10910 2017.

---

Under review as a conference paper at ICLR 2021
Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial net- works. arXiv preprint arXiv: 1812.04494 2018.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint wrXiv:160..0253: 2016.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP), PP. 656-672. IEEE, 2019.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang. Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1778-1787, 2018.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint X7v:1776606083,, 2017.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In Proceedings of the 34th International Conference on Machine Learning-Volume Volumee 70, pp. 2642-2651, JMLR. org, 2017.
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantie image synthesis with spatially-adaptiie normalization. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 2337- 2346, 2019.
Aditi Raghunathan, Jacob Steinhardt, and Percy S Liang. Semidefinite relaxations for certifying robustness to adversarial examples. In Advances in Neural Information Processing Systems, pp. 10877-10887, 2018.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint I 2018.
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In Advances in Neural Information Processing Systems, pp. 8312-8323, 2018.
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generalization. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6976-6987, 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6190 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the incep- tion architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1818-28228 2016.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable certification of per- turbation invariance for deep neural networks. In Advances in Neural Information Processing Systems, pp. 6541-6550, 2018.
Eric Wong and J Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. arXiv preprint arXiv:1711.0085// 2017.
Chaowei Xiao, Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In International Conference on Learning Representations, 2018.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization, arXiv preprint arXiv:1711.01991, 2017.
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. arXiv preprint arXiv:1812.03411, 2018.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 819 828, 2020.
10

---

Under review as a conference paper at ICLR 2021
Fisher Yu, Ari Seff, Yinda Zhang. Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 03365, 2015.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 COMPARISON WITH SONG ET AL.. 2018
We show that adversarial training with examples generated by Song et al. (2018) hurts the classifier's performance on clean images. Table 3 demonstrates the results. We use the same classifier architectures as (Snn et al. (0118) and consider their basic attack. We observe that the test accuracy on clean images drops by 1.3%, 1.4% and 1.1%
on MNIST, SVHN and CelebA respectively. As we show in Table training with examples improves the accuracy, demonstrating difference of our approach with that of Song et al. (2018).
MNIST
SVHN
CelebA
Clean
Adversarial
Clean
Adversarial 86.4%
Clean
Adversarial
Adv. Trained
98.2%
84.5%
96.4%
96.9%
85.9%
Original
99.5%
12.8%
97.8%
14.9%
98.0%
16.2%
Table 3: Accuracy of adversarially trained and original models on clean and adversarial test images from Song et al. (2018).
To further illustrate and compare distributions of real and adversarial images, we use a pre-trained VGG network to extract features of each image from CelebA-HQ. adversarial examples, and those (Sng et al. (2018), and then plot them with t-SNE embeddings as shown in Figure 7 We can see that the embeddings of CelebA-HO real and our adversarial images are blended while the those of CelebA-HQ and Song et al.'s adversarial examples are more segregated. This again provides evidence that our adversarial images stay closer to the original manifold and hence could be more useful as adversarial training data.
blue: CelsbA-HQ
blue: ColebA-HC
red: our adversarial
red: Song et
Figure 7: t-SNE plot comparing distributions of real images with adversarial examples from our approach and Song et
A.2 ADVERSARIAL TRAINING WITH BORN-BEUNDED PERTURBATIONS
We consider adversarial training with norm-bounded perturbations and limit the number of iterations to make the setup comparable with our unrestricted adversarial training. Specifically, we use Iterative-FGSM with and bounded number of steps. Results are shown in Table 4 Note that accuracy of the models drop on clean images although we use a weak attack. This in contrast to training with our unrestricted adversarial examples that improves the accuracy.
A.3 NUMBER OF ITERATIONS
To make sure the iterative process always converges reasonable number of steps, we measure the number of updates required to fool the classifier on 1000 randomly-selected images. Results are shown in Table 5) Note that for targeted attacks we first randomly sample a target class different from the ground-trutie label for each image.
12

---

Under review as a conference paper at ICLR 2021
IFGSM-2
IFGSM-5
Original 88.9%
Adv. Trained 88.4%
Adv. Trained
LSUN CelebA- -Q
87.8%
95.7%
95.1%
94.6%
Table 4: Adversarial Training with norm-bounded perturbations. Iteratve-FGSM (e 4) with a maximum of 2 and 5 iterations considered, and accuracy of the adversarially trained and original models on clean test images are shown.
LSUN
CelebA-HQ
Targeted
Non-targeted
Style-based
9.1 4.2
6.8 + 3.6
7.3 + 3.0
Noise-based
3.7 + 1.8
4.1
Table Average number of iterations (mean + std) required fool the classifier.
A.4 OBJECT DETECTION RESULTS
Figure illustrates results on the object detection task using the RetinaNet target model [Lin et al. (2017) We
observe that small changes in the images lead to incorrect bounding boxes and predictions by the model.
Iteration
Predictions
Predictions
Figure 8: Unrestricted adversarial examples for object detection. Generated images and their corresponding pre- dictions are shown for different number of iterations.
A.5 IMPACT OF AND 3 ON SEMANTIC SEGMENTATION RESULTS
In segmentation results shown Figure we simultancously modify both y and / parameters of the SPADE module (Figure 2) We also consider the impact of modifying each parameter separately. Figure iillustrates the results. As we observe, changing y and / modifies fine details the images which are barely perceptible yet they lead to large changes in predictions of the segmentation model.
A.6 ADVERSARIAL CHANGES SINGLE IMAGES
Figure illustratess how images vary as we manipulate specific layers of the network. We observe that each set of layers creates different adversarial changes. For instance, layers 12 mainly change low-level color details.
13

---

Under review as a conference paper at ICLR 2021
Iteration 0
Iteration
Iteration 8
Iteration 12
Iteration 16
Figure 9: Impact of separately modifying and parameters segmentation results. Modified images at different iterations and corresponding predictions are shown. the first two rows only the y values are changed and in the last two rows only the B values modified.
A.7 ADDITIONAL EXAMPLES
We also provide additional examples and higher-resolution images in the following. Figure TT depicts additional examples the segmentation task. Figure 12 illustrates adversarial examples CelebA-HQ gender classifi- cation, and Figure 13 shows additional examples on the LSUN dataset. Higher-resolution versions for some of the adversarial images shown in Figure 14 which particularly helps distinguish subtle differences between original and noise-basee based images.
14

---

Under review as a conference paper at ICLR 2021
Original
Layers 1-6
Layers 7-12
Layers 12-18
Figure 10: Impact manipulating different layers of the network on generated adversarial images.
Generated
Image
Prediction
Accuracy
48.2%
11.6%
0.0%
0.0%
0.0%
0.0%
Generated Image
Prediction
Accuracy
85.6%
35.8%
12.3%
0.0%
0.0%
0.0%
0.0%
Figure 11:
Unrestricted adversarial examples for semantic segmentation. Generated images, corresponding pre- dictions and their accuracy (ratio of correctly predicted pixels) are shown for different number of iterations.
15

---

Under review as a conference paper at ICLR 2021
Figure 12: Unrestricted adversarial examples CelebA-HQ gender classification. From top bottom: Original, noise-based and style-based adversarial images. Males are classified as females and vice versa.
16

---

Under review as a conference paper at ICLR 2021
Kitchen
Trolley
Tower
Conference room
Living room
Living room
Living room
Minivan
Tower
Kitchen
Living room
Living room
(a) Non-targeted
Cell phone
Mountain Tent
Bedroom
Classroom
Bridge
Restaurant
Cell phone
Mountain Tent
Bedroom
Classroom
Bridge
Restaurant
(b) Targeted
Figure 13:
Unrestricted adversarial examples LSUN for a) non-targeted and b) targeted attacks. From top bottom: original, noise-based and style-based images.
17

---

Under review as a conference paper at ICLR 2021
Figure 14: High resolution versions of adversarial images. From left to right: original, noise-based and syle-based images.
18

---

Under review as a conference paper at ICLR 2021
Figure 14: (cont.) High resolution versions of adversarial examples. From left to right: original, noise-based and style-based images.
19


Reviews:
[{"review_id": "XGAOuBw6Kf0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a mechanism to generate adversarial examples by applying latent variables level manipulation, based on the styleGAN framework. Unlike previous works mostly focused on image level perturbations and geometry transformations, this work tends to control higher level latent sampling such as style, so as to generate a style-adversarial examples. Although a similar idea has been proposed by Song et al. (2018), this work is along the same direction and achieves better performance. The loss is proposed for general classification tasks such as object classification, object detection and semantic segmentation. The experimental results show not only qualitatively confusing human vision but also quantitatively improve the performance on testing clean images.\n\n+ The paper is well written and easy to read.\n+ Experimental results demonstrate the proposed idea qualitatively and quantitatively.\n+ Sufficient ablation analysis to make the proposed method convincing.\n\nHowever, I still have some concerns:\n\n- There’s no experiment to compare with existing methods such as Xie et al. ‘20 and PGD. A standard experimental protocol ex ImageNet should be conducted and fairly compared.\n- Deeper analysis of the impact by y_adv and eta_adv in both object detection and semantic segmentation. Although final results in table 1 shows that style-based adversarial training benefits the performance, ablation studies of y_adv and eta_adv in both tasks should be conducted.\n- Since the attack is in the feature space, the defense should also happen in the feature space [a]; \n\nOverall, I think the paper is valuable. The proposed idea is novel, and sufficient experiments are provided to demonstrate the idea. Rich visualization to help readers understand the concept. I’m willing to raise my rate if my concerns are addressed.\n\n[a] M. Lecuyer et al., Certified Robustness to Adversarial Examples with Differential Privacy: https://arxiv.org/abs/1802.03471", "extra_scores": {}, "extra_text": {}}, {"review_id": "GNlnNAa8_W-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###Summary###\n\n\nThe paper proposes a method of generating adversarial samples to enhance classification performance.\nSpecifically, it finds variables of pre-trained generative model to produce images that the pre-trained classifier gives wrong answers.\nThe new classifier is then trained by adding these samples to the existing training dataset.\nThis method achieved good performance because the distributions of the samples generated in this way is closer to the distribution of the real images than the ones generated with norm-bounded perturbations.\n\n\n###Pros###\n\n\n-\nThe effectiveness of the proposed method is reasonably explained by comparing with the preceding works.\n\n\n-\nAuthors thoroughly analyzed the qualitative results.\nIt helps in understanding the underlying mechanisms.\n\n\n\n###Questions###\n\n\n-\nIs there a difference in performance between using non-targeted adversarial samples and targeted adversarial samples?\nIf so, how different is it?\nWhich of the two to use depends on the outcome?\n\n-\nWhat is the criteria that you divide layers as high-, mid- and low-level ones?\nHave you checked layer-wise effect?\n\n-\nHow one image varies in different settings?\nFor example, generation results in figure 3 using only one original image.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2zHSMvTZkyE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a new method for generating unrestricted adversarial examples.  Based on Style-GAN, this work separates stylistic and noise modifications so as to control higher-level aspects and lower-level aspects of image generation.\n\nBy handling style and noise variables separately and changing the different levels of synthesis networks, the model can input various types of perturbations in generating adversarial images.  As the authors claim, the style variables from different layers affect different aspects of images.  Generation of adversarial images are tested in both un-targeted and targeted attacks.  Overall,\nthe paper is well-motivated, well-written, and the method is evaluated with three tasks, classification, semantic segmentation, and object detection.\n\nOn the other hand, although the different layers of the networks are concerned with different aspects of the images and the proposed method can generate a variety of images, we may not be able to intentionally control specific aspects of the images.  This is an incremental work on top of Style-GAN so that the novelty of the paper is not very high.\n\nPlease make clear how the parameter values are determined.  For example, how did you select the step sizes?", "extra_scores": {}, "extra_text": {}}, {"review_id": "0GA_HFtHTYb", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n- Targets an important problem, adversarial attacks semantically constrained as opposed to being constrained by an artificial norm ball. \n- Extensive results with a wide variety of models, datasets, and more importantly, applications, with not only attack evaluation on standard models, but application to adversarial training, a user study, and evaluating against certified defenses. \n\nCons:\n- The disentangled representations of StyleGAN were used for generating realistic perturbations, and the application of training with said perturbations (adversarial training) was considered [1]. This paper isn't cited, let alone compared to given the significant similarity in the methodology. [2] considered constraining adversarials to be within the output space of a learned generative model, this work was also not cited. \n- The emphasis given to the result that adversarial training improves clean performance isn't fully justified, as unrealistic images, e.g. images not too perturbed, were controlled for by limiting the number of iterations. If one limits the number of iterations for a norm-based attack, or considers a smaller epsilon ball, the same control would be met, but a comparison to norm-based adversarials with the same control is not conducted. Note that adversarial training was originally viewed and used as a regularization method [3,4], improving i,i,d, performance, and the robustness-accuracy tradeoff can straightforwardly be mitigated by considering a weaker form of robustness by using a weaker attack (FGSM [3]), or controlling the strength of the perturbation, where the latter is exactly the filtering performed here. It is intuitive that semantic adversarials would provide benefit for the robustness-accuracy tradeoff, but this experiment certainly does not demonstrate that due to the obvious confounders. \n- The emphasis on the \"breaking\" of a certified defense can also be seen as overclaiming, as breaking a certified defense implies succeeding against the defense within their considered threat model. This is not done here, thus it doesn't violate any of the claims made by randomized smoothing, thus it does not break the certified defense. Though the impracticality of norm-based adversarial defenses are well-understood, this experiment simply shows that if one goes outside of a certified defense's threat model, the certified defense can perform worse, which is entirely expected. \n\nConclusion: On one hand, the research direction is valuable and the experimental results are extensive. On the other hand, comparison to the literature is sorely lacking, there exists techniques which share much of the same functionality of the method [1], a comparison to norm-based adversarials in the adversarial training experiment should have been done to clarify concerns that controlling for unrealistic adversarials is the source of the result, not the contribution of semantic adversarials, and the certified defense is not \"broken\", as the attack went outside of the defense's threat model. \n\nIn summary, this work does not compare to any baselines when baseline experimentation is clearly needed to justify the novelty and contribution of the work, and fair treatment of prior literature is missing, with not only citations missing for quite similar approaches, but claims are made which on the surface invalidate previous work (breaking randomized smoothing) when said previous work was not fairly evaluated (as the attack went outside of the defense's clearly specified threat model, thus not invalidating any of their claims). \n\nI implore the authors to consider their contribution in the context of the literature more carefully. \n\nReferences:  \n [1]: https://openaccess.thecvf.com/content_CVPR_2020/html/Gowal_Achieving_Robustness_in_the_Wild_via_Adversarial_Mixing_With_Disentangled_CVPR_2020_paper.html.  \n [2]: https://openaccess.thecvf.com/content_CVPR_2019/html/Stutz_Disentangling_Adversarial_Robustness_and_Generalization_CVPR_2019_paper.html. \n [3]: https://arxiv.org/abs/1412.6572.  \n [4]: https://arxiv.org/abs/1704.03976", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received two borderline accept recommendations and one accept recommendation from three reviewers with low confidence and a reject recommendation from an expert reviewer. 

Although all reviewers found that the paper addresses an important and challenging problem of semantically constraining adversarial attacks as opposed to constraining them artificially by an artificial norm ball. However, during the discussion phase it has been pointed out that there were some important weaknesses indicating that the paper may need one more evaluation round.  The meta reviewer recommends rejection based on the following observations. 

In terms of evaluation, while it is understandable the authors were unable to compare to Gowal et al. due to the lack of publicly available implementation, showing Song et al.'s adversarials hurt performance and and are farther than the image manifold has been found puzzling, as this was done by Song et al. only to keep human prediction the same while changing model prediction. Furthermore, the paper did not contain a user study similar to Song et al. for a fair comparison Finally, the discussion revealed that the comparison to "norm-bounded adversarial inputs" may not have clarified whether this experiment faithfully demonstrates an advantage for the contribution as the norm could be contained to a point where accuracy is not reduced, and the discussion on the certified defense being "broken" was inconclusive.
Decision: Reject