Generating Semantic Adversarial Examples with Differentiable Rendering
Authors: Lakshya Jain
Steven Chen
Wilson Wu
Uyeong Jang
Varun Chandrasekaran
Sanjit Seshia
Somesh Jha
Venue:  (2020)
Keywords: semantic adversarial examples
inverse graphics
differentiable rendering

Abstract: Machine learning (ML) algorithms, especially deep neural networks, have demonstrated success in several domains. However, several types of attacks have raised concerns about deploying ML in safety-critical domains, such as autonomous driving and security. An attacker perturbs a data point slightly in the pixel space and causes the ML algorithm to misclassify (e.g. a perturbed stop sign is classified as a yield sign). These perturbed data points are called adversarial examples, and there are numerous algorithms in the literature for constructing adversarial examples and defending against them. In this paper we explore semantic adversarial examples (SAEs) where an attacker creates perturbations in the semantic space. For example, an attacker can change the background of the image to be cloudier to cause misclassification. We present an algorithm for constructing SAEs that uses recent advances in differential rendering and inverse graphics. 

Under review as a conference paper at ICLR 2020
GENERATING SEMANTIC ADVERSARIAL EXAMPLES WITH DIFFERENTIABLE RENDERING
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning (ML) algorithms, especially deep neural networks, have demon- strated success in several domains. However, several types of attacks have raised concerns about deploying ML in safety-critical domains, such autonomous driving and security. An attacker perturbs a data point slightly in the concrete feature space (e.g., pixel space) and causes the MI algorithm to produce incorrect output a perturbed stop sign classified as a yield sign). These perturbed data points are called adversarial examples, and there are numerous algorithms in the literature for constructing adversarial examples and defending against them. In this paper we explore semantic adversarial examples (SAEs) where an attacker creates perturbations in the semantic space representing the environment that pro- duces input for the ML model. For example, an attacker can change the back- ground of the image to be cloudier to cause misclassification. We present an al- gorithm for constructing SAEs that uses recent advances in differential rendering and inverse graphics. The generated SAEs are easily realizable in the physical world, and can be used to help the model adapt better to different domains.
INTRODUCTION
Machine learning (ML) techniques, especially Deep Neural Networks (DNNs), have been successful in several domains, such as finance and healthcare. However, several test-time (Biggio et al., 2013; Szegedy et al., 2014; Goodfellow et al., 2015; Kurakin et al., 2016) and training-time (Jagielski et al., 2018: Shafahi al., 2018) attacks have made their adoption in high-assurance applications, such as autonomous driving and security, problematic. ML techniques, such as generative models, have also been used for nefarious purposes such as generating "deepfakes" (Liu al., 2017; Zhu al., 2017). Our focus in this paper is on test-time attacks in which an adversary generates a slightly perturbed sample to fool classifier or an object-detector.
Let X be the sample space and Y be the space of labels. A classifier F is a function from X to Y. Given a sample x E X, most attacks for constructing adversarial examples find a perturbation 6 with small norm (typical norms that are used are l.c. lo. and (2) such that x +6 has different label than x, i.e. F(x) * F(x +6). In this paper we consider the problem of generating semantic adversarial examples (SAEs) (Hosseini & Poovendran, 2018; Joshi et al., 2019; Qiu et al., 2019; Dreossi et al., 2018b). In these examples, there is a richer set transformations T that capture changes to inputs the ML model. We assume norm on T (this norm is induced by various parameters corresponding to the transformations. such as angle of rotation and size of the translation). our universe, an adversary is given a sample x and wishes to find a transformation parameterized by 0 θ with small norm such that F (x 0)) of F(x) (we consider untargeted attacks, but our ideas extend to targeted attacks as well)'
semantically-meaninee
SAES can also be viewed as outcomes of perturbations in "rich" semantic feature space (e.g.., texture of the image) rather than just the concrete feature space (e.g.. pixels). Consequently, SAES are physically realizable, and it is easy to understand how the changes in semantics results in an adversarial example. SAEs have been considered the literature (Xiao al., 2018; Dreossi al., 201 8b;; Huang et al., 2019), but prior works typically consider a small set of fixed transformations (e.g., rotation and translation, or modifying a single object's 's texture). Our goal is to flexibly support
1 is defined in 3.2.

---

Under review as a conference paper at ICLR 2020
richer set of transformations implemented in state of the art renderer (e.g., changing the background of the image, weather conditions, or the time of day), as we specify in detail in 2. There is evidence that SAEs can help with domain adaptation (Volpi et al., 2018) making the control loop more robust (Dreossi et al., 201 further motivating our approach. We also plan to open source our code advance research in this space.
To summarize, the main contributions of this paper are the following:
We present new class of rest-time attacks in the form of SAEs. We demonstrate how to generate SAEs that support a rich set of transformations (refer I 3) using an inverse graphics framework (refer 8 2). Specifically, we show how one can systematically take techniques to perform attacks the pixel space such as FGSM (Goodfellow et al., 2015) and PGD (Madry et al., 2017) and transform them to their semantic counterparts.
We evaluate the generated SAEs on the popular object detector SqueezeDet (Wu et al., 2016). By correctly choosing the semantic parameters, SAEs degrade performance (char- acterized by the mean average precision or mAP) by 28 percentage points (refer 5.1).
We show that by augmenting the dataset using SAEs, we can boost the robustness of SqueezeDet (characterized by mAP) by up to 15 percentage points (refer $ 5.2).
We show that SAEs generated using SqueezeDet do not transfer YOLOv3 (Redmon & Farhadi, 2018), indicating a tight coupling between the model architecture and SAE generation (refer $ 5.3).
While augmentation with SAES improves robustness against SAEs, augmentation using traditional pixel- basel perturbations does not produce the same effect (refer & A.I).
2 RELATED WORK
Adversarial Examples and Robustness: There is extensive research for generating adversarial examples in the pixel space; we henceforth refer to these as pixel-perturbations.. Goodfellow et al. (2015) propose the fast gradient sign method (FGSM) where inputs modified in the direction of the gradients of the loss function with respect to input, causing a variety of models to misclassify their inputs. Madry et al. (2017) generalize this approach and propose the projected gradient descent (PGD) approach working using the same intuition. While these approaches suggest modifications to the raw pixel values, other methods of generating adversarial examples exist. Athalye et al. (2017) introduce an approach to generate 3D adversarial examples (over a chosen distribution of transformationss. Engstrom et (2019) observe that modifying the spatial orientation of images results in misclassification. Similarly, Geirhos et al. (2018) discovered that certain models are biased towards textural cues.
To improve robustness, current approaches include adversarial training (Madry et al., 2017), smoothing-based approaches (Cohen et al., 2019; Lécuyer et al., 2018), or through specific regu- larization (Raghunathan et al., 2018). An alternative approach, utilizing some notion of semantics, is advocated in the work Guo et al. (2017). The authors augment the training set with transformed versions of training images, utilizing basic image transformations (e.g... scale and re-cropping) and total variance minimization, and demonstrate an improvement in robustness. Dreossi et al. (201 8a) improve the robustness of SqueezeDet (Wu et al., 2016) through counterexample guided data aug- mentation; these counterexamples are synthetically generated by sampling from space of transfor- mations and applying them original training images.
The process of finding 3D scene parameters (geometric, textural, lighting, etc.) given images referred to as inverse graphics (Baumgart, 1974). There is a history of using gra- dients to solve this problem (Blanz & Vetter, 2002; Shacked & Lischinski, 2001; Barron & Ma- lik, 2015). Kulkarni et al. (2015) propose a model that learns interpretable representations of im- ages (similar to image semantics), and show how these interpretations can be modified to produce changes in the input space. Pipelines for general differential rendering were proposed by Loper & Black (2014) and Kato et al. (2018). Li et al. (2018) design general-purpose differentiable ray tracer; gradients can be computed with respect to arbitrary semantic parameters such as camera pose, scene geometry, materials, and lighting parameters. Yao et al. (2018) propose a pipeline that, through de-rendering obtains various forms of semantics, geometry, texture, and appearance, which can be rendered using generative model.
Inverse Graphics:
2

---

Under review as a conference paper at ICLR 2020
Semantic Adversarial Examples (SAEs): The notion of SAEs has been proposed in the literature in specific contexts. For example, in the NLP domain, Lei et al. (2018) propose gradient guided greedy algorithm to make semantic changes to text documents. Sharif et al. (2016) propose attacks against facial recognition systems that are both inconspicuous and realizable. Similarly, Evtimov al. (2017) obtain patches that can be added to road signs, rendering object detectors useless. In both these works, however, the modifications to semantics are not found by searching the space of semantic modifications (and found instead by realizing changes in pixel intensities); such ap- proaches will fail to generalize for other domains. Dreossi et al. (2018b) present an approach to generate SAEs based on systematic sampling of the semantic space coupled with verification tech- niques for a closed-loop cyber-physical system containing the ML model, where they treat the ML model as a black box. However, this sampling-basee
"black box" approach faces scalability issues when dealing with high-dimensional spaces and also fails to exploit structure of the MI. model to perform a targeted search for SAEs. Xiao et al. (2019) utilize a differentiable rendering framework to introduce changes in shape and texture that are capable of fooling a variety of ML models for various tasks. However, our approach takes into account advances in differentiable inverse graphics permitting us change a richer set of semantic parameters. An interesting contribution by Qiu et al. (2019) suggests how a generative model can be used to introduce semantic modifications. However, generative models are notoriously difficult train and operate, making them unusuable for wide variety of tasks. Similar issues occur in the work by Joshi et al. (2019).
3 SEMANTIC ADVERSARIAL LEARNING
Consider a space Z of the form X x Y, where X is the sample space and Y is the set of labels. From here on we will assume that X Rm Let be a hypothesis space (e.g.., weights of a DNN). We assume a loss function E H. x Z + E so that given hypothesis w H and a labeled data point (I, 3) E Z, the loss is f(w, x, V). The output the learning algorithm is a classifier, which is function from Re to Y. To emphasize that classifier depends a hypothesis E H, which is output of the learning algorithm, we will denote it as Fw (if w is clear from the context, we will sometimes simply write F).
3.1 TRADITIONAL ADVERSARIAL EXAMPLES
We will focus our discussion on untargeted attacks, but our discussion also applies to targeted at- tacks. An adversary A's goal is to take any input vector x E R² and produce a minimally altered version of adversarial example denoted by A(x), that has the property of being misclassified by a classifier F R. + Y. The adversary wishes to solve the following optimization problem:
m such that F(x + 8) * F(x)
(())
The various terms in the formulation are: μ is a norm on R": μ can be 1.0, 40 4, or 1p (p 2 2). While not representative, these norms are commonly used an approximation of human's visual perception (Sen et al., 2019). 5 is the solution of the optimization problem given above, then the adversarial example A(x) x +8.
FGSM. The fast gradient sign method (FGSM) (Goodfellow et al., 2015) was one of the first un- targeted attacks developed in literature. The adversary crafts an adversarial example for given legitimate sample x by computing (and then adding) the following perturbation:
8 8= esign V Lx))
(1)
The function L (x) is a shorthand for E(wr x, I(x)), where w the hypothesis corresponding to the classifier F. x is the data point and 1(x) is the true label of x (essentially we evaluate the loss function at the hypothesis corresponding to the classifier). The gradient of the function LF is computed with respect to x using sample and label y l(x) inputs. Note that (x) is an n-dimensional vector and sign LL, (x)) is a n-dimensional vector whose tth element is the sign of the VxLp(x)) (x)] The value of the input variation parameter factoring the sign matrix controls the perturbation's amplitude. Increasing its value increases the likelihood of A(x) being misclassified by the classifier F but also makes adversarial examples easier to "detect" by humans. The key idea
3

---

Note that (..) is the projection operator in the parameter space We also assume that the projection operator will keep the parameters in the feasible set, which depends the image (e.g.. translation does not take the car off the road). The operator @ is the aggregation operator (similar to Under review as a conference paper at ICLR 2020
is that FGSM takes step in the direction of the gradient of the loss function with respect to the input, thus attempting to maximize the loss function using its first-order approximation. Recall that stochastic gradient descent (SGD) takes step in the direction that is expectation opposite to the gradient of the loss function because it is trying to minimize the loss function.
PGD. In Projected Gradient Descent (PGD) (Madry al., 2017), find perturbation in iterative manner. The PGD attack can be thought of an iterative version of FGSM. Assume that we using the by norm. Assume Xo is the original sample
Xk+1 (x,,)))) (X+ +s sign( V (x))(x)))
The operator (x.ss)) (y) is the projection operator, iee. it takes as input point and outputs the closest point in the s-ball (using the l (--norm) around x. The iteration stops after certain number of steps (the exact number of steps is hyperparameter).
3.2 SEMANTIC ADVERSARIAL EXAMPLES (SAES)
Let T RRt Q) - Rn be set of transformations parameterized by a space O and μu is a norm over θ. The reader can think of ④ as parameters that control the transformations (e.g., the angle of rotation). Given O, (xx, ()) the image transformed according to the parameters We assume that there is special identity element in & (which we call 1) such that 7xx, 1) x. An adversarial attack in this universe is characterized as follows:
mingge such that F(r(x, + F(x)
m())
In other words, we ideally want to find a "small perturbation" in the parameter space θ that will misclassify the sample?. Consider the function LF LF (Tx, 0)). The derivative with respect to 0 is is the transposed Jacobian matrix of vector-valued function of 0, evaluated at and (zLr(z)la=rt> is the derivative evaluated at T(x, 0)). The semantic version of FGSM (xFGSM) will produce the following 0:
(2Lr(z)22=(xx))) (the notation x
Or x
80
0" sign
0r
The adversarial example A(x) is r(x, 0*). Note that do not assume any special properties about т, such as linearity. We only assume that differentiable.
In similar manner semantic version of the PGD attack (sPGD) be constructed. Let 8g =1 and Xy x. The update steps correspond to the following two equations:
0tt1 = II (0o.s) (Ok (D sign
[3r 89
V2Lp(z)la=rtaalu)
(x)
X;+1 = T(Xo, 0x+1)
addition in )²,,, but in the parameter space O The precise axioms satisfied by + depends on O but one axiom we require is:
(x, 01 + #2) r(r(x, 01),
In fact, our recipe can be used to transform any attack algorithm such as Carlini & Wagner (2017) that adds perturbation to its "semantic version" as follows:
However, the existence of parameter space not homogeneous and is unclear one function can be used to capture the distance amongst all these transformations. Not only should the norm measure the changes in the semantic space, it should also approximate human perception. In its presence, would be able to effectively measure the distortion introduced by the adversary in the semantic space.
function for the semantic space an open research question: the semantic
4

---

Under review as a conference paper at ICLR 2020
Replace 8 with 0.
Replace with T(x, 0).
Use chain rule compute the gradients of terms that involve T(x, 8).
L.DE-RENDERING
3.RENDERING
INPUT
SEMANTIC ADVERSARIAL EXAMPLE
2.ATTACK
Figure I: The input is de-rendered (step 1) to its intermediary representation (IR) )- semantic, graphic, and textural maps. Then, this adversarially perturbed (e.g.. the red car is rotated) as described in $ 3.2 (step 2). The resulting IR is then re-rendered to the generate the SAE (step 3).
Differentiable rendering and inverse graphics. We apply the above framework to images by em- ploying differentiable snndrerrde-renderss in an inverse graphics setting. Such an inverse graphics setting can be thought of two transformations: (a) de-renderer 3 R - S, and (b) a renderer y : R". Here, S is the intermediate representation (IR). the differentiable renderer/de- renderer we utilize (Yao al., 2018), the IR contains semantic map, texture codes, and 3D at- tributes. Let be the set of changes to the IR (e.g., change to the texture code to make it more cloudy) and LE @ corresponds to the identity. Suppose there is operator (8 θ) - S that given a E transforms the IR, i.e. a(s, 0) = s' for s, s' S. In this case, the function T (xx 8) is equal to ((((((x), 0)). We the fact that for differentiable renderers/de-rente 9, are differentiable and hence attacks like sFGSM and sPGD can be implemented.
the functions
4 VALIDATION
In this section, describe the various components used in our implementation generate SAEs, and describe experiments carried out to determine the impact of choice of semantic parameters to- wards generating effective SAEs for object detectors. We stress that the exact choice of components are not important; our contribution is our automated approach to combine these components to gen- erate SAES in a manner that general (see 3.2 for more details) and would work with any inverse graphics framework. We also believe that our methodology will work for other tasks such as clas- sification and semantic segmentation; these tasks require optimizing over different cost function. In other words, if there is existing pixel attack, should be able to easily transform it into semantic attack (with the right inverse graphics framework).
4.1 IMPLEMENTATION DETAILS
The three main components required to successfully generate SAES include: (a) a differentiable inverse graphies framework, (b) victim model (which also differentiable), and (c) an attack strategy. We describe our choices for the proof-of- -oonc-tf-conceptt below.
To obtain the semantics associated with our inputs and generate the final SAEs, we use the inverse graphics framework(i.e. a combination of a semantic, textural and geometric de-rendering pipeline and generative model for rendering) created by Yao al. (2018). The models in this framework were trained entirely using the VKITTI dataset (Gaidon et al., 2016). These images comprise of simulations of cars different road environments virtual worlds. The VKITTI dataset con- structed using novel real-to-virtus cloning methodology, mirroring many elements that exist in real world. The de-rendering pipeline is used to obtain the initial semantic features associated with input images. These semantic features include (a) color: the car's texture codes, which change its color, (b) weather: the weather and time of day, (c) foliage: the surrounding foliage and scenery, (d) rotate: the car's orientation, (e) translate: the car's position in 2D, and (f) mesh: the 3D mesh which
5

---

Under review as a conference paper at ICLR 2020
provides structure to the Building inverse graphics framework for other tasks/datasets is non-trivial process which is outside the scope of this paper.
The final SAEs were produced using the generative model (which is part of the inverse graphics framework). Specific modifications were made to the differentiable graphics framework we used ensure that gradients were easy calculate. The codebase did not originally support end-to-end differentiation as each branch (semantic, geometric, textural) was trained separately. In partieu- lar, several image manipulation operations (normalization, rescaling through nearest-neighbor and bilinear interpolation) were implemented in a non-differentiable manner. We implemented the dif- ferentiable equivalents of these operations allow backpropagation. Furthermore, we implemented a weak perspective projection for vehicle objects, as well as an improved heuristic for inpainting of gaps the segmentation map due to object translations/rotaiions, in order improve the quality of the rendering. Additionally, the inverse graphics framework and the object detector (which we describe next) had contrasting dependencies (for libraries, tensorf Low, cuDNN, and CUDA); resolving these dependencies involved significant code rewriting.
We use the popular and representative SqueezeDet object detector (Wu et al., 2016) as the victim model. This model was originally trained on the KITTI dataset (Geiger et al., 2013). We perform transfer learning on this model using 6339 randomly chosen images from the VKITTI dataset; we wanted the object detector to better adapt images outside the domain it was initially trained for. However, images produced by the differentiable graphics framework contain artifacts (Lee. distor- tions in the images); these artifacts could be mistaken for pixel perturbations and would impact our evaluation results. To deal with this issue, we retrain SqueezeDet using identity transform re- rendered images' produced by the generative model. Finally, utilize these gradients and the semantics associated with each input in crafting adversarial attacks using the iterative sFGSM (for iterations). On average, for CPU-only implementation, generating such a SAE requires 42.81 seconds with about 85% of the time expended in the inverse graphics framework. We also stress that our choice of the number of iterations restricted by our choice of the differentiable graphics framework. Using more iterations resulted in unintelligible outputs. We believe that the generation of SAEs is not inhibited by the complexity of the semantic parameter space; better engineering can substantially improve the outputs and the time it takes to generate them.
Figure 2: Semantic space adversarial examples. Benign re-rendered VKITTI image (left), adver- sarial examples generated by iterative sFGSM over a combination of semantic features (right). Cyan boxes indicate car detected, purple indicated pedestrian, and yellow indicate cyclist. The adversarial example introduces small changes in car positions and orientations, and noticeable changes their color. This causes the network to detect pedestrians where there are none (top) and to fail detect a car in the immediate foreground (bottom).
4.2 SELECTING SEMANTIC PARAMETERS
In the pixel perturbation setting, all pixels are equal i.e. any pixel be perturbed. Whether such homogeneity naturally exists in the semantic space is unclear. However, we have additional flexi- bility; we can choose to modify any of the above listed semantic parameters independently without
Images passed through the de-rendering and rendering framework, without modifying the IR any asso- ciated semantic parameters.

---

Under review as a conference paper at ICLR 2020
altering the others, i.e. perform single parameter modifications. Alternatively, modify any subset of the parameters in unison, i.e. perform mulli-ppaammen modifications. The degree mod- ification is determined by the input variation/step-stii hyperparameter [0. 1]. In the context of pixel perturbations, the step-size corresponds to the maximum permissible change of pixel. For SABs, the value of is proportional to the magnitude of the geometric and textural changes induced; the effect depends on the semantic parameter under consideration.
Large values of e result in unrealistic images created by the generative model (examples of this include perturbing the mesh to the point where cars are twisted into shapes no longer resembling vehicles). To avoid such issues and simulate realistic transformations, we use a different step-size for each semantic parameter. We test various values of for each semantic parameter, and report the best choice for brevity. Specifically, (a) color: & = 0.05, (b) weather: E 0.25, (c) foliage: e 0.01, and (f) mesh: = 0.025. We stress these hyperpa- rameters were obtained after extensive visual inspection (by viewers independently): norm-based approaches typically serve proxy for visual verification (Sen et al., 2019). Additionally, choice in hyperparameters enables us to use the same ground truth labels throughout our experi- ments; e.g., produced SAEs have bounding box coordinates that enable truth labels as their benign counterparts". We also measure the Fréchet Inception Distance (FID) scores (Heusel et al., 2017) for the generated SAEs; we observe that the score is 0.102362 (com- pared to 0.01651 for generating pixel perturbations), providing quantitative evidence that the SAFS are similar to the benign images they were generated from.
0.10, (d) rotate: = 0.01, (e) translate:
the same ground
We produce 50 SAES for each semantic parameter combination choice. We then evaluate the efficacy of generated SAEs on SqueezeDet by measuring its (a) recall percentage, and (b) mean average precision, or mAP, in percentage. These metrics have been used in earlier works (Xie et al., 2017).
Parameter
color
weather
foliage
translate
rotate
mesh
recall
100
100
100
100
100
98.7
mAP
99.5
98.8
99.7
99.2
98.2
98.7
Table 1: Performance of SqueezeDet on SAEs generated using single parameter modifications. The model had (a) recall 100, and (b) mAP = 99.4 on beniga/non-addeeraaiall inputs. We observed that single parameter modifications are ineffective.
From Table is clear that single parameter modification is ineffective generating SAEs. Thus, generate SAEs using the multi-parameter modification method. To this end, we generated SAEs using the 57 remaining combinations semantic parameters. One could consider a weighted com- bination of different semantie parameters based pre-defined notion of precedence. However, choose a non-wen noi-weeghted combination. The results our experiments are in Table 2. For brevity, we omit most of the combinations that do not result in significant performance degradation (and discuss the insight we gained from them in 3 5.1). In the remainder of the paper, we report our evaluation using the translate + rotate mesh parameter combination to generate SAES.
Parameters
Iranslate rotate
Iranslate rotale mesh
translate mesh
rolale + mesh
recall
100
100
100
100
mAP
82
65.9
80.8
98.7
Table 2: Performance of SqueezeDet SAEs generated using multi-paramedd modifications. The model had (a) recall = 100, and (b) mAP = 99.4 on certain combinations of multiple parameters are effective towards launching attack.
inputs. We observed that
"This fact is useful when evaluate model robustness through retraining the models with SAEs inputs, which we discuss 5.2
7

---

Under review as a conference paper at ICLR 2020
5 EVALUATION
We designed and carried out experiments to answer the following questions: (1) Do SAEs cause performance degradation in SqueezeDet?, (2) Can the generated SAEs be used for improving ro- bustness?, and (3) Do the SAEs (generated using SqueezeDet) transfer to YOLOV3 (Redmon & Farhadi, 2018)?
We use 6339 images for training our Squeezelet model, and evaluate the model using 882 SAES. To evaluate the robustness, we augment the training dataset with 1547 SAEs and retrain the model. The various components of our framework and the datasets used are highlighted in $ 4.1. Note that SqueezeDet's ' loss function comprises three terms corresponding to (a) bounding box regresson, (b) confidence score regression, and (c) classification loss. In our experiments, we target the confidence score regression loss term to impact the mAP and recall of the model. All code was written in python. Our experiments were performed on two servers. The SAE generation was carried out on server with an NVIDIA Titan GP102 GPU, 8 CPU cores, and 15GB memory. All training and evaluation was carried out on server with 264 GB memory, using NVIDIA's GeForce RTX 2080 GPUs and 48 CPU cores. Our experiments suggest that: (1) SAES are indeed effective in degrading the performance Sque We also observe that the model is susceptible to changes that target the geometry of the input (cars in this case) rather than the changes in the background (refer ğ 5.1), (2) The generated SAEs do, in fact, help in improving model robustness. Our experiments show that SAE-based data augmentation can improve mAP by up to 15 percentage points (refer & 5.2), and (3) SAES generated with SqueezeDet do not transfer to YOLOv3. (see $ 5.3).
We do report other metries (classification accuracy, background error, etc.) associated with detection as our experiments are not designed to alter them.
5.1 EFFECTIVENESS OF SAES
The results in Table in S 4.2 demonstrate the effectiveness of SAEs, and offer two insights.
First, the victim model was more susceptible to transformations that modify the geometry of the input (such as translate and mesh) than other types of transformations. This has dire implications for anfety-critical applications: for the cars in our inputs, modifications in the mesh parameter results in deformed cars as outputs. These are common occurrences sites of accidents, and need to be detected correctly. A combination of translations and rotations also seem to compound the degrada- tion to the performance of the network (refer Table 2). This is most likely due the introduction of unique angles and visual perspectives that are not frequently encountered in assembled datasets. Unlike pixel perturbations, SAES are easy to interpret, í.e. we are able to understand how the model fails to generalize specific changes in input semantics. Additionally, they easier to realize i.e. the situations described above (related to translation and deformation of vehicles) occur on a daily basis. Intuitively, changing the geometry of the car can be viewed as targeting the perception of what a car really is - if the human can recognize that the object in question a car but a model cannot, then the model is not exposed to the sufficient variety of car shapes, positions, and orientations that it may encounter in real-world scenarios; i.e. it is unable to domain adapt (Tzeng et al., 2017).
The second insight we gain is that the model was more susceptible SABS caused by changing multiple parameters simultaneously. We evaluate the model with 882 SAES generated using a com- bination of the parameters listed in & 4.2. We observe that compared to the baseline performance on non-ad awwaaadrraraabbniigeeee inputs (recall = 93.63, mAp = 85.95). SAEs cause significant perfor- mance degradation (recall = 93.17, mAP = 57.78). As stated before, these combinations are easily realizable, and the model's poor performance is indicative of poor domain adaptation.
5.2 DATA AUGMENTATION To INCREASE ROBUSTNESS
As we have established that SAEs are effective in attacking SqueezeDet, we wished to enhance the model's robustness through data augmentation, out two sets of experiments. In the first, we incrementally (re)trained the benign SqueezeDet model on combination of benign inputs and SAEs (4792+1547) for 24000 iterations. In the second, tuned our benign model using just SAES (1547) for 6000 iterations. The results of our experiments presented in Table 3.
in Dreossi et al. (2018b). To this end, we carried
8

---

Under review as a conference paper at ICLR 2020
Model
Baseline
Retrained (SAE + Benign)
Tuned (SAE)
recall
93.17
92.97
92.15
mAP
57.78
72.76
72.63
Table 3: Performance of SqueezeDet on SAEs when (b) the model is retrained (on combination of SAEs + benign inputs), and (c) the model is tuned (on just SAEs), compared to (a) the baseline model (trained on benign images) on SAEs. Both retraining and tuning improve mAP.
It is clear that both approaches provide comparable increase in mAP while not impacting recall. Ad- ditionally, we found that making a model robust to semantic perturbations through either procedure described earlier allowed to achieve good performance on benign inputs. On benign inputs, we found that for the Retrained (SAE + Benign) model, recall = 93.7 and mAP = 84.73, while for Tuned (SAE), recall 91.9 and mAP 79.1. This is comparable to the performance of the baseline model (which was trained and validated on benign inputs), where recall 93.6 and mAP = 86.17.
Our results suggests that SAE-based augmentation promising direction for exploration; based on insight from 3, we could formulate a framework for semantic adversarial training, similar to (Madry et al., 2017). We leave the exact formulation to future work.
5.3 INVESTIGATING TRANSFERABILITY
To understand if SAEs transfer, tuned YOLOv3 network that initially trained with the KITTI dataset, for 50 epochs. To do so, we use the same 6339 benign re-rendered images from the VKITTI dataset (as did before), generated by the inverse graphics framework. The mAP of this network is 11.28%. When we tested the YOLOv3 network with the generated SAEs, we observed drop in mAP 87.50% (which is not significant in the case with SqueezeDet). This suggests that SAEs are effective against the network they are generated from, suggesting tight coupling between the two. Similar observations have been made with pixel perturbations (Tramèr et al., 2017); further analysis is required to understand this phenomenon, which we hope to pursue in future work.
While we investigate if SAEs transfer across different models and architectures, an interesting ques- tion revolves around the transferability of SAEs across tasks. We believe that SAES generated to fool object detectors will not work against classification semantic segmentation networks as the generation process was not targeted for these tasks. That being said, it is trivial to modify our general construction (by changing the cost/loss function) and generate SAEs for tasks such classification and semantic segmentation. We leave generating such SAEs for future work.
CONCLUSIONS
In this paper, we describe semantic adversarial examples (SAEs), where adversaries perturb the semantics of inputs produce outputs that misclassified. Such instances easier realize in the physical world, and are more interpretable than their traditional pixel-based counterparts. We propose an algorithm to construct SAES using advances in differentiable rendering, and evaluate the effectiveness of our approach. We observe that SAES cause performance degradation in object detector networks (SqueezeDet), and that data augmentation using SAEs increases robustness the model.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. CoRR, 0/4/1707.07355 20997,2011 2017. URL http:::t///xxi..rrp/pppe httprg/bb//777...
J. T. Barron and J. Malik. Shape, illumination, and reflectance from shading. IEEE Transactions on Pattern Analysis and Machine Intelligence. 3708):1770-1687, Aug 2015. doi: 10.1109/TPAMI. 2014.2377712.
Bruce Guenther Baumgart, Geometric Modeling for Computer Vision. PhD thesis, Stanford, CA, USA, 1974. AAI7506806.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smnićć Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. In Joint European conference on machine learning and knowledge discovery databases, pp. 387-402. Springer, 2013.
Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. SIGGRAPHH'99 Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 09 2002. doi:
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposiun on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Jeremy M. Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via ran- domized smoothing. CoRR, abs/1992.00291, 2019. URL htLhttp///axxx..orrg wttp:////.aabb...... 02918.
Tommaso Dreossi, Shromona Ghosh, Xiangyu Yue, Kurt Keutzer, Alberto L. Sangiovanni- Vincentelli, and Sanjit Seshia. Counterexample-guidee data augmentation. CoRR, abs/1805.06966 2018a. URL /tRL:////xxxvv../////aa///
06962.
Tommaso Drcossi, Somesh Jha, and Sanjit A. Seshia. Semantic adversarial deep learning. CoRR, abs/1804.07045. 2018b. URL httt:::/aaxxv.oog///oo hatp://a/////// /1004. 07045.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Ex- ploring the landscape of spatial robustness. In International Conference on Machine Learning,
pp. 1802-1811, 2019.
Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Song. Robust physical-world attacks on machine learning models. CoRR, abs/1707.08945. 2017. URL
org/ abs 1707. 08945.
Adrien Gaidon, Qiao Wang, Yohann Cabon, and Elconora Vig. Worlus as proxy for multi- object tracking analysis. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4349, 2016.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The KITTI dataset. International Journal of Robotics Research (IJRR), 2013.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. CoRR, 1..//811..1225 2018. URL http:://aaxii..o//// abs/1811 12231.
Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and harnessing adversarial examples. In International Conference Learning Representations, 2015. URL http:// arxiv. org.v.org//o abs 412.6572
Chuan Guo, Mayank Rana, Moustapha Cissé, and Laurens der Maaten. Countering adversarial images using input transformations. CoRR, abs/1711.00117,, 2017. URL /taxiii. org/abs/ 71 00117.
10

---

Under review as a conference paper at ICLR 2020
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a nash equilibrium. CoRR, abs/1706.08500, 2017. URL
org/ abs 1706. 08500.
Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. CoRR, abs/1804.0000 18040009999 2018. URL htt.: arwiv. org abs . C0499.
Lifeng Huang, Chengying Gao, Yuyin Zhou, Changqing Zou, Cihang Xie, Alan Yuille, and Ning Liu. Upe: Learning universal physical camouflage attacks on object detectors, 2019.
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learn- ing. In 2018 IEEE Symposium on Security and Privacy (SP), pp. 19-35, 05 2018. doi: 10.1109/SP:2018.00000
Ameya Joshi, Amitangshu Mukherjee, Soumik Sarkar, and Chinmay Hegde. Semantic adversarial attacks: Parametric transformations that fool deep classifiers. CoRR, abs/1904.08489, 2019. URL http: /tttp://arxvv...//// orgoo//////o... hatg//a////a /ttos/1900 08489.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. pp. 3907-3916,
06 2018. doi: 10110//CVPR.201800044
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in neural information processing systems, pp. 2539-2547, 2015.
Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial examples in the physical world. CoRR, abs/1607.02533. 2016. URL http://arxiv org/ /obb/16001 02533.
Mathias Lécuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. 2019 IEEE Symposium on Security and Privacy (SP), pp. 656-672. 2018.
Qi Lei, Lingfei Wu, Pin- Yu Chen, Alexandros G. Dimakis, Inderjit S. Dhillon, and Michael Wit- brock. Discrete attacks and submodular optimization with applications to text classification. CoRR, abs/1812.00151,, 2018. http:tt://aaxitt org/ hatp//bss//1....
Tzu-Mao Li, Mika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. In SIGGRAPII Asia 2018 Technical Pupers, pp. 222. ACM, 2018.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-imaage translation networks. CoRR, abs/1 002117030004444 2017. URL http: wtwaarivvvvv//.va//.aaaaaaaaavva org/ abs////////// 111111073300......
Matthew Loper and Michael Black. Opendr: An approximate differentiable renderer. 09 2014. doi: 11.1007/978-3-3199 --1197788---111-188-001111
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ArXiv, abs/1706.06083, 2017.
Haonan Qiu, Chaowei Xiao, Lei Yang. Xinchen Yan, Honglak Lee, and Bo Li. Semanticadv: Gener- ating adversarial examples via
image editing. CoRR, abs/1906.07927, 2019. URL pattt::/arxiie org/ abs/1906 07927.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial exam- ples. CoRR, abs/1801.009 10s00100,,,,44, 2018. URL
org/ http//bss1/111 111//111111000 09344.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement.
arXiv preprint
178.07180..02767, 2018
Ayon Sen, Xiaojin Zhu, Liam Marshall, and Robert Nowak. Should adversarial attacks use pixel p-norm? arXiv preprint arXiv:1906.024433 2019.
Ram Shacked and Dani Lischinski. Automatic lighting design using a perceptual quality metric. Comput. Graph. Forum, 20, 09 2001. doi: 10.

---

Under review as a conference paper at ICLR 2020
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems, pp. 6103-6113, 2018.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1528-1540. ACM, 2016.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference Learning Representations, 2014. URL http: / araii. org abs 1312.616 6199.
Florian Tramèr, Nicolas Papernot, lan Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv: 170403343, 2017.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint arXiv: 12152, 2018.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7167-7176, 2017.
Riccardo Volpi. Hongseok Namkoong, Ozan Sener, John C. Duchi, Vittorio Murino, and Sil- vio Savarese. Generalizing to unseen domains via adversarial data augmentation. CoRR, abs/1805.1200 2018.
Bichen Wu, Forrest N. landola, Peter H. Jin, and Kurt Keutzer. SeueezeDet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving. CoRR, abs/1612.01051, 2016. URL http::/t///. /arxiviv www.i.wwgg//..ooto...
org/ abs 1612. .2.20000 01051.
Chaowei Xiao, Jun Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially trans- formed adversarial examples. CoRR, abs/1801. 02612.
2018. URL http::/arxiv.cr/
Chaowei Xiao, Dawei Yang. Bo Li, Jia Deng, and Mingyan Liu. Meshadv; Adversarial meshes for visual recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Cihang Xie, Jianyu Wang. Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 1369-1378, 2017.
Shunyu Yao, Tzu-Ming Harry Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, William T. Freeman, and Joshua B. Tenenbaum. 3d-aware scene manipulation via inverse graphics. In Advances in neural information processing systems, 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In 20/7 IEEE International Conference on Computer Vision (ICCV). pp. 2242 2251, 10 2017. doi: 10. 20///
12

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 PIXEL PERTURBATIONS VS. SAES
Our experiments suggested that pixel perturbations are more effective degrading SqueezeDet's performance (recall=2.9, mAP=0.05); conjecture this is due to the larger feature space within which a solution for the optimization can be found i.e. the space of pixels larger than the space of specific semantic parametenstrransforationn
consider.
To measure the robustness provided by pixel perturbations against semantic perturbations, we per- formed the same experiment in $ 5.2. In experiment, retrained the benign SqueezeDet model on a combination of benign inputs and pixel perturbations (4792+1547) for 24000 iterations. In another experiment, we tuned our model using just pixel perturbations (1547) for 6000 iterations. The results of our experiments are presented in Table 4.
Model
Retrained (Pixel Benign)
Tuned (Pixel)
recall
92.4
90.43
mAP
56.7
55.35
Table 4: Performance of SqueezeDet SAEs when the model is (a) retrained (on pixel perturbations benign inputs), and (b) tuned (on just pixel perturbations). Retraining and tuning (with pixel perturbations) is ineffective against SAEs.
We observed that data augmentation using pixel perturbations does not increase the robustness to SAEs. Pixel perturbations are more general, and do not capture the effects induced by SAES. Con- sequently, wished understand if could get the best of both worlds i.e. robustness against both pixel perturbations and SAEs. We report the results of this experiment in Appendix A.
A.2 HYBRID TUNING
To understand if we are able to obtain robustness against both pixel perturbations and SAEs, we repeated our experiments from earlier sections ($ 5.2); this time, we tuned the model robust SAEs (obtained after 6000 iterations) using pixel perturbations for 6000 iterations denoted Tuned (Pixel); we also tuned the model robust pixel perturbations (obtained after 6000 iterations) using SAEs for 6000 iterations denoted Tuned (SAF).
Model
Tuned (SAB)
Tuned (Pixel)
Model
Tuned (SAE)
Tuned (Pixel)
recall
93.2
91.7
recall
29.5
90.6
mAP
73.8
65.7
mAP
7.65
75.15
(a) Validation set used: SAEs
(b) Validation set used: pixel perturbations
Table 5: Performance of dual tuning (i.e. tuning a model robust to SAEs with pixel perturbations, and tuning a model robust to pixel perturbations with SAEs). The generality of pixel perturbations masked by the specificity of SAEs (Table column 1).
We validated both the generated models against (a) SAEs, and (b) pixel perturbations. Our results are presented in Table 5. While the order of tuning did not greatly impact robustness in the face of SAEs, it had direct impact for pixel perturbations (as evident from Table 5b). We explain the observation in Table 5b using manifold learning. The detector is assumed to generalize over unobserved data on the data manifold. Due to the limited amount of sampling, the detector fails to generalize SAEs lying these uncovered regions; learning them aids generalization. Pixel perturbations, however, do not lie on the data manifold (due to its larger degree freedom). Tuning the model using pixel perturbation is analogous to learning thickened manifold (with the same uncovered regions as before). Now, we tune this model, the detector learns uncovered regions in thickened (and incorrect) manifold; this impacts generalization, further making it susceptible to
13

---

Under review as a conference paper at ICLR 2020
pixel perturbations. Another intuition could be based the generality of pixel perturbations; tuning first on such general perturbations impacts generalization (Tsipras et al., 2018). When such a model is tuned on SAEs, we gain generalization at the expense of robustmess.
14


Reviews:
[{"review_id": "Skg7z5hFYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to generate \"semantic adversarial examples\" (SAEs) by applying small perturbations in the semantic space, by contrast to more standard adversarial examples that apply perturbations to the image intensities. The concept of SAE has already  been considered in a few papers (where for example the color distribution of the images are changed). \n\nWhat the paper proposes is to consider small perturbations of the parameters of a scene. For example, a car can be slighted rotated in 3D, or its color changed, without changing its 2D bounding box. If the perturbation is differentiable, and with a differentiable rendering, it is possible to generate SAEs with optimization algorithms that generalize algorithms developed for \"classical\" adversarial examples (FGSM and PGD).\nIn practice the perturbations are performed by using the method from Yao et al. \"3D-Aware Scene Manipulation via Inverse Graphics\" (NIPS'18).\n\nThe paper shows that their SAEs disturbed an object detection method (SqueezeDet), and that training SqueezeDet with SAEs improves its robustness to their SAEs.\n\nI found the paper interesting, however I am not convinced by the general concept:\n* While the method from Yao et al is supposed to work on real images, it requires to detect and estimate the pose and the shape of 3D objects, which is still challenging to do on real images. This is probably why the paper considers only synthetic images (VKITTI), to be able to generate a large number of SAEs.  \n\n* The generated images are also valid images, in the sense that for a slightly different scene, they would have been generated by the image renderer used for VKITTI. It sounds more like what the authors have is a method that can generate scenes to improve the performance of an object detector when trained on synthetic images, rather than a method  to be more robust to malicious attacks. This is very interesting, however the paper is not branded toward this  application.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HylO_-G19B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review: This paper focuses on generating adversarial perturbation in semantic space. The main contribution of this paper is to propose a general method to generate semantic adversarial examples by using advances in differential rendering and inverse graphics. \n\nPros:\n1. Generally, the presentation is clear and easy to follow.\n2. A general way to transform any pixel-attack algorithm to its “semantic version” is novel.\n\nCons:\n1. There are already a lot of ways to make adversarial examples even semantic adversarial examples. It’s not enough to just propose a new way to make adversarial examples. I think this paper would be more compelling if proposed SAEs have some special property (e.g, easy to take effect in the real world or stronger robustness against defense strategy).\n2. In section 5.3, the authors show that data augmentation using SAEs increase the robustness to SAEs, but pixel perturbation AEs do not. This result is of little value. It is obvious that data augmentation using SAEs can increase more robustness to SAEs or pixel-perturbation AEs can increase more robustness to pixel-perturbation AEs. The authors are suggested to compare changes in general robustness caused by two types of data augmentations. For example, compare minimum adversarial distortion.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgq2bPJ5S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of semantic adversarial attacks with a differentiable de-rendering and rendering pipeline. More specifically, this paper proposed a variant of FGSM (Goodfellow et al. 2015) and PGD (Madry et al. 2017) by extending the traditional Lp-bounded adversarial attacks in the rich semantic space. It considered a list of semantic parameters including color, weather, foliage, rotation, transformation, and object shape. To facilitate back-propagation and improve the quality of rendering, this paper re-implemented the differentiable equivalents of several image manipulation operations based on the previous work (Yao et al. 2018). For experimental evaluations, this paper selected the object detector SqueezeDet (We et al. 2016) as the target model for attack on the virtual KITTI dataset. Experiments demonstrated that the generated semantic adversarial examples (SAEs) can attack the SqueezeDet (see Table 1 and Table 2). By re-training with augmented data by the proposed method, the robustness of SqueezeDet (see Table 3) can be further improved.\n\nOverall, this is an okay paper with incremental technical novelty and clear presentation. Reviewer has several concerns regarding the experiments.\n\n(1) This paper only conducts experiments on virtual KITTI dataset, a synthetic benchmark for object detection and semantic segmentation. In general, studying the adversarial examples in the synthetic domain seems not a significant contribution. Reviewer would like to know the performance on the real dataset such as Cityscape (used in Yao et al. 2018) and other challenging indoor datasets such as ADE20K. At least, reviewer would like to know whether re-training on adversarial examples help to improve the performance on real dataset?\n\n(2) The conclusion is largely based on the 1547 semantic adversarial examples generated, while there are more than 4K synthetic images in the dataset. This seems contradicts against the flexibility of generating semantic adversarial examples described in the paper (e.g., single parameter modifications). Reviewer suspects the proposed differentiable rendering pipeline is not very effective so that generating SAEs requires quite a bit exhaustive search over the parameter space. Please comment on the average running time for generating a semantic adversarial example. How does that compare to generating a pixel-based adversarial example?\n\n(3) While several different quantitative analyses have been conducted, this paper only provides two examples as the qualitative result (see Figure 2). It would be more convincing if this paper provides more such examples in the appendix. In addition, ablation studies on semantic parameters are currently missing. Furthermore, reviewer wonders if it is possible to report the FID score and make sure the generated adversarial examples have the same distribution as ground-truth images.\n\n(4) SqueezeDet is the only model used in the paper. Please also consider other models and report the attack performance and transferability. In a high-level, reviewer would like to know whether the proposed differentiable rendering method generalizes to other tasks including semantic segmentation and depth prediction.\n\n(5) The following paper is related (see Figure 5 of MeshAdv paper), but not even mentioned here. Reviewer would like to see the comparison between the proposed method and the MeshAdv baseline.\n\n-- MeshAdv: Adversarial Meshes for Visual Recognition. Xiao et al. In CVPR 2019.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a way for generating adversarial examples using discrete perturbations, i.e., perturbations that, unlike pixel ones, carry some semantics. Thus, in order to do so, they assume the existence of an inverse graphics framework. Results are conducted in the VKITTI dataset. Overall, the main serious concern expressed by the reviewers has to do with the general applicability of this method, since it requires an inverse graphics framework, which all-in-all is not a trivial task, so it is not clear how such a method would scale to more “real” datasets. A secondary concern has to do with the fact that the proposed method seems to be mostly a way to perform semantic data-augmentation rather than a way to avoid malicious attacks. In the latter case, we would want to know something about the generality of this method (e.g., what happens a model is trained for this attacks but then a more pixel-based attack is applied). As such, I do not believe that this submission is ready for publication at ICLR. However, the technique is an interesting idea it would be interesting if a later submission would provide empirical evidence about/investigate the generality of this idea. 
Decision: Reject