Generating Semantic Adversarial Examples with Differentiable Rendering

1 INTRODUCTION . Machine learning ( ML ) techniques , especially Deep Neural Networks ( DNNs ) , have been successful in several domains , such as finance and healthcare . However , several test-time ( Biggio et al. , 2013 ; Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Kurakin et al. , 2016 ) and training-time ( Jagielski et al. , 2018 ; Shafahi et al. , 2018 ) attacks have made their adoption in high-assurance applications , such as autonomous driving and security , problematic . ML techniques , such as generative models , have also been used for nefarious purposes such as generating “ deepfakes ” ( Liu et al. , 2017 ; Zhu et al. , 2017 ) . Our focus in this paper is on test-time attacks in which an adversary generates a slightly perturbed sample to fool a classifier or an object-detector . Let X be the sample space and Y be the space of labels . A classifier F is a function from X to Y . Given a sample x ∈ X , most attacks for constructing adversarial examples find a perturbation δ with a small norm ( typical norms that are used are l∞ , l0 , and l2 ) such that x+δ has a different label than x , i.e . F ( x ) 6= F ( x+δ ) . In this paper we consider the problem of generating semantic adversarial examples ( SAEs ) ( Hosseini & Poovendran , 2018 ; Joshi et al. , 2019 ; Qiu et al. , 2019 ; Dreossi et al. , 2018b ) . In these examples , there is a richer set of transformations T that capture semantically-meaningful changes to inputs to the ML model . We assume a norm on T ( this norm is induced by various parameters corresponding to the transformations , such as angle of rotation and size of the translation ) . In our universe , an adversary is given a sample x and wishes to find a transformation parameterized by θ ∈ Θ with small norm such that F ( τ ( x , θ ) ) 6= F ( x ) ( we consider untargeted attacks , but our ideas extend to targeted attacks as well ) 1 . SAEs can also be viewed as outcomes of perturbations in a “ rich ” semantic feature space ( e.g. , , texture of the image ) rather than just the concrete feature space ( e.g. , , pixels ) . Consequently , SAEs are physically realizable , and it is easy to understand how the changes in semantics results in an adversarial example . SAEs have been considered in the literature ( Xiao et al. , 2018 ; Dreossi et al. , 2018b ; Huang et al. , 2019 ) , but prior works typically consider a small set of fixed transformations ( e.g. , rotation and translation , or modifying a single object ’ s texture ) . Our goal is to flexibly support a 1τ is defined in § 3.2. richer set of transformations implemented in a state of the art renderer ( e.g. , changing the background of the image , weather conditions , or the time of day ) , as we specify in detail in § 2 . There is evidence that SAEs can help with domain adaptation ( Volpi et al. , 2018 ) or making the control loop more robust ( Dreossi et al. , 2018b ) , further motivating our approach . We also plan to open source our code to advance research in this space . To summarize , the main contributions of this paper are the following : • We present a new class of test-time attacks in the form of SAEs . We demonstrate how to generate SAEs that support a rich set of transformations ( refer § 3 ) using an inverse graphics framework ( refer § 2 ) . Specifically , we show how one can systematically take techniques to perform attacks in the pixel space such as FGSM ( Goodfellow et al. , 2015 ) and PGD ( Madry et al. , 2017 ) and transform them to their semantic counterparts . • We evaluate the generated SAEs on the popular object detector SqueezeDet ( Wu et al. , 2016 ) . By correctly choosing the semantic parameters , SAEs degrade performance ( characterized by the mean average precision or mAP ) by 28 percentage points ( refer § 5.1 ) . • We show that by augmenting the dataset using SAEs , we can boost the robustness of SqueezeDet ( characterized by mAP ) by up to 15 percentage points ( refer § 5.2 ) . • We show that SAEs generated using SqueezeDet do not transfer to YOLOv3 ( Redmon & Farhadi , 2018 ) , indicating a tight coupling between the model architecture and SAE generation ( refer § 5.3 ) . • While augmentation with SAEs improves robustness against SAEs , augmentation using traditional pixel-based perturbations does not produce the same effect ( refer § A.1 ) . 2 RELATED WORK . Adversarial Examples and Robustness : There is extensive research for generating adversarial examples in the pixel space ; we henceforth refer to these as pixel-perturbations . Goodfellow et al . ( 2015 ) propose the fast gradient sign method ( FGSM ) where inputs are modified in the direction of the gradients of the loss function with respect to input , causing a variety of models to misclassify their inputs . Madry et al . ( 2017 ) generalize this approach and propose the projected gradient descent ( PGD ) approach working using the same intuition . While these approaches suggest modifications to the raw pixel values , other methods of generating adversarial examples exist . Athalye et al . ( 2017 ) introduce an approach to generate 3D adversarial examples ( over a chosen distribution of transformations ) . Engstrom et al . ( 2019 ) observe that modifying the spatial orientation of images results in misclassifications . Similarly , Geirhos et al . ( 2018 ) discovered that certain models are biased towards textural cues . To improve robustness , current approaches include adversarial training ( Madry et al. , 2017 ) , smoothing-based approaches ( Cohen et al. , 2019 ; Lécuyer et al. , 2018 ) , or through specific regularization ( Raghunathan et al. , 2018 ) . An alternative approach , utilizing some notion of semantics , is advocated in the work of Guo et al . ( 2017 ) . The authors augment the training set with transformed versions of training images , utilizing basic image transformations ( e.g. , , scale and re-cropping ) and total variance minimization , and demonstrate an improvement in robustness . Dreossi et al . ( 2018a ) improve the robustness of SqueezeDet ( Wu et al. , 2016 ) through counterexample guided data augmentation ; these counterexamples are synthetically generated by sampling from a space of transformations and applying them to original training images . Inverse Graphics : The process of finding 3D scene parameters ( geometric , textural , lighting , etc . ) given images is referred to as inverse graphics ( Baumgart , 1974 ) . There is a history of using gradients to solve this problem ( Blanz & Vetter , 2002 ; Shacked & Lischinski , 2001 ; Barron & Malik , 2015 ) . Kulkarni et al . ( 2015 ) propose a model that learns interpretable representations of images ( similar to image semantics ) , and show how these interpretations can be modified to produce changes in the input space . Pipelines for general differential rendering were proposed by Loper & Black ( 2014 ) and Kato et al . ( 2018 ) . Li et al . ( 2018 ) design a general-purpose differentiable ray tracer ; gradients can be computed with respect to arbitrary semantic parameters such as camera pose , scene geometry , materials , and lighting parameters . Yao et al . ( 2018 ) propose a pipeline that , through de-rendering obtains various forms of semantics , geometry , texture , and appearance , which can be rendered using a generative model . Semantic Adversarial Examples ( SAEs ) : The notion of SAEs has been proposed in the literature in specific contexts . For example , in the NLP domain , Lei et al . ( 2018 ) propose a gradient guided greedy algorithm to make semantic changes to text documents . Sharif et al . ( 2016 ) propose attacks against facial recognition systems that are both inconspicuous and realizable . Similarly , Evtimov et al . ( 2017 ) obtain patches that can be added to road signs , rendering object detectors useless . In both these works , however , the modifications to semantics are not found by searching the space of semantic modifications ( and found instead by realizing changes in pixel intensities ) ; such approaches will fail to generalize for other domains . Dreossi et al . ( 2018b ) present an approach to generate SAEs based on systematic sampling of the semantic space coupled with verification techniques for a closed-loop cyber-physical system containing the ML model , where they treat the ML model as a black box . However , this sampling-based “ black box ” approach faces scalability issues when dealing with high-dimensional spaces and also fails to exploit structure of the ML model to perform a targeted search for SAEs . Xiao et al . ( 2019 ) utilize a differentiable rendering framework to introduce changes in shape and texture that are capable of fooling a variety of ML models for various tasks . However , our approach takes into account advances in differentiable inverse graphics permitting us to change a richer set of semantic parameters . An interesting contribution by Qiu et al . ( 2019 ) suggests how a generative model can be used to introduce semantic modifications . However , generative models are notoriously difficult to train and operate , making them unusuable for a wide variety of tasks . Similar issues occur in the work by Joshi et al . ( 2019 ) . 3 SEMANTIC ADVERSARIAL LEARNING . Consider a space Z of the form X × Y , where X is the sample space and Y is the set of labels . From here on we will assume that X = < n . Let H be a hypothesis space ( e.g. , , weights of a DNN ) . We assume a loss function ` : H × Z 7→ R so that given a hypothesis w ∈ H and a labeled data point ( x , y ) ∈ Z , the loss is ` ( w , x , y ) . The output of the learning algorithm is a classifier , which is a function from < n to Y . To emphasize that a classifier depends on a hypothesis w ∈ H , which is output of the learning algorithm , we will denote it as Fw ( if w is clear from the context , we will sometimes simply write F ) . 3.1 TRADITIONAL ADVERSARIAL EXAMPLES . We will focus our discussion on untargeted attacks , but our discussion also applies to targeted attacks . An adversary A ’ s goal is to take any input vector x ∈ < n and produce a minimally altered version of x , an adversarial example denoted by A ( x ) , that has the property of being misclassified by a classifier F : < n → Y . The adversary wishes to solve the following optimization problem : minδ∈ < n µ ( δ ) such that F ( x+ δ ) 6= F ( x ) The various terms in the formulation are : µ is a norm on < n ; µ can be l∞ , l0 , l1 , or lp ( p ≥ 2 ) . While not representative , these norms are commonly used as an approximation of a human ’ s visual perception ( Sen et al. , 2019 ) . If δ is the solution of the optimization problem given above , then the adversarial example A ( x ) = x+δ . FGSM . The fast gradient sign method ( FGSM ) ( Goodfellow et al. , 2015 ) was one of the first untargeted attacks developed in literature . The adversary crafts an adversarial example for a given legitimate sample x by computing ( and then adding ) the following perturbation : δ = ε sign ( ∇xLF ( x ) ) ( 1 ) The function LF ( x ) is a shorthand for ` ( w , x , l ( x ) ) , where w is the hypothesis corresponding to the classifier F , x is the data point and l ( x ) is the true label of x ( essentially we evaluate the loss function at the hypothesis corresponding to the classifier ) . The gradient of the function LF is computed with respect to x using sample x and label y = l ( x ) as inputs . Note that ∇xLF ( x ) is an n-dimensional vector and sign ( ∇LF ( x ) ) is a n-dimensional vector whose ith element is the sign of the∇xLF ( x ) [ i ] . The value of the input variation parameter ε factoring the sign matrix controls the perturbation ’ s amplitude . Increasing its value increases the likelihood of A ( x ) being misclassified by the classifier F but also makes adversarial examples easier to “ detect ” by humans . The key idea is that FGSM takes a step in the direction of the gradient of the loss function with respect to the input , thus attempting to maximize the loss function using its first-order approximation . Recall that stochastic gradient descent ( SGD ) takes a step in the direction that is on expectation opposite to the gradient of the loss function because it is trying to minimize the loss function . PGD . In Projected Gradient Descent ( PGD ) ( Madry et al. , 2017 ) , we find a perturbation in an iterative manner . The PGD attack can be thought of an iterative version of FGSM . Assume that we are using the lp norm . Assume x0 is the original sample x. xk+1 = ΠBp ( x , ε ) ( xk +ε sign ( ∇xLF ( x ) ) ) ( 2 ) The operator ΠBp ( x , ε ) ( y ) is the projection operator , i.e . it takes as input a point y and outputs the closest point in the ε-ball ( using the lp-norm ) around x . The iteration stops after a certain number of steps ( the exact number of steps is a hyperparameter ) .

Summary: This paper studies the problem of semantic adversarial attacks with a differentiable de-rendering and rendering pipeline. More specifically, this paper proposed a variant of FGSM (Goodfellow et al. 2015) and PGD (Madry et al. 2017) by extending the traditional Lp-bounded adversarial attacks in the rich semantic space. It considered a list of semantic parameters including color, weather, foliage, rotation, transformation, and object shape. To facilitate back-propagation and improve the quality of rendering, this paper re-implemented the differentiable equivalents of several image manipulation operations based on the previous work (Yao et al. 2018). For experimental evaluations, this paper selected the object detector SqueezeDet (We et al. 2016) as the target model for attack on the virtual KITTI dataset. Experiments demonstrated that the generated semantic adversarial examples (SAEs) can attack the SqueezeDet (see Table 1 and Table 2). By re-training with augmented data by the proposed method, the robustness of SqueezeDet (see Table 3) can be further improved.