Beyond Pixel Norm-Balls: Parametric Adversaries using an Analytically Differentiable Renderer

Many machine learning image classifiers are vulnerable to adversarial attacks , inputs with perturbations designed to intentionally trigger misclassification . Current adversarial methods directly alter pixel colors and evaluate against pixel norm-balls : pixel perturbations smaller than a specified magnitude , according to a measurement norm . This evaluation , however , has limited practical utility since perturbations in the pixel space do not correspond to underlying real-world phenomena of image formation that lead to them and has no security motivation attached . Pixels in natural images are measurements of light that has interacted with the geometry of a physical scene . As such , we propose a novel evaluation measure , parametric normballs , by directly perturbing physical parameters that underly image formation . One enabling contribution we present is a physically-based differentiable renderer that allows us to propagate pixel gradients to the parametric space of lighting and geometry . Our approach enables physically-based adversarial attacks , and our differentiable renderer leverages models from the interactive rendering literature to balance the performance and accuracy trade-offs necessary for a memory-efficient and scalable adversarial data augmentation workflow . 1 INTRODUCTION . Research in adversarial examples continues to contribute to the development of robust ( semi- ) supervised learning ( Miyato et al. , 2018 ) , data augmentation ( Goodfellow et al. , 2015 ; Sun et al. , 2018 ) , and machine learning understanding ( Kanbak et al. , 2018 ) . One important caveat of the approach pursued by much of the literature in adversarial machine learning , as discussed recently ( Goodfellow , 2018 ; Gilmer et al. , 2018 ) , is the reliance on overly simplified attack metrics : namely , the use of pixel value differences between an adversary and an input image , also referred to as the pixel norm-balls . The pixel norm-balls game considers pixel perturbations of norm-constrained magnitude ( Goodfellow et al. , 2015 ) , and is used to develop adversarial attackers , defenders and training strategies . The pixel norm-ball game is attractive from a research perspective due to its simplicity and well-posedness : no knowledge of image formation is required and any arbitrary pixel perturbation remains eligible ( so long as it is “ small ” , in the perceptual sense ) . Although the pixel norm-ball is useful for research purposes , it only captures limited real-world security scenarios . Despite the ability to devise effective adversarial methods through the direct employment of optimizations using the pixel norm-balls measure , the pixel manipulations they promote are divorced from the types of variations present in the real world , limiting their usefulness “ in the wild ” . Moreover , this methodology leads to defenders that are only effective when defending against unrealistic images/attacks , not generalizing outside of the space constrained by pixel norm-balls . In order to consider conditions that enable adversarial attacks in the real world , we advocate for a new measurement norm that is rooted in the physical processes that underly realistic image synthesis , moving away from overly simplified metrics , e.g. , pixel norm-balls . Our proposed solution – parametric norm-balls – rely on perturbations of physical parameters of a synthetic image formation model , instead of pixel color perturbations ( Figure 2 ) . To achieve this , we use a physically-based differentiable renderer which allows us to perturb the underlying parameters of the image formation process . Since these parameters indirectly control pixel colors , perturbations in this parametric space implicitly span the space of natural images . We will demonstrate two advantages that fall from considering perturbations in this parametric space : ( 1 ) they enable adversarial approaches that more readily apply to real-world applications , and ( 2 ) they permit the use of much more significant perturbations ( compared to pixel norms ) , without invalidating the realism of the resulting image ( Figure 1 ) . We validate that parametric norm-balls game playing is critical for a variety of important adversarial tasks , such as building defenders robust to perturbations that can occur naturally in the real world . We perform perturbations in the underlying image formation parameter space using a novel physicallybased differentiable renderer . Our renderer analytically computes the derivatives of pixel color with respect to these physical parameters , allowing us to extend traditional pixel norm-balls to physicallyvalid parametric norm-balls . Notably , we demonstrate perturbations on an environment ’ s lighting and on the shape of the 3D geometry it shades . Our differentiable renderer achieves state-of-the-art performance in speed and scalability ( Section 3 ) and is fast enough for rendered adversarial data augmentation ( Section 5 ) : training augmented with adversarial images generated with a renderer . Existing differentiable renders are slow and do not scalable to the volume of high-quality , highresolutions images needed to make adversarial data augmentation tractable ( Section 2 ) . Given our analytically-differentiable renderer ( Section 3 ) , we are able to demonstrate the efficacy of parametric space perturbations for generating adversarial examples . These adversaries are based on a substantially different phenomenology than their pixel norm-balls counterparts ( Section 4 ) . Ours is among the first steps towards the deployment of rendered adversarial data augmentation in real-world applications : we train a classifier with computer-generated adversarial images , evaluating the performance of the training against real photographs ( i.e. , captured using cameras ; Section 5 ) . We test on real photos to show the parametric adversarial data augmentation increases the classifier ’ s robustness to “ deformations ” happened in the real world . Our evaluation differs from the majority of existing literature which evaluates against computer-generated adversarial images , since our parametric space perturbation is no-longer a wholly idealized representation of the image formation model but , instead , modeled against of theory of realistic image generation . 2 RELATED WORK . Our work is built upon the fact that simulated or rendered images can participate in computer vision and machine learning on real-world tasks . Many previous works use rendered ( simulated ) data to train deep networks , and those networks can be deployed to real-world or even outperform the state-of-the-art networks trained on real photos ( Movshovitz-Attias et al. , 2016 ; Chen et al. , 2016 ; Varol et al. , 2017 ; Su et al. , 2015 ; Johnson-Roberson et al. , 2017 ; Veeravasarapu et al. , 2017b ; Sadeghi & Levine , 2016 ; James & Johns , 2016 ) . For instance , Veeravasarapu et al . ( 2017a ) show that training with 10 % real-world data and 90 % simulation data can reach the level of training with full real data . Tremblay et al . ( 2018 ) even demonstrate that the network trained on synthetic data yields a better performance than using real data alone . As rendering can cheaply provide a theoretically infinite supply of annotated input data , it can generate data which is orders of magnitude larger than existing datasets . This emerging trend of training on synthetic data provides an exciting direction for future machine learning development . Our work complements these works . We demonstrate the utility of rendering can be used to study the potential danger lurking in misclassification due to subtle changes to geometry and lighting . This provides a future direction of combining with synthetic data generation pipelines to perform physically based adversarial training on synthetic data . Adversarial Examples Szegedy et al . ( 2014 ) expose the vulnerability of modern deep neural nets using purposefully-manipulated images with human-imperceptible misclassification-inducing noise . Goodfellow et al . ( 2015 ) introduce a fast method to harness adversarial examples , leading to the idea of pixel norm-balls for evaluating adversarial attackers/defenders . Since then , many significant developments in adversarial techniques have been proposed ( Akhtar & Mian , 2018 ; Szegedy et al. , 2014 ; Rozsa et al. , 2016 ; Kurakin et al. , 2017 ; Moosavi Dezfooli et al. , 2016 ; Dong et al. , 2018 ; Papernot et al. , 2017 ; Moosavi-Dezfooli et al. , 2017 ; Chen et al. , 2017 ; Su et al. , 2017 ) . Our work extends this progression in constructing adversarial examples , a problem that lies at the foundation of adversarial machine learning . Kurakin et al . ( 2016 ) study the transferability of attacks to the physical world by printing then photographing adversarial images . Athalye et al . ( 2017 ) and Eykholt et al . ( 2018 ) propose extensions to non-planar ( yet , still fixed ) geometry and multiple viewing angles . These works still rely fundamentally on the direct pixel or texture manipulation on physical objects . Since these methods assume independence between pixels in the image or texture space they remain variants of pixel norm-balls . This leads to unrealistic attack images that can not model real-world scenarios ( Goodfellow , 2018 ; Hendrycks & Dietterich , 2018 ; Gilmer et al. , 2018 ) . Zeng et al . ( 2017 ) generate adversarial examples by altering physical parameters using a rendering network ( Liu et al. , 2017 ) trained to approximate the physics of realistic image formation . This data-driven approach leads to an image formation model biased towards the rendering style present in the training data . This method also relies on differentiation through the rendering network in order to compute adversaries , which requires high-quality training on a large amount of data . Even with perfect training , in their reported performance , it still requires 12 minutes on average to find new adversaries , we only take a few seconds Section 4.1 . Our approach is based on a differentiable physically-based renderer that directly ( and , so , more convincingly ) models the image formation process , allowing us to alter physical parameters – like geometry and lighting – and compute derivatives ( and adversarial examples ) much more rapidly compared to the ( Zeng et al. , 2017 ) . We summarize the difference between our approach and the previous non-image adversarial attacks in Table 1 . Differentiable Renderer Applying parametric norm-balls requires that we differentiate the image formation model with respect to the physical parameters of the image formation model . Modern realistic computer graphics models do not expose facilities to directly accommodate the computation of derivatives or automatic differentiation of pixel colors with respect to geometry and lighting variables . A physically-based differentiable renderer is fundamental to computing derivative of pixel colors with respect to scene parameters and can benefit machine learning in several ways , including promoting the development of novel network architectures ( Liu et al. , 2017 ) , in computing adversarial examples ( Athalye et al. , 2017 ; Zeng et al. , 2017 ) , and in generalizing neural style transfer to a 3D context ( Kato et al. , 2018 ; Liu et al. , 2018 ) . Recently , various techniques have been proposed to obtain these derivatives : Wu et al . ( 2017 ) ; Liu et al . ( 2017 ) ; Eslami et al . ( 2016 ) use neural networks to learn the image formation process provided a large amount of input/output pairs . This introduces unnecessary bias in favor of the training data distribution , leading to inaccurate derivatives due to imperfect learning . Kato et al . ( 2018 ) propose a differentiable renderer based on a simplified image formation model and an underlying linear approximation . Their approach requires no training and is unbiased , but their approximation of the image formation and the derivatives introduce more errors . Loper & Black ( 2014 ) ; Genova et al . ( 2018 ) use automatic differentiation to build fully differentiable renderers . These renderers , however , are expensive to evaluate , requiring orders of magnitude more computation and much larger memory footprints compared to our method . Our novel differentiable renderer overcomes these limitations by efficiently computing analytical derivatives of a physically-based image formation model . The key idea is that the non-differentiable visibility change can be ignored when considering infinitesimal perturbations . We model image variations by changing geometry and realistic lighting conditions in an analytically differentiable manner , relying on an accurate model of diffuse image formation that extend spherical harmonicsbased shading methods ( Appendix C ) . Our analytic derivatives are efficient to evaluate , have scalable memory consumption , are unbiased , and are accurate by construction ( Table 2 ) . Our renderer explicitly models the physics of the image formation processes , and so the images it generates are realistic enough to illicit correct classifications from networks trained on real-world photographs .

Summary: This work presents a method to generate adversary examples capable of fooling a neural network classifier. Szegedy et al. (2013) were the first to expose the weakness of neural networks against adversarial attacks, by adding a human-imperceptible noise to images to induce misclassification. Since then, several works tackled this problem by modifying the image directly in the pixel space: the norm-balls convention. The authors argue that this leads to non-realistic attacks and that a network would not benefit from training with these adversarial images when performing in the real world. Their solution and contributions are parametric norm-balls: unlike state-of-the-art methods, they perform perturbations in the image formation space, namely the geometry and the lighting, which are indeed perturbations that could happen in real life. For that, they defined a differentiable renderer by making some assumptions to simplify its expression compared to solving a light transport equation. The main simplifications are the direct illumination to gain computation efficiency and the distant illumination and diffuse material assumptions to represent lighting in terms of spherical harmonics as in Ramamoorthi et al. (2001), which require only 9 parameters to approximate lighting. This allows them to analytically derivate their loss function according to the geometry and lighting and therefore generate their adversary examples via gradient descent. They show that their adversary images generalize to other classifiers than the one used (ResNet). They then show that injecting these images into the training set increase the robustness of WideResNet against real attacks. These real attack images were taken by the authors in a laboratory with varying illumination.