Finding Physical Adversarial Examples for Autonomous Driving with Fast and Differentiable Image Compositing

1 INTRODUCTION . Computer vision has made revolutionary advances in recent years by leveraging a combination of deep neural network architectures with abundant high-quality perceptual data . One of the transformative applications of computational perception is autonomous driving , with autonomous cars and trucks already being evaluated for use in geofenced settings , and partial autonomy , such as highway assistance , leveraging state-of-the-art perception embedded in vehicles available to consumers . However , a history of tragic crashes involving autonomous driving , most notably Tesla ( Thorbecke , 2020 ) and Uber ( Hawkins , 2019 ) reveals that modern perceptual architectures still have some limitations even in non-adversarial driving environments . In addition , and more concerning , is the increasing abundance of evidence that state-of-the-art deep neural networks used in perception tasks are highly vulnerable to adversarial perturbations , or imperceptible noise that is added to an input image and deliberately designed to cause misclassification ( Goodfellow et al. , 2014 ; Yuan et al. , 2019 ; Modas et al. , 2020 ) . Furthermore , several lines of work consider specifically physical adversarial examples which modify the scene being captured by a camera , rather than the image ( Kurakin et al. , 2016 ; Eykholt et al. , 2018 ; Sitawarin et al. , 2018 ; Dutta , 2018 ; Duan et al. , 2020 ) . Despite this body of evidence demonstrating vulnerabilities in deep neural network perceptual architectures , it is nevertheless not evident that such vulnerabilities are consequential in realistic autonomous driving , even if primarily using cameras for perception . First , most such attacks involve independent perturbations to a given input image . Autonomous driving is a dynamical system , so that a fixed adversarial perturbation to a scene is perceived through a series of distinct , but highly interdependent perspectives . Second , self-driving is a complex system that maps perceptual inputs to control outputs . Consequently , even if we succeed in causing the control outputs to deviate from normal , the vehicle will now perceive a sequence of frames that is different from those encountered on its normal path , and typically deploy self-correcting behavior in response . For example , if the vehicle is driving straight and then begins swerving towards the opposite lane , its own perception will inform the control that it ’ s going in the wrong direction , and the controller will steer it back on course . To address these limitations , Bayesian Optimization ( BO ) ( Archetti and Candelieri , 2019 ) was recently proposed as a way to design physical adversarial examples ( 2 black rectangles on road pavement ) in Carla autonomous driving simulations ( Dosovitskiy et al. , 2017 ) against end-to-end autonomous driving architectures ( Boloor et al. , 2020 ) . The key challenge with this approach , however , is that attack design must execute actual experiments ( e.g. , simulations , or actual driving ) for a larger number of iterations ( 1000 in the work above ) , making it impractical for large-scale or physical driving evaluation . Furthermore , it is not clear how well BO scales as we increase the complexity of the adversarial space beyond 2 black rectangles . We propose a highly scalable framework for designing physically realizable adversarial examples against end-to-end autonomous driving architectures . Our framework is illustrated in Figure 1 , and develops a differentiable pipeline for digitally approximating driving scenarios . The proposed approximation makes use of image compositing , learning homography and color mappings from a birds-eye view of embedded adversarial examples to projections of these in images based on actual driving frames , and sampling sequences of actual frames with small added random noise to control to ensure adequate sampling of possible perspectives . The entire process can then be fed into automatic differentiators to obtain adversarial examples that maximize a car ’ s deviation from its normal sequence of controls ( e.g. , steering angle ) for a target driving scenario . We evaluate the proposed framework using Carla simulations in comparison with the state-of-the-art BO method . Our experiments show that the resulting attacks are significantly stronger , with effects on induced deviations and road infractions often considerably outperforming BO , at a small fraction of actual driving runs required for training . Furthermore , we show that our approach yields attacks that are robust to unforeseen variations in weather and visibility . Related Work : Attacks on deep neural networks for computer vision tasks has been a subject of extensive prior research ( Goodfellow et al. , 2014 ; Yuan et al. , 2019 ; Modas et al. , 2020 ; Vorobeychik and Kantarcioglu , 2018 ) . The most common variation introduces imperceptible noise to pixels of an image in order to induce error in predictions , such as misclassification of the image or failure to detect an object in it . A more recent line of research has investigated physical adversarial examples ( Kurakin et al. , 2016 ; Athalye et al. , 2017 ; Eykholt et al. , 2018 ; Sitawarin et al. , 2018 ; Dutta , 2018 ; Duan et al. , 2020 ) , where the explicit goal is to implement these in the physical scene , so that the images of the scene subsequently captured by the camera and fed into a deep neural network result in a prediction error . In a related effort , Liu et al . ( 2019 ) developed a differentiable renderer that allows the attacker to devise higher-level perturbations of an image scene , such as geometry and lighting , through a differentiable renderer . However , most of these approaches attack a fixed input scene , whereas autonomous driving is a complex dynamical system . Several recent approaches investigate physical attacks on autonomous driving that attempt to account for the fact that a single object is modified and viewed through a series of frames ( Ackerman , 2019 ; Kong et al. , 2020 ; Boloor et al. , 2020 ) . However , these either still consider digital attacks , albeit restricted to a small area ( e.g. , replacing a road sign with a noisy road sign ) and do not consider a vehicles self-correcting behavior ( for example , ( Kong et al. , 2020 ) ) , or rely on many expensive driving experiments in order to identify adversarial patterns ( Boloor et al. , 2020 ) . 2 PROPOSED METHOD . Autonomous driving systems are equipped with decision algorithms that produce control signals for a vehicle , based on high-level instructions—such as a given route or destination—and inputs from cameras and other sensors that make continuous measurements of the vehicle ’ s physical environment . We assume the decision algorithm is in the form of a differentiable function—such as a neural network—that maps video frames from the camera , along with other inputs , to the control outputs . Given such a network or function , our goal is to determine if it is vulnerable to attack . Specifically , we seek to build a scalable and efficient method to find modifications that can be applied to the physical environment , and result in a stream of video frames which cause the control network to produce output signals that disrupt the vehicle ’ s operation , moving it away from the expected ideal trajectory . This task is challenging since the relationship between modifications to the physical environment and the network ’ s inputs is complex : the video frames correspond to images of the environment from a sequence of changing viewpoints , where the sequence itself depends on the network ’ s control outputs . The precise effect of any given modification can be determined only by actually driving the vehicle in the modified environment , or by using a high-quality simulator with a sophisticated physics engine . However , it is expensive to use actual driving or an expensive simulator when searching for the right modification , since neither process is differentiable with respect to parameters of the modification , and would require repeated trials with candidate modifications in every step of the search process . Instead , we propose a fast approximation to produce video frames for a given environment given a candidate modification that is differentiable with respect to parameters of the modification . Our approach requires a small number of initial calibration runs—of actual driving or a sophisticated simulator—after which , the search for optimal parameters can be carried out with end-to-end gradientbased optimization . Specifically , we consider the case when the modification takes the form of figures ( such as rectangles ) drawn on a restricted stretch of the road , and task the optimization with finding their optimal shape and color so as to maximize deviation from the controller ’ s trajectory prior to modification . We now describe our model for the physical modification , our approximate mapping to create video frames for a given modification , and our optimization approach based on this mapping . 2.1 PARAMETERIZED PHYSICAL MODIFICATION . We assume modifications are in the form of a collection of K figures ( e.g. , rectangles ) that will be painted on a flat surface in the environment ( e.g. , road ) . Let Φ = { xSk , xCk } Kk=1 denote the parameters of this modification , with xSk corresponding to the shape parameters , and x C k the RGB color , of the kth figure . These parameters are defined with respect to co-ordinates in some canonical—say , top-down—view of the surface . We let M ( nc ; xS ) denote a scalar-valued mask that represents whether a pixel at spatial location nc ∈ Z2 in the canonical view is within a figure with shape parameters xS . This function depends simply on the chosen geometry of the figures , and has value of 1 for pixels within the figure , 0 for those outside , and real values between 0 and 1 for those near the boundary ( representing partial occupancy on a discrete pixel grid to prevent aliasing artifacts ) . Since the spatial extents for different figures may overlap , we next account for occlusions by assuming that the lines will be painted in order . Accordingly , we define a series of visibility functions Vk ( nc ; Φ ) , each representing the visibility of the kth figure at pixel nc , after accounting for occlusions . We set the function for the last figure as VK ( nc ; Φ ) = M ( nc ; xSK ) , and for the other figures with k < K , Vk ( nc ; Φ ) = M ( nc ; x S k ) K∏ k′=k+1 ( 1− Vk′ ( nc ; Φ ) ) . ( 1 ) 2.2 APPROXIMATE FRAMES VIA COMPOSITING . The next step in our pipeline deals with generating the video inputs that the controller network is expected to receive from a modified environment for given parameter values Φ . These frames will represent views of the environment , including the surface with the painted figures , from a sequence of viewpoints as the car drives through the scene . Of course , the precise viewpoint sequence will depend on the trajectory of the car , which will depend on the control outputs from the network , that in turn depends on the frames . Instead of modeling the precise trajectory for every modification , we instead consider a small set of T representative trajectories , corresponding to those that the vehicle will follow when driven with small perturbations around control signals the network outputs , when operating in the unmodified environment . In our experiments , we use T = 4 trajectories : one from driving the car with the actual output control signals , and three from random noise added to these outputs . Given the fact that actual control is closed-loop , it is not evident that this simple approach would work ; however , our experiments below show that it is remarkably effective , despite its simplicity . This gives T sequences of video frames , one for each trajectory , where we assume each sequence contains F frames . We let Ĩtf ( n ) denote the f th “ clean ” image in the tth sequence , representing a view of the environment without any modifications . Here , n ∈ Z2 indexes pixel location within each image , and the intensity vector Ĩtf ( n ) ∈ R3 at each location corresponding to the recorded RGB values . These clean images can be obtained by driving the car—actually , or in simulation—in the original environment . For each frame in each sequence , we also determine a spatial mapping nc = Gtf ( n ) that maps pixel locations in the image to the canonical view . We model each Gtf ( n ) as a homography : the parameters of which can be determined by either using correspondences between each image and the canonical view of the surface—from calibration patterns rendered using the simulator , or from user input—or by calibrating the vehicle ’ s camera and making accurate measurements of ego-motion when the vehicle is being driven . Additionally , we also determine color mapping parameters Ctf ∈ R3×3 , btf ∈ R3 for each frame representing an approximate linear relationship between the RGB colors xC of the painted figures , and their colors as visible in each frame . These parameters are also determined through calibration . Given this set of clean frames and the geometric and color mapping parameters , we generate corresponding frames with views of the modified environment simply as : Itf ( n ; Φ ) = ( 1− K∑ k=1 Vk ( G t f ( n ) ; Φ ) ) Ĩtf ( n ) + K∑ k=1 Vk ( G t f ( n ) ; Φ ) ( Ctfx C k + b t f ) . ( 2 )

Summary: This paper proposes a scalable and efficient approach for finding adversarial physical modification to the video inputs of autonomous driving. Assuming the perturbations are in form of a collection of several rectangles, the model parameterizes the physical modifications. By simply ignoring the closed-loop of viewpoint sequence and frames, the model directly creating adversarial frames with compositing methods. Some approximated algorithms are used to ensure the model can be optimized by the gradient-based method. With the improvement above, the iteration speed of the model is greatly improved.