Finding Physical Adversarial Examples for Autonomous Driving with Fast and Differentiable Image Compositing

1 INTRODUCTION . Computer vision has made revolutionary advances in recent years by leveraging a combination of deep neural network architectures with abundant high-quality perceptual data . One of the transformative applications of computational perception is autonomous driving , with autonomous cars and trucks already being evaluated for use in geofenced settings , and partial autonomy , such as highway assistance , leveraging state-of-the-art perception embedded in vehicles available to consumers . However , a history of tragic crashes involving autonomous driving , most notably Tesla ( Thorbecke , 2020 ) and Uber ( Hawkins , 2019 ) reveals that modern perceptual architectures still have some limitations even in non-adversarial driving environments . In addition , and more concerning , is the increasing abundance of evidence that state-of-the-art deep neural networks used in perception tasks are highly vulnerable to adversarial perturbations , or imperceptible noise that is added to an input image and deliberately designed to cause misclassification ( Goodfellow et al. , 2014 ; Yuan et al. , 2019 ; Modas et al. , 2020 ) . Furthermore , several lines of work consider specifically physical adversarial examples which modify the scene being captured by a camera , rather than the image ( Kurakin et al. , 2016 ; Eykholt et al. , 2018 ; Sitawarin et al. , 2018 ; Dutta , 2018 ; Duan et al. , 2020 ) . Despite this body of evidence demonstrating vulnerabilities in deep neural network perceptual architectures , it is nevertheless not evident that such vulnerabilities are consequential in realistic autonomous driving , even if primarily using cameras for perception . First , most such attacks involve independent perturbations to a given input image . Autonomous driving is a dynamical system , so that a fixed adversarial perturbation to a scene is perceived through a series of distinct , but highly interdependent perspectives . Second , self-driving is a complex system that maps perceptual inputs to control outputs . Consequently , even if we succeed in causing the control outputs to deviate from normal , the vehicle will now perceive a sequence of frames that is different from those encountered on its normal path , and typically deploy self-correcting behavior in response . For example , if the vehicle is driving straight and then begins swerving towards the opposite lane , its own perception will inform the control that it ’ s going in the wrong direction , and the controller will steer it back on course . To address these limitations , Bayesian Optimization ( BO ) ( Archetti and Candelieri , 2019 ) was recently proposed as a way to design physical adversarial examples ( 2 black rectangles on road pavement ) in Carla autonomous driving simulations ( Dosovitskiy et al. , 2017 ) against end-to-end autonomous driving architectures ( Boloor et al. , 2020 ) . The key challenge with this approach , however , is that attack design must execute actual experiments ( e.g. , simulations , or actual driving ) for a larger number of iterations ( 1000 in the work above ) , making it impractical for large-scale or physical driving evaluation . Furthermore , it is not clear how well BO scales as we increase the complexity of the adversarial space beyond 2 black rectangles . We propose a highly scalable framework for designing physically realizable adversarial examples against end-to-end autonomous driving architectures . Our framework is illustrated in Figure 1 , and develops a differentiable pipeline for digitally approximating driving scenarios . The proposed approximation makes use of image compositing , learning homography and color mappings from a birds-eye view of embedded adversarial examples to projections of these in images based on actual driving frames , and sampling sequences of actual frames with small added random noise to control to ensure adequate sampling of possible perspectives . The entire process can then be fed into automatic differentiators to obtain adversarial examples that maximize a car ’ s deviation from its normal sequence of controls ( e.g. , steering angle ) for a target driving scenario . We evaluate the proposed framework using Carla simulations in comparison with the state-of-the-art BO method . Our experiments show that the resulting attacks are significantly stronger , with effects on induced deviations and road infractions often considerably outperforming BO , at a small fraction of actual driving runs required for training . Furthermore , we show that our approach yields attacks that are robust to unforeseen variations in weather and visibility . Related Work : Attacks on deep neural networks for computer vision tasks has been a subject of extensive prior research ( Goodfellow et al. , 2014 ; Yuan et al. , 2019 ; Modas et al. , 2020 ; Vorobeychik and Kantarcioglu , 2018 ) . The most common variation introduces imperceptible noise to pixels of an image in order to induce error in predictions , such as misclassification of the image or failure to detect an object in it . A more recent line of research has investigated physical adversarial examples ( Kurakin et al. , 2016 ; Athalye et al. , 2017 ; Eykholt et al. , 2018 ; Sitawarin et al. , 2018 ; Dutta , 2018 ; Duan et al. , 2020 ) , where the explicit goal is to implement these in the physical scene , so that the images of the scene subsequently captured by the camera and fed into a deep neural network result in a prediction error . In a related effort , Liu et al . ( 2019 ) developed a differentiable renderer that allows the attacker to devise higher-level perturbations of an image scene , such as geometry and lighting , through a differentiable renderer . However , most of these approaches attack a fixed input scene , whereas autonomous driving is a complex dynamical system . Several recent approaches investigate physical attacks on autonomous driving that attempt to account for the fact that a single object is modified and viewed through a series of frames ( Ackerman , 2019 ; Kong et al. , 2020 ; Boloor et al. , 2020 ) . However , these either still consider digital attacks , albeit restricted to a small area ( e.g. , replacing a road sign with a noisy road sign ) and do not consider a vehicles self-correcting behavior ( for example , ( Kong et al. , 2020 ) ) , or rely on many expensive driving experiments in order to identify adversarial patterns ( Boloor et al. , 2020 ) . 2 PROPOSED METHOD . Autonomous driving systems are equipped with decision algorithms that produce control signals for a vehicle , based on high-level instructions—such as a given route or destination—and inputs from cameras and other sensors that make continuous measurements of the vehicle ’ s physical environment . We assume the decision algorithm is in the form of a differentiable function—such as a neural network—that maps video frames from the camera , along with other inputs , to the control outputs . Given such a network or function , our goal is to determine if it is vulnerable to attack . Specifically , we seek to build a scalable and efficient method to find modifications that can be applied to the physical environment , and result in a stream of video frames which cause the control network to produce output signals that disrupt the vehicle ’ s operation , moving it away from the expected ideal trajectory . This task is challenging since the relationship between modifications to the physical environment and the network ’ s inputs is complex : the video frames correspond to images of the environment from a sequence of changing viewpoints , where the sequence itself depends on the network ’ s control outputs . The precise effect of any given modification can be determined only by actually driving the vehicle in the modified environment , or by using a high-quality simulator with a sophisticated physics engine . However , it is expensive to use actual driving or an expensive simulator when searching for the right modification , since neither process is differentiable with respect to parameters of the modification , and would require repeated trials with candidate modifications in every step of the search process . Instead , we propose a fast approximation to produce video frames for a given environment given a candidate modification that is differentiable with respect to parameters of the modification . Our approach requires a small number of initial calibration runs—of actual driving or a sophisticated simulator—after which , the search for optimal parameters can be carried out with end-to-end gradientbased optimization . Specifically , we consider the case when the modification takes the form of figures ( such as rectangles ) drawn on a restricted stretch of the road , and task the optimization with finding their optimal shape and color so as to maximize deviation from the controller ’ s trajectory prior to modification . We now describe our model for the physical modification , our approximate mapping to create video frames for a given modification , and our optimization approach based on this mapping . 2.1 PARAMETERIZED PHYSICAL MODIFICATION . We assume modifications are in the form of a collection of K figures ( e.g. , rectangles ) that will be painted on a flat surface in the environment ( e.g. , road ) . Let Φ = { xSk , xCk } Kk=1 denote the parameters of this modification , with xSk corresponding to the shape parameters , and x C k the RGB color , of the kth figure . These parameters are defined with respect to co-ordinates in some canonical—say , top-down—view of the surface . We let M ( nc ; xS ) denote a scalar-valued mask that represents whether a pixel at spatial location nc ∈ Z2 in the canonical view is within a figure with shape parameters xS . This function depends simply on the chosen geometry of the figures , and has value of 1 for pixels within the figure , 0 for those outside , and real values between 0 and 1 for those near the boundary ( representing partial occupancy on a discrete pixel grid to prevent aliasing artifacts ) . Since the spatial extents for different figures may overlap , we next account for occlusions by assuming that the lines will be painted in order . Accordingly , we define a series of visibility functions Vk ( nc ; Φ ) , each representing the visibility of the kth figure at pixel nc , after accounting for occlusions . We set the function for the last figure as VK ( nc ; Φ ) = M ( nc ; xSK ) , and for the other figures with k < K , Vk ( nc ; Φ ) = M ( nc ; x S k ) K∏ k′=k+1 ( 1− Vk′ ( nc ; Φ ) ) . ( 1 ) 2.2 APPROXIMATE FRAMES VIA COMPOSITING . The next step in our pipeline deals with generating the video inputs that the controller network is expected to receive from a modified environment for given parameter values Φ . These frames will represent views of the environment , including the surface with the painted figures , from a sequence of viewpoints as the car drives through the scene . Of course , the precise viewpoint sequence will depend on the trajectory of the car , which will depend on the control outputs from the network , that in turn depends on the frames . Instead of modeling the precise trajectory for every modification , we instead consider a small set of T representative trajectories , corresponding to those that the vehicle will follow when driven with small perturbations around control signals the network outputs , when operating in the unmodified environment . In our experiments , we use T = 4 trajectories : one from driving the car with the actual output control signals , and three from random noise added to these outputs . Given the fact that actual control is closed-loop , it is not evident that this simple approach would work ; however , our experiments below show that it is remarkably effective , despite its simplicity . This gives T sequences of video frames , one for each trajectory , where we assume each sequence contains F frames . We let Ĩtf ( n ) denote the f th “ clean ” image in the tth sequence , representing a view of the environment without any modifications . Here , n ∈ Z2 indexes pixel location within each image , and the intensity vector Ĩtf ( n ) ∈ R3 at each location corresponding to the recorded RGB values . These clean images can be obtained by driving the car—actually , or in simulation—in the original environment . For each frame in each sequence , we also determine a spatial mapping nc = Gtf ( n ) that maps pixel locations in the image to the canonical view . We model each Gtf ( n ) as a homography : the parameters of which can be determined by either using correspondences between each image and the canonical view of the surface—from calibration patterns rendered using the simulator , or from user input—or by calibrating the vehicle ’ s camera and making accurate measurements of ego-motion when the vehicle is being driven . Additionally , we also determine color mapping parameters Ctf ∈ R3×3 , btf ∈ R3 for each frame representing an approximate linear relationship between the RGB colors xC of the painted figures , and their colors as visible in each frame . These parameters are also determined through calibration . Given this set of clean frames and the geometric and color mapping parameters , we generate corresponding frames with views of the modified environment simply as : Itf ( n ; Φ ) = ( 1− K∑ k=1 Vk ( G t f ( n ) ; Φ ) ) Ĩtf ( n ) + K∑ k=1 Vk ( G t f ( n ) ; Φ ) ( Ctfx C k + b t f ) . ( 2 )

Summary: The paper proposes an end-to-end differentiable method for finding adversarial patterns to be added to the environment for autonomous driving. It utilizes image composition with homography thus it can compose the adversarial pattern into the image frames of all image frames of a driving sequence. Combined with a neural-network based controller which outputs the steering angle, the proposed method can find adversarial examples more efficiently comparing to a Bayesian optimization(BO) based baseline while resulting in trajectories with greater deviation.