Finding Physical Adversarial Examples for Autonomous Driving with Fast and Differentiable Image Compositing
Authors: Jinghan Yang
Adith Boloor
Ayan Chakrabarti
Xuan Zhang
Yevgeniy Vorobeychik
Venue:  (2021)
Keywords: Adversarial examples
autonomous driving

Abstract: There is considerable evidence that deep neural networks are vulnerable to adversarial perturbations applied directly to their digital inputs. However, it remains an open question whether this translates to vulnerabilities in real-world systems. Specifically, in the context of image inputs to autonomous driving systems, an attack can be achieved only by modifying the physical environment, so as to ensure that the resulting stream of video inputs to the car's controller leads to incorrect driving decisions. Inducing this effect on the video inputs indirectly through the environment requires accounting for system dynamics and tracking viewpoint changes. We propose a scalable and efficient approach for finding adversarial physical modifications, using a differentiable approximation for the mapping from environmental modifications—namely, rectangles drawn on the road—to the corresponding video inputs to the controller network. Given the color, location, position, and orientation parameters of the rectangles, our mapping composites them onto pre-recorded video streams of the original environment. Our mapping accounts for geometric and color variations, is differentiable with respect to rectangle parameters, and uses multiple original video streams obtained by varying the driving trajectory. When combined with a neural network-based controller, our approach allows the design of adversarial modifications through end-to-end gradient-based optimization. We evaluate our approach using the Carla autonomous driving simulator, and show that it is significantly more scalable and far more effective at generating attacks than a prior black-box approach based on Bayesian Optimization.

Under review as a conference paper at ICLR 2021
FINDING PHYSICAL ADVERSARIAL EXAMPLES FOR AUTONOMOUS DRIVING WITH FAST AND DIFFERENTIABLE IMAGE COMPOSITING
Anonymous authors Paper under double-blind review
ABSTRACT
There is considerable evidence that deep neural networks are vulnerable to adversar- ial perturbations applied directly to their digital inputs. However, it remains an open question whether this translates vulnerabilities in real-world systems. Specifi- cally, in the context of image inputs to autonomous driving systems, an attack can be achieved only by modifying the physical environment, resulting stream of video inputs to the car's controller leads to incorrect driving decisions. Inducing this effect the video inputs indirectly through the environ- ment requires accounting for system dynamics and tracking viewpoint changes. We propose scalable and efficient approach for finding adversarial physical modi- fications, using differentiable approximation for the mapping from environmental modifications namele rectangles drawn on the road-to to the corresponding video inputs to the controller network. Given the color, location, position, and orientation parameters of the rectangles, our mapping composites them onto pre-recorded video streams of the original environment. Our mapping accounts for geometric and color variations, differentiable with respect to rectangle parameters, and uses multiple original video streams obtained by varying the driving trajectory. When combined with a neural network-based controller, our approach allows the design of adversarial modifications through end-to-end gradient-based optimization. We evaluate our approach using the Carla autonomous driving simulator, and show that it is significantly more scalable and far more effective at generating attacks than a prior black-box approach based on Bayesian Optimization.
to ensure that the
INTRODUCTION
Computer vision has made revolutionary advances in recent years by leveraging combination of deep neural network architectures with abundant high-quality perceptual data. One of the transformative applications of computational perception is autonomous driving, with autonomous already being evaluated for use in geofenced settings, and partial autonomy, such as highway assistance, leveraging state-of-the-art perception embedded in vehicles available consumers. However, a history of tragic crashes involving autonomous driving, most notably Tesla (Thorbecke) 2020) and Uber (Hawkins) 2019) reveals that modern perceptual architectures still have some limitations even in non-adversarial driving environments. In addition, and more concerning, is the increasing abundance of evidence that state-of-the-art deep neural networks used in perception tasks are highly vulnerable to adversarial perturbations, or imperceptible noise that is added to an input image and deliberately designed to cause misclassification (Goodfellow al. 2014 Yuan et al.) 2019 Modas et al.) 2020). Furthermore, several lines of work consider specifically physical adversarial examples which modify the scene being captured by a camera, rather than the image (Kurakin et al.)
and trucks
2016 Eykholt et al. 2018 Sitawarin et al. 2018 Dutta 2018 Duan al. 2020).
Despite this body of evidence demonstrating vulnerabilities deep neural network perceptual architectures, it is nevertheless not evident that such vulnerabilities are consequential in realistic autonomous driving, even if primarily using cameras for perception. First, most such attacks involve independent perturbations to given input image. Autonomous driving is dynamical system, so that a fixed adversarial perturbation to a scene is perceived through a series of distinct, but highly interdependent perspectives. Second, self-driving is complex system that maps perceptual inputs

---

Under review as a conference paper at ICLR 2021
Parameters
Frames from Unmadified Environment
Approximately Rendered Frames
Compositing Homography Color
Controller
Figure I: Overview. We collect and calibrate frames from the unmodified environment (shown in the green box), and given choice of attack pattern parameters, composite the pattern to create approximate renderings of frames corresponding to placing the pattern in the environment. Our composition function is differentiable with respect the attack pattern parameters, and we are thus able to use end-to-ene gradient- based optimization when attacking a differentiable control network, to cause the network to output incorrect controls that cause the vehicle to deviate from its intended trajectory (from the green to the blue trajectory, as shown in the right column), and crash.
to control outputs. Consequently, even if we succeed in causing the control outputs to deviate from normal, the vehicle will now perceive sequence of frames that is different from those encountered on its normal path, and typically deploy self-correcting behavior in response. For example, if the vehicle is driving straight and then begins swerving towards the opposite lane, its perception will inform the control that it's going in the wrong direction, and the controller will steer it back on course.
To address these limitations, Bayesian Optimization (BO) (Archetti and Candelieri, 2019) recently proposed as a way to design physical adversarial examples (2 black rectangles on road pavement) in Carla autonomous driving simulations (Dosovitskiy al. 2017) against end-to-end autonomous driving architectures (Boloor et al. (020). The key challenge with this approach, however, that attack design must execute actual experiments (e.g., simulations, or actual driving) for a larger number of iterations (1000 in the work above), making it impractical for large-scale or physical driving evaluation. Furthermore, it is not clear how well BO scales as we increase the complexity of the adversarial space beyond 2 black rectangles.
We propose highly scalable framework for designing physically realizable adversarial examples against end-to-end autonomous driving architectures. Our framework is illustrated in Figure [ and develops a differentiable pipeline for digitally approximating driving scenarios. The proposed approximation makes use of image compositing, learning homography and color mappings from a birds-eye view of embedded adversarial examples to projections of these in images based on actual driving frames, and sampling sequences of actual frames with small added random noise to control to ensure adequate sampling of possible perspectives. The entire process can then be fed into automatic differentiators obtain adversarial examples that maximize car's 's deviation from its normal sequence of controls (e.g., steering angle) for a target driving scenario.
We evaluate the proposed framework using Carla simulations in comparison with the state-of-the-art BO method. Our experiments show that the resulting attacks are significantly stronger, with effects on induced deviations and road infractions often considerably outperforming BO, small fraction of actual driving runs required for training. Furthermore, we show that our approach yields attacks that are robust to unforeseen variations in weather and visibility.
Related Work: Attacks on deep neural networks for computer vision tasks has been a subject of
extensive prior research (Goodfellow al. 2014 Yuan al., 2019 Modas et al. 2020 Vorobeychik and Kantarcioglu 2018). The most common variation introduces imperceptble noise to pixels of an image in order to induce error in predictions, such as misclassification of the image failure to detect object in it. A more recent line of research has investigated physical adversarial examples
(Kurakin et al. 2016 Athalye et al.) 2017 Eykholt et al. 2018 Sitawarin al. 2018 Dutta 2018) Duan et al. 2020), where the explicit goal is to implement these in the physical scene, so that the images of the scene subsequently captured by the camera and fed into deep neural network result in prediction error. In a related effort, Liu et al. (2019) developed differentiable renderer that allows the attacker to devise higher-level perturbations of an image scene, such as geometry and lighting, through a differentiable renderer. However, most of these approaches attack fixed input
2

---

Under review as a conference paper at ICLR 2021
scene, whereas autonomous driving is complex dynamical system. Several recent approaches investigate physical attacks on autonomous driving that attempt to account for the fact that a single object is modified and viewed through series of frames (Ackerman 2019 Kong et al.. 2020 Boloor et al. 2020). However, these either still consider digital attacks, albeit restricted to a small area (e.g.. replacing a road sign with noisy road sign) and do not consider a vehicles self-correcting behavior (for example, (Kong et al. 2020)) or rely on many expensive driving experiments in order to identify adversarial patterns (Boloor et al. 2020
2 PROPOSED METHOD
Autonomous driving systems are equipped with decision algorithms that produce control signals for vehicle, based on high-level instructions-succh as given route or destination-ant inputs from cameras and other sensors that make continuous measurements of the vehicle's 's physical environment. We assume the decision algorithm is in the form of a differentiable function- ssuhhssuchhhee as neural network-thal maps video frames from the camera, along with other inputs, to the control outputs. Given such a network or function, our goal is to determine if it vulnerable to attack. Specifically, we seek to build scalable and efficient method to find modifications that can be applied to the physical environment, and result in a stream of video frames which cause the control network to produce output signals that disrupt the vehicle's operation, moving it away from the expected ideal trajectory.
This task is challenging since the relationship between modifications to the physical environment and the network's 's inputs is complex: the video frames correspond images of the environment from sequence of changing viewpoints, where the sequence itself depends on the network's 's control outputs. The precise effect of any given modification can be determined only by actually driving the vehicle in the modified environment, or by using high-quality simulator with a sophisticated physics engine. However, is expensive use actual driving or an expensive simulator when searching for the right modification, since neither process is differentiable with respect to parameters of the modification, and would require repeated trials with candidate modifications in every step of the search process.
Instead, we propose fast approximation to produce video frames for a given environment given a candidate modification that differentiable with respect to parameters of the modification. Our approach requires small number of initial calibration runs-of actual driving or sophisticated simulator-afie which, the search for optimal parameters can carried out with end-to-end gradient- based optimization. Specifically, we consider the case when the modification takes the form of figures (such as rectangles) drawn on a restricted stretch of the road, and task the optimization with finding their optimal shape and color so as maximize deviation from the controller's ss trajectory prior to modification. We now describe our model for the physical modification, our approximate mapping to create video frames for given modification, and our optimization approach based on this mapping.
2.1 PARAMETERIZED PHYSICAL MODIFICATION
We assume modifications are in the form of a collection of K figures (e.g., rectangles) that will be painted on a flat surface in the environment (e.g., road). Let + {=E xx k=1 denote the parameters of this modification, with z corresponding to the shape parameters, and x the RGB color, of the Athh figure. These parameters are defined with respect to co-ordinates in some canonical say, top-down- view of the surface.
C
We let ()) denote scalar-valued mask that represents whether pixel at spatial location n. E 2 in the canonical view is within a figure with shape parameters x.._ This function depends simply on the chosen geometry of the figures, and has value of 1 for pixels within the figure, 0 for those outside, and real values between 0 and 1 for those near the boundary (representing partial occupancy discrete pixel grid prevent aliasing artifacts).
Since the spatial extents for different figures may overlap, we next account for occlusions by assuming that the lines will be painted in order. Accordingly, we define a series of visibility functions (n, 2). each representing the visibility of the 4eh figure at pixel ne. after accounting for occlusions. We set the function for the last figure as Vx (ne: b) = M (ne; )x). and for the other figures with i < K,
(m 2) M M
II
(1 Ver 2))
(1)
A ==
3

---

Under review as a conference paper at ICLR 2021
2.2 APPROXIMATE FRAMES VIA COMPOSITING
The next step in our pipeline deals with generating the video inputs that the controller network is expected to receive from a modified environment for given parameter values Φ These frames will represent views of the environment, including the surface with the painted figures, from a sequence of viewpoints as the car drives through the scene. Of course, the precise viewpoint sequence will depend on the trajectory of the car, which will depend on the control outputs from the network, that in turn depends on the frames. Instead of modeling the precise trajectory for every modification, we instead consider small set of 'T' representative trajectories, corresponding to those that the vehicle will follow when driven with small perturbations around control signals the network outputs, when operating in the unmodified environment. In our experiments, we use T 4 trajectories: one from driving the car with the actual output control signals, and three from random noise added to these outputs. Given the fact that actual control is closed-loop, is not evident that this simple approach would work; however, our experiments below show that it is remarkably effective, despite its simplicity.
This gives T' sequences of video frames, one for each trajectory, where we assume each sequence contains F frames. We let (n) denote the fth "clean" image in the 1 sequence, representing view of the environment without any modifications. Here, n E z² indexes pixel location within each image, and the intensity vector (n) E R at each location corresponding to the recorded RGB values. These clean images can be obtained by driving the car actuall,, or in simulation- in the original environment.
For each frame in each sequence, we also determine a spatial mapping ne (n) that maps pixel locations in the image to the canonical view. We model each (n)(n) as a homography: the parameters of which can be determined by either using correspondences between each image and the canonical view of the surface- from calibration patterns rendered using the simulator, or from user input or by calibrating the vehicle's camera and making accurate measurements of ego-motion when the vehicle being driven. Additionally, we also determine color mapping parameters Cý E 3x3 b R3
for each frame representing an approximate linear relationship between the RGB colors 20 of the painted figures, and their colors as visible in each frame. These parameters are also determined through calibration. Given this set of clean frames and the geometric and color mapping parameters, we generate corresponding frames with views of the modified environment simply as:
[2 (100))0(111
K
K
Σ (m) 17(m) Σ
(m) (Cj2 + bb).
(2)
2.3 GRADIENT-BASBR OPTIMIZATION
Given the "forward" process of generating a set of frames for a given set of modification values 4 we finally describe our approach to finding the value of & that misleads the control network. We let
D({1g1z]))) denote the controller network, which takes as input a sequence of frames { } in a single trajectory and generates corresponding sequence of real-valued control signals, such steering angle at each time instant. Our goal is to find the value of t that maximizes deviation of these outputs from those for an unmodified environment. We cast this minimization of loss over our T trajectories, i.e.,
[m]]
arg min
in which -(.-)) measures divergence between two sequences of control outputs. In our experiments where the control network outputs a sequence of steering angles, we use the absolute value of the sum of the differences between the angles as our divergence metric.
We carry out this optimization iteratively using gradient descent with Adam (Kingma and Ba 2014) because, as shown next, we are able to compute gradients of the loss in (3) with respect to the modification parameters &. Since the controller network D(-) is assumed to be a neural network (or a differentiable function), we are able to use standard back-propagation to compute gradients v (m;; b)) of the loss with respect to each intensity in the images of the modified environment.
4

---

Under review as a conference paper at ICLR 2021
The gradients with respect the color parameters {af) can then be computed based
as:
v
Σ (Ct) t
Σ (n): (z((
(4)
Computing gradients with respect to the shape parameters {ax) requires an approximation, since the mask functions M(.) are not generally differentiable with respect to these parameters. We adopt a simple local linearization approach: for every scalar parameter & in the shape parameters zig for each figure, we discretize its range into a fixed set of equally separated values. Then, given the current (continuous) value of 0, we let 0- and 0+ represent the consecutive pair of values in this discrete set, such that f- 5 S g+, and denote o and Lg+ the full set of current parameter values, with 0 replaced by 0+ and 0- respectively. We make the approximation that if a E IR such that 0 = af+ + (1 a))0 then
(5) Therefore, although we only use frames (n; [n; Q) with the actual value of t as input to the control network, we also generate an extra pair of images (n::; Do ), 1 (n;; Pot) ) for each frame for every element 0 of the shape parameters. We then use these frames to compute parameter gradients as:
(m; 2) ~ (m; 26t) ) (1 -a) (m;; D
V(a)= = -E v (1) (m) Do+) (m 2o-)), ( = v(a) x (0+ --). (6) t.f.r
Therefore, our approximate framework allows us to compute gradients of the loss in (3) with respect to all the shape and color parameters in Φ and use gradient descent to minimize this loss.
3 EXPERIMENTS
In this section, we experimentally evaluate our approach, which we refer to as GradOpt, and demon- strate that it is both more efficient-requiiiin fewer actual or simulated drives-ant and effective- successfully finding attack patterns- the state-of-the-art Bayesian Optiiization (BO) method (Boloor al. 2020). To able to carry out a large scale evaluation, we perform our experiments in simulation using packaged version of the Carla simulator (Dosovitskiy et al., (017) provided by Boloor et al. (2020) that allows the addition of painted road patterns to existing environments. Our experiments evaluate attacks against the control network that is included with Carla: this is neural network-bart based controller that uses only camera inputs and outputs steering angles, and was trained using imitation learning. We run evaluations only on scenarios where this controller drives successfully without infractions in the unmodified environment.
Our experiments consider forty scenarios, each requiring driving through stretch of road in virtual town. Each scenario begins an episode with the vehicle spawned at a given starting waypoint, and the controller is then tasked with reaching defined destination waypoint. The episode runs until the vehicle reaches this destination or a time-limit expires (e.g., if the car crashes). Our scenarios are of three types: (a) the expected behavior is for the car go straight (16 scenarios), (b) veer left (12 scenarios), or (c) right (12 scenarios)-eillee through an intersection, or on a straight or curved road.
Our threat model assumes an attacker can draw a pattern on the road with the intent of causing the controller to output erroneous steering angles and the car to deviate from its intended path. Accordingly, we task both BO and our method, GradOpt, with finding such an attack pattern for each scenario. We consider patterns that are unions of rectangles (i.e., each "figure" in Sec. 2.1 is a rectangle), where the shape of each rectangle determined by four parameters (i.e., of E ) rotation, width, length, and horizontal offse !I We report results when searching over shape and color parameters for different numbers of rectangles K K-ranging from K = (7 parameters) to K = 5 (35 parameters), and additionally for the single rectangle case, also when searching over only shape parameters while keeping color fixed to black (4 parameters). We learn these parameters with respect to the top-view co-ordinate frame of a canvas, that during evaluation will be passed to the simulator to be superimposed on the road (and then captured by the camera part of the scene in all frames).
We assume the rectangle is vertically centered prior to rotation. This is similar to the parameterization used in Boloor al. 2020), except that we also search over length (for both GradOpt and BO).
5

---

Under review as a conference paper at ICLR 2021
We train both BO and GradOpt in simulated setting without any pedestrians other cars (this models a realistic scenario where an attacker is unlikely to test or calibrate their attacks with others present). We evaluate the success of the attack actual simulations with Carla, in terms of two metrics. The first measures deviation between the paths with and without the attack pattern, both without pedestrians other cars. Here, deviation is defined as:
t=T
Deviation 27 min W. Wel + min IW, Web
where W, and W2 are sequence of vehicle locations when driving with and without the attack pattern, at a fixed set of time instances defined as those when the region of the road where the attack pattern is to be placed is visible while driving in the unmodified environment (i.e., without the pattem). The second metric in our evaluation is the total infraction penalty when driving with pedestrians and other here-here we use the standard penalties as defined by the Carla Autonomous Driving Challenge (Ros et al., 2019) (where, for example, a lane violation carries penalty of 2 points, hitting static object or another vehicle of 6, hitting pedestrian of 9, etc.). For each attack and scenario, we run simulations ten times (with the simulator randomly spawning pedestrians and cars each time) and average the infraction penalty scores. As mentioned before, the controller has zero infractions when driving in the unmodified environment in all these scenarios. Finally, while both BO and GradOpt are trained in a clear noon weather setting, we measure infractions on that setting as well as three others: cloudy noon, rainy noon, and clear sunset.
3.1 ATTACK OPTIMIZATION
Our approach requires two steps for every scenario: (1) collecting a set of frame sequences and calibrating them, and (2) performing gradient-based optimization. For (I), we collect frames from four trajectories one obtained by driving the car with the direct outputs of the controller, and three others with perturbations to these outputs. We estimate the homographies Gr for every frame f in trajectory by running additional simulations with calibration patterns painted on the canvas: we run twelve simulations for each trajectory, with five calibration points of different colors in each simulation, and use the sixty automatically detected correspondences to estimate the homography matrix for every frame. We also learn a common set of color transform parameters for all frames in all trajectories, which we obtain by running the simulator twenty-tw times, each time with the entire canvas painted a different high-contrast color, on the unperturbed trajectory. Therefore, our method calls the simulator total of 74 times. Note, this because we chose to use automated calibration to be consistent with the access provided to BO below (Boloor et al.) (020). and the number of calls could be reduced with manual annotation (or by using calibrated accelcration information from the simulator vehicle).
Proposed Method:
Once we have a set of calibrated frames, we employ gradient- based optimization (see Sec. 2.3 We use the absolute value of the sum of differences in steering angles as the divergence metrie (-) in (3) We run the optimization with four different random starts, each for 250 iterations, for a total of 1000 iterations. We begin with learning rate of 0.1 for each start, and drop it by a factor of 0. after the first 100, and the first 200 iterations. Each time we drop the learning rate, we also reset the parameters those that yielded the lowest value of the loss thus far.
Bayesian Optimization: We employ BO with the same objective as ours based on the same diver- gence metric, and closely follow the setting in (Boloor et al. 2020 we run the optimization for a total of 1000 iterations, of which the first 400 are random exploration. Note that while this is the same number of iterations as we use for our method, every iteration of BO requires running a full episode with the Carla simulator.
Every iteration of gradient- based updates using our method takes 8 seconds, whereas simulating full episode using the Carla simulator takes more than 80 seconds- 1080Ti GPU. Since BO requires 1000 calls to the simulator, it takes more than 22 hours to find the optimal attack pattern for each scenario. In contrast, we only call the simulator 74 times and then run 1000 iterations of gradient- for the four random starts, this corresponds to running time of around four hours per scenario. Thus, our method affords a significant advantage in computational cost, and as we show next, is also more successful at finding optimal attack patterns.
Run-time:
on an NVIDIA
optimization. Ignoring the potential of parallelizing the iterations

---

Under review as a conference paper at ICLR 2021
Route Start
Route End
* Crash Point
Trajectories
Unmodifiidd
w/ BO Pattern
w/ GradOpt Pattern
Figure 2: Trajectory deviations induced by attack. For four example scenarios, we illustrate the effect of the attack patterns found by the proposed GradOpt method and BO, by showing trajectories taken by the vehicle in the simulator when these patterss are placed on the road, and comparing these the original trajectory in the unmodified environment.
Deviation
Infraction Penalty
K
GradOpt (T=1)
BO
GradOpt
%
BO
GradOpt
% 2
(#Rect.)
1-b
0.79
0.98
72%
0.73
3.84
3.97
88%
1
0.86
1.00
72%
0.82
4.30
5.21
75%
2
0.85
1.16
82%
0.89
4.50
6.40
80%
3 0.90
1.23
75%
0.95
4.53
6.03
88%
4
0.74
1.29
80%
1.10
3.69
6.40
90%
5
0.82
1.33
82%
0.99
4.70
7.43
82%
Table 1: Average deviation (in simulations without cars pedestrians) and infraction penalties (averaged across 10 simulations with cars and pedestrians) across all scenarios for GradOpt (proposed) and BO, when searching for parameters of different numbers of rectangles (1-b refers to searching only the shape parameters of one rectangle, fixing its color black). All simulations are "clear noon" weather as also used for attack optimization. The % 2 column reports the percentage of instances where GradOpt has higher or equal score than BO, Also shown are average deviations when using GradOpt with only one trajectory (I' 1), instead of four, for optimization.
3.2 RESULTS
We begin by describing an evaluation of the proposed GradOpt method's ability to find successful attacks in all forty scenarios, and comparing this to the prior BO approach of Boloor et al. (2020). repoots quantitative results terms of the metries deviation and infraction penalty discussed above, computed with simulations in the same standard weather setting used for attack optimization. We report averages for these metrics over all scenarios, as well as the percentage of cases when GradOpt performs well better than BO, when searching for patterns of increasing complexity (in terms of number of rectangles K). We find that GradOpt is significantly more successful than BO, despite also being computationally less expensive as discussed earlier. It has higher average values of both deviation and infraction penalties, with the gap growing significantly for higher values of K K-indicating that GradOpt is much better able carry out optimization parameter space and leverage the ability to use more complex patterns. Moreover, we find that it yields attack patterns that as more successful than BO in more than 70% of cases in all settings, with the advantage rising to 82% for K = 5. Figure 2 shows some example scenarios comparing trajectory deviations induced by the attack patterns discovered by the two algorithms.
higher-dimensional
Recall that GradOpt trains with frames collected from four trajectories as a means of approximately accounting for self-correcting behavior of the control network, as it begins to deviate from its original trajectory which would result in frames from different viewpoints. To evaluate the importance of this approach, Table
I also includes an ablation measuring average deviation when optimizing with GradOpt using only one (T 1) trajectory. We find that the discovered attack patterns in this are less successful when used actual simulation, leading to lower average deviation scores than optimizing with multiple rajectories-althne these scores remain competitive with those from BO.
7

---

Under review as a conference paper at ICLR 2021
Straight GradOpt
Left
Right
#Rect.
BO
% 2
BO
GradOpt
% 2
BO
GradOpt
%
1-b
2.23
1.60
88%
4.70
4.75
92%
5.14
6.39
83%
2.01
2.79
81%
5.53
5.25
60%
6.11
8.42
83%
2
1.89
4.41
100%
5.28
5.60
67%
7.20
9.85
67%
3
2.73
4.30
88%
5.47
5.53
75%
5.98
8.82
92%
0.91
4.11
100%
5.00
7.00
83%
6.07
8.86
83%
5
1.66
6.20
94%
5.60
7.05
70%
7.83
9.46
83%
Table 2: Infraction penalties for each type of scenario where the expected trajectory is driving straight, left, or right. These results are for simulations in "clear noon" conditions.
K
Cloudy Noon
Rainy Noon
Clear Sunset GradOpt
#Rect.
BO
GradOpt
% >
BO
GradOpt
%
BO
%
1-b
2.29
3.89
85%
2.69
3.87
82%
2.41
2.60
88%
1
3.19
3.28
82%
2.98
3.33
82%
3.36
3.28
90%
2
2.87
5.29
90%
4.03
5.03
80%
2.92
3.75
75%
3 3.06
4.85
88%
3.28
5.61
88%
2.60
3.60
93%
4
2.39
5.80
88%
3.18
5.67
78%
2.38
4.41
93%
5
3.56
5.50
85%
3.81
5.53
80%
2.67
5.65
90%
Table 3: Infraction penalties, across all scenarios, in simulations with weather conditions different from the used for attack optimization ("clear noon").
We take a closer look at the vulnerability of different types of scenarios by separately reporting infraction penalties for each in Table We see that scenarios where the expected behavior is driving straight are the hardest to attack, likely because they are the simplest to drive in. BO tends to achieve only moderate infraction score in these settings, even higher values of K. contrast, GradOpt reveals that even these scenarios in fact vulnerable when one is allowed consider more complex attack patterns-achieving achieving an average deviation that is four times higher than BO at K Conversely, driving right scenarios the most vulnerable with both methods being successful even with simple patterns, with GradOpt again yielding higher deviation and more infractions.
Finally, evaluate the robustness of our attack approach by evaluating their success in different climate and visibility conditions than those used for attack optimization. Table presents results for simulations with four such climate settings, and expected, see a drop in penalty scores compared to the standard setting in Table Nevertheless, most of the attacks induce infractions, especially at higher values of K, with GradOpt again being significantly more successful than BO.
find that both BO and GradOpt do
4 CONCLUSION
A great deal of attention has been devoted to understanding adversarial perturbations in computational perception, with autonomous driving the most common motivation. However, most prior research has considered these to be independent for each input image. In contrast, autonomous driving is dynamic, and even if perturbation succeeds fool the car in a particular frame, can fail in another frame, and he car can self-correct. Thus, to fully understand vulnerabilities in autonomous vehicle architectures we need to evaluate them in an end-to-end fashion using driving scenarios. However, this is inherently challenging because the resulting experimentation, be it using a simulator or actual driving, is extraordinarily time consuming. We developed a novel framework that allows us to largely avoid costly driving experiments, relying instead on novel compositing approach which is fully differentiable. Our approach significantly more potent at discovering physically realizable adversarial examples than prior art, while also requiring far fewer runs of actual driving trials or simulations. Moreover, we find that the vulnerabilities we discover are robust, and persist even under variability in weather conditions.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Evan Ackerman. Teslas autopilot fooled by stickers. IEEE Spectrum. 242(3224):6, 2019. doi:
URL https://///sss ieee www..eee...rrr-ttaaaaaaeaaa aatrrrrecccssshsttturrrrrrrrrtte think/transportation/ self self-drrvinggthree wtaeeessaalll-aesssssseee- coaoo--rr-aann-- ctn-serr-tterr-- strraanstteeeell----eeeeeee-eaee tetteeeeeeeaaauuuuulo---oooooooo
lane
Francesco Archetti and Antonio Candelieri. Bayesian Optimization and Data Science. Springer,
2019.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. CoRR, abs/1 .07377, 2017. U httpp///arrv..ogg/h/////77777779
Adith Boloor, Karthik Garimella, Xin He, Christopher Gill, Yevgeniy Vorobeychik, and Xuan Zhang. Attacking vision-basee based perception end-to-end autonomous driving models. Journal of Systems Architecture, 110:101766, 2020.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1-16, 2017.
Ranjie Duan, Xingjun Ma, Yisen Wang, James Bailey, A. K. Qin, and Yun Yang. Adversarial camouflage: Hiding physical-world attacks with natural styles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Raj Gautam Dutta. Security of autonomous systems under physical attacks: With application to self-driving cars. 2018.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-workd attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint
2014.
Andrew Hawkins. Uber is at fault for fatal self- crash, but it's not alone. The Verge,
2019. URL http:///wwwwaaeeeeeee/////////// fault- self wriff-dri-i-aaa--ntteee probable-cause
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL http cite 2.t.14412.99000mmment Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
/arxiv.org/abs/144
Zelun Kong, Junfeng Guo, Ang Li, and Cong Liu. PhysGAN: Generating ahysical-woord--essileee adversarial examples for autonomous driving. In Conference on Computer Vision and Pattern Recognition, pages 14254-14263, 2020.
Alexey Kurakin, lan J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. CORR, hew///60002533 2016. URL http://arniv..org/ab///11......2
Hsueh-Ti Derek Liu, Michael Tao, Chun-Liang Li, Derek Nowrouzezahrai, and Alec Jacobson. Beyond pixel norm-balls Parametric adversarics using an analytically differentiable renderer. In International Conference Learning Representations, 2019,
A. Modas, R. Sanchez-Matilla Frossard, and A. Cavallaro. Toward robust sensing for autonomous vehicles: An adversarial perspective. IEEE Signal Processing Magazine, 37(4):14-22 2020.
German Ros, Vladlen Koltun, Felipe Codevilla, and Antonio M. Lopez. Carla autonomous driving challenge, 2019. https:/ccutlachalllng..om
Chawin Sitawarin, Arjun Nitin Bhagoji, Arsalan Mosenia, Mung Chiang, and Prateek Mittal. DARTS: deceiving autonomous cars with toxic signs. CoRR, abs/1 802.06430, 2018. URL http://axxiv.rrr// Abs/1802.004330

---

Under review as a conference paper at ICLR 2021
Catherine Thorbecke. Tesla on autopilot had steered driver towards same barrier before fatal crash, NTSB says. ABC News, 2020. URL
wot..com/Busnnsssssslllaaaaaaaae steered
Yevgeniy Vorobeychik and Murat Kantarcioglu. Adversarial Machine Learning. Morgan & Claypool, 2018.
Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 30(9):2805 2824, 2019.
10

---

Under review as a conference paper at ICLR 2021
APPENDIX
In this appendix, we provide further details of our shape parameterization and calibration, as well as additional results and visualizations from our experiments.
A PATTERN SHAPE PARAMETERS
Each rectangle in our shape is parameterized by four values 2xf [w, 1, 0,,00, 0), corresponding to width, length, horizontal offset, and rotation or orientation. These parameters are defined with respect to the top-down view of a 400 x 400 pixel "canvas" that is composited onto the road surface. Each rectangle first drawn aligned with the and axess of this canvas to be of width and length pixels, and vertically centered and horizontally offset so that its left edge is at a = O, and then rotated about the center of the canvas by angle 8. Portions of rectangles that lay outside the canvas after this process were clipped from the pattern. Our parameterization expands on the one originally used by Boloor al. (2020) in two respects: by searching over length instead of fixing it to the length of the canvas, and having a separate orientation for each rectangle rather than a common one for all rectangles.
B CALIBRATION DETAILS
We estimate homographies between the canvas and each frame from 60 corresponding pairs as described in Sec. using a direct linear transform. While doing so, we ensure the matrix has the correct sign so that homogeneous co-ordinates of points projected in the frame have a positive third co-ordinate when they are visible, and a negative one when they are "behind the camera". When compositing patterns on the frame, this allows to retain only the portion of the pattern that would be visible from that viewpoint. The color transforms are estimated simply from the color correspondences using least-squares fit.
C ADDITIONAL RESULTS
In main evaluation, we reported deviations (without cars or pedestrian) only in the overall evaluation Table [ and reported infraction penalties with cars and pedestrians for all other comparisons. For completeness, infraction penalties computed in simulations without cars or pedestrians (with the caveat that some of the highest penalties defined by the challenge for collisions with pedestrians and cars).
report deviation scores for those comparisons here, as well as
Table 4 reports deviation scores separately for different types of scenarios, and Table the corre- Finally, in Tables 6
sponding infraction penalties when driving without pedestrians report both deviation and car and pedestrian-free infraction penalty scores for simulations in the different non-standard weather conditions. We find that these results are qualitatively consistent with those in our main evaluation in Sec. B
K
Straight GradOpt
Left GradOpt
Right GradOpt
#Rect.
BO
%
BO
%
BO
%
I-b
0.50
0.56
62%
0.93
1.43
92%
1.04
1.12
67%
0.48
0.61
69%
0.91
1.39
83%
1.30
1.13
67%
2 0.49
0.82
88%
1.10
1.47
92%
1.09
1.30
67%
3
0.52
0.81
69%
1.27
1.59
83%
1.04
1.42
75%
0.42
0.86
75%
0.93
1.82
100%
0.98
1.34
67%
5
0.45
0.97
81%
0.96
1.68
92%
1.15
1.44
75%
Table 4: Deviations simulations without cars or pedestrians, in standard "clear noon" weather conditions, for each type of scenario.
11

---

Under review as a conference paper at ICLR 2021
Straight GradOpt
Left
Right GradOpt
#Rect.
BO
%
BO
GradOpt
%
BO
% 2
1-b
1.62
1.00
88%
2.33
3.75
83%
3.33
4.08
67%
1.50
1.88
75%
4.17
4.25
75%
5.17
7.33
83%
0.75
1.88
100%
2.67
4.17
83%
6.00
6.50
67%
2.12
3.12
81%
3.67
4.17
75%
5.17
6.33
92%
0.50
2.38
94%
1.67
4.58
92%
5.33
7.42
92%
1.62
3.62
88%
3.00
5.25
75%
6.33
8.33
100%
Table Infraction penalties without cars or pedestrians for each type of scenario, in standard "clear noon" simulations. Note these differ from standard infraction penalties in Sec. where penalties are computed with simulated cars and pedestrians.
Deviation
Infraction Penalty
BO
GradOpt
2
BO
GradOpt
>
#Rect.
1-b
0.90
0.90
90%
1.05
2.40
72%
0.51
0.97
90%
1.65
2.45
80%
0.59
1.16
95%
2.00
3.65
85%
0.73
1.17
97%
1.10
3.35
90%
0.74
1.22
95%
1.30
4.20
82%
0.56
1.30
90%
1.90
4.35
88%
Table 6: Deviation and infraction penalties, both computed without cars or pedestrians, over all scenarios in "cloudy noon" conditions.
Deviation
Infraction Penalty
BO
GradOpt
% 2
BO
GradOpt
% >
#Rect.
1-b
0.77
1.0
75%
1.05
2.30
90%
0.56
0.96
78%
1.50
2.95
85%
0.47
1.05
88%
1.60
3.15
93%
0.83
1.07
80%
2.10
3.80
85%
0.84
1.13
80%
1.60
3.90
88%
0.47
1.20
88%
1.60
3.95
97%
Table 7: Deviation and infraction penalties, both computed without cars or pedestrians, over all scenarios in "rainy noon" conditions.
Deviation
Infraction Penalty
K
BO
GradOpt
% 2
BO
GradOpt
% 2
#Rect.
1-b
0.50
0.73
65%
1.30
1.25
65%
0.40
0.70
75%
1.55
2.00
75%
0.40
0.90
90%
1.05
1.95
90%
0.39
0.94
85%
0.75
2.40
85%
0.40
1.10
93%
1.45
2.80
93%
0.37
1.19
90%
1.70
2.60
90%
Table 8: Deviation and infraction penalties, both computed without cars pedestrians, over all scenarios "clear sunset" conditions.
12

---

Under review as a conference paper at ICLR 2021
D VISUALIZATION
Finally, we use an example "drive straight" scenario to visualize the behavior the controller when driving with attack patterns, with K 4 rectangles each, returned by GradOpt and BO. We show these patterns, in the top-down canvas view, in Fig. 3 Then, we show frames from the vehicle's camera feed it drives through the road with the respective patterns painted on the road in various climate conditions, in simulations with pedestrians and cars in Fig. 4 and without in Fig. 5
For this scenario, the pattern returned by BO unable to cause significant deviation in the vehicle's trajectory as it drives across the stretch of road with the pattern painted on it. In contrast, GradOpt's pattern is able cause the car veer sharply the left in all but the "clear sunset" climate setting (which as we see looks visually the most different from the base (aating)-cusstin it to crash into another car in Fig. and the opposite sidewalk Fig. 5
GradOpt
BO
Figure 3: Attack patterns with K 4 rectangles returned by GradOpt and BO for the example "drive straight" scenario illustrated in Figures 4 and 5
13

---

Under review as a conference paper at ICLR 2021
Simulations with GradOpt Pattern
Clear Noon
Cloudy Noon
Rainy Noon
Clear Sunset
Simulations with BO Pattern
Clear Noon
Cloudy Noon
Rainy Noon
Clear Sunset
Figure 4: Frames from driving simulations, with cars and pedestrians in different weather conditions, after introducing attack patterns from GradOpt (top) and BO (bottom).
14

---

Under review as a conference paper at ICLR 2021
Simulations with GradOpt Pattern
Clear Noon
Cloudy Noon
Rainy Noon
Clear Sunset
Simulations with BO Pattern
Clear Noon
Clear Noon
Rainy Noon
Clear Sunset
Figure 5: Frames from driving simulations, without cars or pedestrians in different weather conditions, after introducing attack patterns from GradOpt (top) and BO (bottom).
15


Reviews:
[{"review_id": "pNvQR-9SN7o", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an end-to-end differentiable method for finding adversarial patterns to be added to the environment for autonomous driving. It utilizes image composition with homography thus it can compose the adversarial pattern into the image frames of all image frames of a driving sequence. Combined with a neural-network based controller which outputs the steering angle, the proposed method can find adversarial examples more efficiently comparing to a Bayesian optimization(BO) based baseline while resulting in trajectories with greater deviation.\n\nThe proposed method relies on an approximation of the image frames of the trajectories by adding random noise to the controller outputs and use those trajectories for learning. This dramatically reduces the number of calls to the simulator comparing to the baseline BO method. The paper states that \"Given the fact that actual control is closed-loop, it is not evident that this simple approach would work; however, our experiments below show that it is remarkably effective, despite its simplicity.\" Is it possible the reason for this approximation to work is that the scenarios are not complicated enough? If for more complicated scenarios where more calls to the simulator are needed, the benefit of the proposed method over the baseline would be much smaller. \n- Related question: are the trajectories with the noisy control resulting in infraction or not?\n\nOther questions / comments:\n1. The choice of using four trajectories as an approximation seems random. Any ablation study on the number of trajectories?\n2. The results in the three tables only contain average values without standard deviation.\n3. In sec 2.3 \"we use the absolute value of the sum of the differences between the angles as our divergence metric.\" It seems to me that the sum of the absolute values of the differences between the angles makes more sense because the positive and negative difference values would cancel out without taking absolute values before the summation. Similarly in Sec 3.1.\n4. Related to 3, I'm also curious about the design choice of using the controller deviation as the optimization objective function while using the trajectory deviation to measure the effectiveness of the adversarial attack. Those two may not necessarily correlate (An example will be if the car is supposed to go straight, while zero steering and both a left and a following correcting right steering will keep the car straight thus resulting in very similar trajectory, the controls are more different. On the other hand, a single left or right steering will result in large trajectory error as it accumulates.)\n5. All experiment results are in simulation so it's hard to draw conclusion regarding real driving scenarios.\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "b5E6fQ2hrFh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to design physical adversaries to attack end-to-end autonomous driving systems. The proposed approach maps adversarial patterns onto video frames recorded from real world to generate adversarial examples for deviating the control of a vehicle. \n\nThe problem considered by this work is significant as physical adversarial attacks pose serious threats to the safety of autonomous driving. The paper is well written and most of the technical details are clearly articulated. The experiments demonstrates the efficacy of the proposed solution with results reported under different physical conditions . \n\nOne main concern I have about this work is that its motivation does not seem strong enough. Since both the training and evaluation are performed based on simulation, how real video streams help in developing a more robust approach remains unclear to me. The authors may need to more clearly argue why simulated images are less advantageous in this case. \n\nThe geometric and color parameters are obtained from manual calibration in this work. While this makes learning easier, designing an automatic matching method to learn these parameters from data would be more interesting, and potentially leads to more robust attacks against realistic environments. \n\nSome other comments:\n1) It’s a little bit difficult to understand why a few painted boxes on the road could fool an autonomous system. It would be helpful if the authors could provide some explanations of how these physical modifications affect the vehicle's controller.\n2) How are M and V determined in Eq. (1)?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PiAAuZ1S01U", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a scalable and efficient approach for finding adversarial physical modification to the video inputs of autonomous driving. Assuming the perturbations are in form of a collection of several rectangles, the model parameterizes the physical modifications. By simply ignoring the closed-loop of viewpoint sequence and frames, the model directly creating adversarial frames with compositing methods. Some approximated algorithms are used to ensure the model can be optimized by the gradient-based method. With the improvement above, the iteration speed of the model is greatly improved.\n\nStrengths:\n+ This paper proposes a highly scalable framework for designing physically realizable adversarial examples against end-to-end autonomous driving architectures, which makes the much stronger attack results.\n+ In the simulated climate settings, the proposed method demonstrates robustness against unforeseen variations in weather and visibility.\n+ With a small number of initial calibration running, the search for optimal parameters can be carried out with end-to-end gradient-based optimization, instead of relatively slower Bayesian Optimization.\n\nWeaknesses:\n+ It lacks of a convincing explanation about why ignoring the closed-loop of viewpoint sequence and frames the model can still work (in Section 2.2).\n+ The mask function is still not differentiable in most cases, thus approximation should be adopted. The authors should clarify how much such approximation affects performance.\n+ There are too many long sentences makes some parts of the article difficult to understand. Please try to improve the presentation\n\nI hope to hear the authors response regarding the weakness listed above during the rebuttal period.", "extra_scores": {}, "extra_text": {}}, {"review_id": "M_8zGEnK6Hg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To summarize, the authors propose a road-painting attack with rectangles to deceive a controller network such that the car will deviate from the correct trajectory. The simulation is done on CARLA.\n\n**The threat model**\nPainting roads with rectangles is very interesting. The closest one I saw is patching stop signs with rectangle markers [Eykholt 2018] as cited in the paper. Meanwhile, this setup brings many questions.\n\nFirst, the attack space is incredibly small (parameterized color rectangles on the road). With such a small space, I would expect the space allowed for changing the controller network output is also small. On the contrary, in traditional pixelwise adversarial attack, the attack vector is high-dimensional so the vulnerabilities can stack up to change the network output to arbitrary values. In the current setup, such stacking-up is unlikely to happen. Therefore, even the attack is successful, I believe tuning with gradient-based or gradient-free optimizers does no help much; the baseline attack success rate could be already high.\nIn other words, I doubt that even random rectangles may already cause deviations and infractions. Searching with BayesOpt or GradOpt may not help much; there could be a wide range of parameters that can cause reasonable deviations already.\n\nThis setup is also perceivable by humans and thus not stealth. In traditional adversarial attacks, the perturbations are small enough to be ignored by humans but will cause a deep network to fail; the current setup is not the case; therefore it can be easily defended by humans.\n\n**GradOpt v.s. BO**\n- BO is a black-box optimizer that has no access to the inner structure of the controller. GradOpt in this paper is a white-box model and it is unfair to compare BO with GradOpt.\n- GradOpt is a very standard gradient-based optimization pipeline for adversarial attacks involved with 3D projection and rendering.  Judging from the images, I believe the rectangles are not shaded, so an affine transformation already suffices. This limits the novelty of the method.\n\n**Constraints for the attack vector**\nI could not find the description of the constraints for the attack vectors. If there is no constraint; then there exists a trivial solution: just paint the road to a constant color using an infinitely large rectangle and there are no lanes anymore. It will be very interesting to see the evaluation metrics in this case just for ablation purposes.\n\n**Transferability**\nWith a small attacking space, I am curious whether GradOpt learns to find the vulnerabilities in the network, or learns to cover the important regions on the road. Also it is unclear whether it will succeed for real images and videos. The submission lacks transferability experiments to study those scenarios.\n\n[Update after reading authors' comments]\nThe critical issue is resolved: it seems their method is indeed better than plain BO, which in turn is better than random parameters. Though I still have a little doubt about how practical it can be in real scenarios, I increase the score to marginally accept.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to ICLR.  Overall the reviewers and I think that this paper presents some nice contributions to the adversarial attacks literature, demonstrating a low-sample-complexity, "physically-realizable" attack in a domain of clear importance and interest in machine learning.  The move to considering more "in the loop" adversarial examples is particularly compelling, and the threat model and improvement over BO methods are both compelling here.

The main downside of this paper, of course, is the fact that the "physical adversarial examples" are of course nothing of the sort: they are simulated.  Rather, they are just simulated in a manner that may plausibly be slightly more amenable to real-world deployment. The authors claim that they don't carry out an evaluation on a real system because it is "dangerous" is a bit overly dramatic: the tests could easily be carried out in a controlled environment, and demonstration on an actual physical system (even, e.g., and RC car) would vastly improve the impact of this work.  As it is, the paper is borderline, but ultimately slightly below the high bar set by ICLR publications.  I would strongly encourage the authors to reconsider the inclusion of the word "physical" in the title, as it honestly sets expectations high for a promise that the paper cannot deliver on, or (even better) to run real experiments on even a small physical system, demonstrating the transferability there.  The paper ultimately has the potential for a high impact in this field, if these issues are addressed.
Decision: Reject