Generating Transferable Adversarial Simulation Scenarios for Self-Driving via Neural Rendering
Authors: Yasasa Abeysirigoonawardena
Kevin Xie
Chuhan Chen
Salar Hosseini Khorasgani
Ruiting Chen
Ruiqi Wang
Florian Shkurti
Venue: CoRL 2023 Poster (2023)
Keywords: robotics
adversarial attacks
simulation

Abstract: Self-driving software pipelines include components that are learned from a significant number of training examples, yet it remains challenging to evaluate the overall system's safety and generalization performance. Together with scaling up the real-world deployment of autonomous vehicles, it is of critical importance to automatically find simulation scenarios where the driving policies will fail. We propose a method that efficiently generates adversarial simulation scenarios for autonomous driving by solving an optimal control problem that aims to maximally perturb the policy from its nominal trajectory. Given an image-based driving policy, we show that we can inject new objects in a neural rendering representation of the deployment scene, and optimize their texture in order to generate adversarial sensor inputs to the policy. We demonstrate that adversarial scenarios discovered purely in the neural renderer (surrogate scene)  can often be successfully transferred to the deployment scene, without further optimization. We demonstrate this transfer occurs both in simulated and real environments, provided the learned surrogate scene is sufficiently close to the deployment scene.

Generating Transferable Adversarial Simulation Scenarios for Self-Driving via Neural Rendering
Yasasa Abegsirigoonwwardenn", Kevin Xie Chuhan Chen', Salar Hosseini',
Ruiting Chen', Ruiqi Wang', Florian Shkurti 'University of Toronto, Vector Institute, 'Carnegie Mellon Universty. Stanford University, UofT Robotics Institute
Abstract: Self-driving software pipelines include components that are learned from a significant number of training examples, yet it remains challenging eval- uate the overall system's safety and generalization performance. Together with scaling up the real-world deployment of autonomous vehicles, it is of critical im- portance to automatically find simulation scenarios where the driving policies will fail. We propose method that efficiently generates adversarial simulation scenar- ios for autonomous driving by solving an optimal control problem that aims to maximally perturb the policy from its nominal trajectory. Given an image-based driving policy, we show that we can inject new objects in a neural rendering repre- sentation of the deployment scene, and optimize their texture in order to generate adversarial sensor inputs to the policy. We demonstrate that adversarial scenar- ios discovered purely in the neural renderer (surrogate scene) can often be suc- cessfully transferred to the deployment scene, without further optimization. We demonstrate this transfer occurs both in simulated and real environments, provided the learned surrogate scene is sufficiently close to the deployment scene.
1 Introduction
Safety certification of self-driving stack would require driving hundreds of millions of miles on real roads, according to [1], to be able to estimate miles per intervention with statistical significance. This could correspond to decades of driving and data collection. Procedural generation of driving simulation scenarios has emerged as a complementary approach for designing unseen test environ- ments for autonomous vehicles in a cost-effective way. Currently, generation of simulation scenarios requires significant human involvement, for example to specify the number of cars and pedestrians in the scene, their initial locations and approximate trajectories [2], well selection of assets to be added to the simulator. In addition to being challenging to scale, having human in the loop can result in missing critical testing configurations.
In this paper, we cast adversarial scenario generation as a high-dimenssonal optimal control prob- lem. Given known image-based driving policy that we want to attack, as well as the dynamics of the autonomous vehicle, we aim optimize photorealistic simulation environment such that produces sensor observations that are 3D-viewpoint--onnisten,, but adversarial with respect to the policy, causing it to deviate from its nominal trajectory. The objective of the optimal control prob- lem is to maximize this deviation through plausible perturbations of objects in the photorealistic environment.
Our optimal control formulation requires differentiation through the sensor model in order com- pute the derivative of the sensor output with respect the underlying state perturbation. However, most existing photorealistic simulators for autonomous vehicles are not differentiable; they can only be treated black boxes that allow forward evaluation, but not backpropagation. Instead of using an off-the-shelf photorealistic simulator and adding assets to match the scene, we train an editable
"ooresponndecce to yasasalcs toronto. edu
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Unperturbed
Random
Attack
Adversarial
(desloyment
Adversarial
XOSA
(NcRF)
Figure
(FPV) of our adversarial attack transfer to an RC car with overhead trajectory view the right. Row I: Unperturbed policy execution; Row 2: Random search texture attack; Row 3: Our adversarial attack directly transferred the real deployment scene, without additional optimization; Row 4: Our adversarial attack discovered in the surrogate NeRF simulator.
neural rendering model that imitates the deployment scene, allowing to insert new objects in the simulator and to optimize their texture through gradient- based optimization. This editable neural rendering model acts as a surrogate physics and rendering simulator, enabling us to differentiate through it in an efficient way order to attack the driving policy's input sensor observations.
Unlike many existing types of adversarial attacks in the literature [3, 4, 5J. work aims discover environment perturbationssattacks that satisfy the following properties: (a) They are temporally- consistent. The influence of the attack is not instantaneous, it is amortized through time via the optimal control loss function. (b) They are transferable. An attack discovered in the surrogate scene should ideally be adversarial the actual deployment scene. (c) They are object-centric. The attack introduces and edits objects as opposed to unstructured high-frreqency perturbations across all pixels. Specifically, we make the following contributions:
1. We formulate adversarial scenario generation across time as an optimal control problem that relies learned, surrogate NeRF simulator. The solution to this problem yields 3D- view-consistent, object-centric, adversarial attacks that often transfer to the deployment environment. We show how to solve this problem efficiently using implicit differentiation. 2. Differentiable rendering of our surrogate NeRF model enables gradient-based adversarial object insertion and scales to high dimensional parameterizations of multiple objects.
3. We show that our adversarial attacks discovered the surrogate NeRF simulator can be realized in the real-world and retain their ability to disrupt the policy.
We experimentally validate framework by reconstructing scenes using only pose-annotated im- ages and generate adversarial object insertion attacks with multiple trajectories.
Related Work
Adversarial scenarios for autonomous driving. Perceptual adversarial attacks make modifications to prerecorded sensor data from real driving sessions to fool the perception system. Since this sensor data is fixed, they lack the ability to resimulate and typically only operate on the individual frame level. Previous works, [4, 6] attempt to attack a LiDAR object detection module by artificially inserting an adversarial mesh on top of car rooftops objects in prerecorded LiDAR sequence. They extend the scope of their attack further by incorporating textures to be able attack image- based object detectors as well [3]. In both these works, the inserted object has a very low resolution

---

Images Deployment Soone
Altacks Transferred to Deployment Scene
Overhead View
Deployment Scene CARLA
Transfer Attack
Diferentiatle simulator Neural Renderel
Grodiant- based adversaral
Synthetic Scone Noural Renderer
Adverserial
View
Figure 2: Our method can be summarized in the four steps shown. (a) In the top left, we obtain posed images from the deployment scene which can be a simulator the real world. (b) In the bottom left, we reconstruct a surrogate scene by fitting NeRF to the posed images as differentiable simulator and observe only minor perceptual gap. (c) Having the surrogate scene, we can insert objects. which are also represented as NeRFs, and attack their color fields to generate textural attacks. (d) The discovered adversarial objects are introduced back into the deployment scene.
and nondescript geometry. Recent self-driving simulators, such as DriveGAN [7], GeoSim [8] and UniSim [9] address these issues, with the latter enabling manipulable sensor-based simulators based on prerecorded datasets. These works, however, have not dealt with discovering attacks.
Another prominent line of works produce dynamic state-level adversarial attacks. These gener- ally target the control/planning system only by perturbing trajectories of other agents in the scene. Without considering the perception system, these methods use simplified traffic and state-based sim- ulators that do not incorporate 3D rendering [10. 11, 12].
Closest to our work, a few methods have proposed to attack end-to-end policies by adding pertur- bations to existing self-driving simulators. In [13], the trajectories of other agents CARLA scene modified generate a collision event. Due the non-differentiability of the simulator, black-box Bayesian optimization is used. Gradient-bass -based attacks top of simulators have also been investigated. However, the requirement of differentiability has so far limited their scope to very simplified geometries that are composited post-hoc onto renderings from CARLA. In [5], flatly col- ored rectangles are composited on top of frames from the CARLA simulator and optimized to cause maximal deviation of an end-to-end image-based neural network steering controller. Similarly, work in [14] attempts to play a video sequence of adversarial images on billboard the scene using im- age composition. To knowledge, no works in this setting have been able to demonstrate transfer of adversarial attacks the real world, these attacks rely pre-existing simulator that they augment. Compared to these, our attacks are entirely performed is reconstructed from only posed images captured from any deployment scene. Furthermore, our surrogate neural simulator allows for inserting arbitrary objects reconstructed from posed images.
surrogate neural simulator that
The driving simulator VISTA [15] generates high fidelity sensor data using large collection of real world data. In our case, are able train NeRF using the data, allowing to generalize to wider range of novel views. [16] samples adversarial masking of existing LiDAR data using reinforcement learning. Work on perception error models [17] avoids using simulator altogether and instead focuses on learning surrogate policy that uses lower dimensional salient features, which are attacked. However, it would be very difficult to infer the real world perceptual disturbance that would cause the attack, so these attacks are very challenging to transfer to the real world.
Robust adversarial examples. Adversarial attacks for classification have commonly used minimal perturbations on the input images [18] that may always transfer to the physical world another
3

---

domain. To enhance robustness to domain transfer [19] proposes a class of adversarial transforma- tions that optimize for adversarial attacks under a distribution of image transforms [20].
Background
3.1 Neural Rendering
Neural 3D representations, such neural radiance fields (NeRF), have seen significant activity in recent years due to their ability to reconstruct real world objects and scenes to very high detail using only posed images. A survey of recent progress in neural rendering can be found [21].
In [22], physics simulations of known objects are combined with their learned NeRF to create high fidelity dynamic environments for training deep reinforcement learning agents that transfer to the real world. In our work, we use composition of NeRFs insert and optimize adversarial objects. This is shown in Fig. 3, and the details are in Sections 4. and 4.2. We render the scene using the volume rendering equation:
(w) 1/" o(x +
(w)+ (w) (x) (f. o(o + (w)at) L(x + lw; -w)dt
Where [xx w) is the intensity at a location I given in world space in the direction w. L,, and a are the learned color and density fields in NeRF, For the sake of performance, we choose to use grid-based volume representations. Structured grid NeRFs reduce computation cost by storing direct density and color variables [23] or latent features [24, 25] on explicit grids. essence, they trade extra memory utilization for large performance improvements. Instant Neural Graphics Primitives (iNGP) [26] uses multi-scale dense grids and sparse hash grids of features that are decoded color and density by MLP. We chose to use iNGP because it balances performance and memory tradeoffs well.
4 Method
Our framework generates successful adversarial attacks of end-to-end image-based self-driving poli- cies with only access to posed images from deployment scene. An overview of the high-level steps in our framework is shown in Figure
We now briefly describe the setting and our adversarial attack method. More details are included in Appendix B. Let 2, denote the state of the car at time t, x=" denote a reference trajectory to track and CTE the cross-track error. Starting from Eqn. 6, we set the cost function C(xx)) of our problem as the car's proximity to the reference x²:
C(a.) = (TT(()
In other words, we want to maximize deviation from the desired trajectory. We set the constraint function G(I,, Ft41. 0) be the following set of constraints:
U₁ = To (o:)
(3)
Or (xx)
(4) F++1 = fe(xs, (5)
Where и is the fixed driving policy", h is the neural rendering sensor model that outputs image observations on given the state of the car. The renderer depends on 0, the parameters of adversarial NeRF objects and 7. the fixed rendering parameters of the background scene NeRF, Finally, fc denotes the dynamics of the ego vehicle that must be considered, since we want find adversarial trajectories that are consistent across multiple frames.
4.1 Differentiable Renderer
Traditional simulators like CARLA do not admit computation of gradients. Thus, prior works rely on artificially compositing simplistic textured geometries on top of rendered images from CARLA and
"We train policy and provide details in Appendix C.2.

---

Driving Loop
neural rendering model h. o(31))
dynomics
driving policy
image observation
Ry
control
renderer
(e ue)
To (1)
Ot
(2(
object renderer
I+++1
realized
fro, Tt, ..., 2T)
trojectory
(())
optimization step
adversariol parometers
reference
{ aI: ...., 277
trojectory
devistion
Figure 3: A computation diagram of our algorithm for generating adversarial attacks. The inner driving loop consists of three components: the neural rendering model, the differentiable driving policy, and the differentiable kinematic car model. We inject the adversarial perturbation to the surrogate scene by composing the outputs of one or more neural object renderers (the single object case is shown above for simplicity) with the output of the neural scene renderer. The parameters of the object renderer(s) are optimized to maximize the deviation of the realized trajectory from the reference trajectory, while keeping the parameters of the driving policy and scene renderer frozen.
obtaining gradients with respect the composited alteration [14]. We NeRFs learn surrogate models of the scene and sensor model instead. This surrogate model not only gives automated method to reconstruct scenes from pose-annotated images, but also provides efficient gradient com- putation giving us a differentiable form for the sensor h. For the purposes of optimization, we found traditional NeRF representations to be intractable terms of compute and memory requirements (during gradient computation). Thus, we opt to use the multi-resolution hash grid representation, Instant-NGP [26].
Note that, similar existing work, detach the gradients of the image observation with respect to the camera coordinates (which are attached to the ego vehicle) [27]. We include more details regarding this in Appendix B.3.
4.2 Adversarial Object Insertion
We use insertion and texturing of multiple objects as our adversarial perturbations the back- ground scene. To do this, we first reconstruct regular objects, such as cars, individual NeRFs from pose-annotated images. For our object NeRFs simply store color values di- rectly on the voxel grids of Instant-NGP, which are tri-linearly interpolated within each voxel. By choosing these color voxel grids our ad- versarial parameters 0, we can perform inde- pendent adversarial texture attacks over multi- ple objects.
The object NeRFs can be easily composed with our background scene NeRF, This is done via alpha compositing, which leverages opacity and depth values that can be easily computed.
Figure 4: Base car the left; random texture in the middle; adversarial texture on the right.
4.3 Gradient computation via implicit differentiation
We use implicit differentiation for gradient computation [28], also known as the adjoint method, which enables constant memory gradient computation with respect to trajectory length. In discrete time, the adjoint method amounts to propagating gradients through an implicit relationship G for
5

---

problems of the form:
min T(0) = Σ ((x)) such that G1, 0
14,
(6)
Explicitly, the method performs forward simulation to compute the variables zy and then subse- quently backward pass to compute adjoint variables by solving the equations:
aC(x))
aG( Fe+1)
X
Ac+1
(7)
dz
dz,
Or
with the boundary condition:
aGl (G(2T-1, AT
(C(x))
(8)
OET
Oer
Finally, the gradient of the loss can be calculated as:
0G( x1. 0) azo
aG(I1-1,
VoJ =1
(9)
dru
00
08
Throughout both passes we do not need to store large intermediate variables and only need to accu- mulate the gradient at each step.
4.4 Gradient-based Adversarial Attack
Obtaining gradients for the problem in Eqn. (6) should possible with an autodifferentiation framework such as PyTorch [29]. We find that naively computing the gradient via backpropagation results in memory issues variables used to compute the integral in Eqn. 1 being stored until the end of the trajectory. We achieve drastic memory savings by using the adjoint method [30] which only keeps track of the adjoint variables À along the trajectory. In our case, the adjoint variables are dirce-dimensional, allowing us to only use as much memory as it takes to compute a single jacobian vector product of composition of models given by (5), (3), (4) in the optimization problem Eqn. (6).
scale up trajectory lengths due all the intermediary compute
To summarize, the computation of gradient-based based adversarial attack proceeds follows:
1. We rollout our policy in surrogate simulator to compute the loss and the trajectory T1T in Eqn. (6).
2. We perform a backward pass to compute adjoint variables for gradient computation.
3. Using the adjoint variables, we compute the gradient VoJ and update parameters A.
Experiments
To demonstrate the effectiveness of our framework, we aim to reconstruct driving scenario from posed images, generate adversarial attacks and validate that those attacks transfer the deployment scene. Through our experiments, we would like to answer the following key questions:
(Q1) Can gradient based optimization find better adversarial examples than random search?
(Q2) Are NeRF models suitable surrogate renderers for gradient based adversarial optimization?
(Q3) Are adversarial attacks transferable from NeRF back to the deployment domain?
5.1 Experimental Details
CARLA Deployment Scenes. We first validate our method in simulation, treating CARLA as a proxy for real deployment scene. We perform experiments on 3-way intersection of the CARLA [31] simulator. For the 3-way intersection, consider 3 different trajectories be fol- lowed by the ego vehicle. For the object models, train surrogate NeRF models for two sample objects, fire hydrant and small car using only posed images (any other object could be used). We

---

manually insert 2 small cars and 3 fire hydrants into the driving scene in an initial placement. Our adversarial attacks jointly optimize the NeRF color parameters and object rigid transforms.
Real World Deployment Scenes. Our real-world experiments are performed on an autonomous RC car driving around square race track in an indoor room. It is difficult to manufacture adversarial attacked objects with complex shapes in the real world. Hence, for practicality, object representing flat square texture pattern that can be projected by display monitor in the real world and optimize its color parameters. In order to create a first version of the attack we choose to directly compose the adversarial texture on to the robot camera feed. We then move on to a more difficult task of physically realizing this attack, for this we opted to display the texture on a monitor simplify lighting conditions. Additional details of our real world experimental setup are given in C.4.1
insert a NeRF
5.2 Evaluation Metries
We measure the effectiveness of an attack with our adversarial objective, the cross track error the ve- hicle. We use the road center as the reference and even unperturbed driving policy has some zero deviation which report under "Unperturbed" in Table 1. To characterize the insensitivity of our method to random seeds, we run 5 separate attacks per scenario for both the gradient-based and ran- dom attacks with different random initializations of the adversarial parameters. We report the mean and standard deviation of our metric.
Figure 5: Selected overhead views and snap- shots from adversarial deployment trajec- tories in the real world (top row: moni- tor displays adversarial texture discovered in NeRF), and in CARLA (bottom row: adver- sarial objects inserted in the simulator).
Our proposed method of attack is via gradient-based optimization using the method outlined in Section 4.4. The gradient-bused attack uses 50 iterations of optimization using Adam, with a learning rate of 0.1. Due to the high dimensional parameterization, detailed in B.3.1, bayesian opti- mization becomes computationally intractable. Therefore, as a baseline for our method, perform random search parameter attack on the NeRF surrogate model that samples parameters from a Gaussian distribution with mean zero and a standard deviation of 5. We chose this standard deviation to match the distribution over parameters we found in our gradient attacks. We use the same number of function evaluations, selecting the best achieved attack among the 50 random samples for the CARLA experiment. For real-world experiments, we didn't find much variation between random attacks in the surrogate simulator, showing the difficulty of random search in high dimensional parameter spaces.
5.3 Experimental Results
Example gradient attack trajectories are shown in Figure 5. We include more visualization of results for deployments of adversarial attacks, both in CARLA simulation the real world and preliminary results of retraining the CARLA policy using data, in Appendix D. In Table total cross track errors caused by our adversarial attack against the expert lane following controller.
compare the
We observe in all 3 CARLA scenarios (averaged over 5 seeds each) that our adversarial attacks using gradient optimization consistently produce significant deviation from the lane center. When transferring these attacks back into the deployment scene, we see that although the magnitude of the deviation is reduced, we still retain significant increase over the unperturbed or random search set- ting. The difference is likely due visual imperfections our surrogate NeRF simulator compared to the deployment scene. The random search perturbations are far less effective, remaining near the baseline unperturbed trajectory for out of the 3 cases.
7

---

CARLA Deployment
Surrogate Scene
CARLA Deployment
Scenario
Unperturbed
Random
Gradient
Random
Gradient
Straight Right
1166
1132 7
2347 + 49
1193 + 19
1702. - 160
1315
2084 + 10
4105 + 847
1476 + 12
2101. + 75
Left
1448
1460 8
4125 + 124
1158 - 163
2240. = 574
Physical Deployment
Surrogate Scene
Physical Deployment
Setup
Unperturbed
Random
Gradient
Random
Gradient
Green Screen
46
+
248 + 72
48
34 + 4
157 +
Monitor
47 3
76 - 48
Table I: Comparison of the total cross-track error for all the scenario tested. Results are shown for the following cases: (1) no attack in the deployment scene (unperturbed), (2) an adversarial attack (random or gradient) in the surrogate NeRF scene, (3) an attack in the deployment scene. We separate results from the CARLA and physical deployments, we show that gradients in our surrogate simulator are useful for finding adversarial attacks and these attacks remain effective when transferred to the deployment environment.
For the real world experiment, we observed a similar result. Random attacks consistently fail to elicit deviation from the driving policy both in the surrogate and deployment scenes. Over 5 random seeds, not a single random attack was able to cause the vehicle to exit the track. Gradient attacks on the other hand are reliably able find strong attacks with little variance the surrogate scene. attacks to the real world, find the attacks to retain their strength in the
When transferring green screen setup. The strength of the attack is relatively diminished when using the monitor to project the attack but is nonetheless consistently higher than the random attack and causes the vehicle to understeer and exit the track on occasion. We suspect this is due to the display properties of the monitor which can alter the appearance of the adversarial perturbation.
Limitations
Despite showing the ability to generate 3D-consistent adversarial scene perturbations, there are few avenues for improvement. First, we assume that the vision- based driving policy is differen- tiable. Recent works have shown high potential for modular end-to-end learned policies that could leverage synergies between vision and planning, such as neural motion planning [32], transfuser [33] and many others [34]. We discuss three potential methods to handle non-differentiable policies in Appendix E. Second, while do optimize for both adversarial textures and object poses, experiments in section D.3 of the Appendix show that the latter produces significantly non-smooth loss landscapes that necessitated multi-start gradient optimization methods to handle local minima.
7 Conclusion
We presented method for generating 3D-consistent object-based adversarial perturbations in au- tonomous driving scenarios. Unlike previous approaches that rely on making edits on top of fixed pre-recorded data or black-box simulators, we develop differentiable simulator directly with a neu- ral radiance field representation of geometry and texture of a scene that admits gradients through the rendering of camera and depth observations. Through alpha-compositing, we can introduce new objects also represented as neural radiance fields into the scene and optimize color perturbations of the objects. We validate framework both in simulation and on real-world RC race track driving scenario showing successful sim-to-real transfer of discovered attacks. While our particular implementation is only first step towards demonstrating NeRF based adversarial attack genera- tion. we believe that our framework shows promising new direction for automatic evaluation of autonomous vehicles. We expect our method to benefit greatly from continued improvements being made to neural rendering and their wider adoption for AV/roboicc simulation.
8

---

References
[1] K. Nidhi and S. M. Paddock. Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability? https: //ws:///wwww/ rand. org/pubs/ research_reportss 478 html, 2016. [Online; accessed 19-July-2018..
[2] A. C. Madrigal. Inside waymo's secret world for seliiiiinggsl--iliiii cars. The Atlantic, Aug
2017. URL ottp://s//////aaaaaaaaaaccc.oo.a
537648/.
[3] Tu, H. Li, X. Yan, M. Ren, Y. Chen, M. Liang, E. Bitar, Yumer, and Urtasun. Exploring adversarial robustness of multi-sensor perception systems in self driving, Jan. 2022. URL https: wtt./orrggaabooooooooo abs/a/as..1..////////////....... 067844
[4] Tu, M. Ren, Manivasagam, M. Liang, B. Yang, R. Du, Cheng, and Urtasun. Physically Realizable Adversarial Examples for LiDAR Object Detection, Apr. 2020. URL
attps://arzee org/ absp///b///// abs/ 00543.
[5] J. Yang, A. Boloor, A. Chakrabarti, X. Zhang, and Y. Vorobeychik. Finding Physical Adversar- ial Examples for Autonomous Driving with Fast and Differentiable Image Compositing, June 2021. URL htttttts///riiii org/ abs /2010.01000 08844.
[6] Y. Cao, C. Xiao, D. Yang, J. Fang, R. Yang, M. Liu, and B. Li. Adversarial objects against lidar- based autonomous driving systems. CoRR, has//907.004411 2019. URL http:://arx o org/abs 1910/9907..000 05418.
[7] S. W. Kim, J. Philion, A. Torralba, and S. Fidler. Drivegan: Towards a controllable high- quality neural simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5820-5829, June 2021.
[8] Y. Chen, F. Rong. S. Duggal, S. Wang, X. Yan, S. Manivasagam, S. Xue, E. Yumer, and R. Ur- tasun. Geosim: Realistic video simulation via geometry-aware composition for self-driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 7230-7240, June 2021.
[9] Z. Yang, Chen, Wang, Manivasagam, Ma, A. J. Yang, and R. Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 1389-1399, June 2023.
[10] Wang, A. Pun, Tu, Manivasagam, A. Sadat, Casas, M. Ren, and R. Urtasun. AdvSim: Generating Safety-Critich Critical Scenarios for Self- Diii--rivng Vehicles, Jan. 2022. URL https:// arxiv. mxxiv.or org. abs 2101 22111066600 06549.
[11] D. Rempe, J. Philion, I. Guibas, S. Fidler, and O. Litany. Generating useful accident-prone driving scenarios via a learned traffic prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17305 17315, 2022.
[12] M. Igl. D. Kim, A. Kuefler, P. Mougin, P. Shah, K. Shiarlis, D. Anguelov, M. Palatucci, B. White, and S. Whiteson. Symphony: Learning realistic and diverse agents for autonomous driving simulation, 2022. URL https:/ :ttptpp:://axii...////
arxiv..iv... wrw.wbs......... wwwpg/////......./////////// 2200 03195.
[13] Y. Abeyssriggonnwwardena. F. Shkurti, and G. Dudek. Generating adversarial driving scenarios in high-fidelity simulators. In 20/9 International Conference on Robotics and Automation (ICRA), pages 8271-8277. IEEE, 2019.
[14] N. Patel, P. Krishnamurthy, S. Garg, and F. Khorrami. Overriding Autonomous Driving Sys- tems Using Adaptive Adversarial Billboards. IEEE Transactions Intelligent Transportation Systems, 23(8); 1386-1 1396, Aug, 2022.

---

[15] A. Amini, T.H.. Wang, I. Gilitschenski, W. Schwarting, Z. Liu, S. Han, S. Karaman, and D. Rus. Vista 2.0: An open, data-driven simulator for multimodal sensing and policy learn- ing for autonomous vehicles. In 2022 International Conference on Robotics and Automation
(ICRA), pages 2419-2426, 2022.
[16] M. Koren, S. Alsaif, R. Lee, and M. J. Kochenderfer. Adaptive stress testing for autonomous vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1 1-7. IEEE, 2018.
[17] C. Innes and S. Ramamoorthy. Testing rare downstream safety violations via upstream adaptive sampling of perception error models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 12744-12750. IEEE, 2023.
[18] N. Akhtar, A. Mian, N. Kardan, and M. Shah. Advances in adversarial attacks and defenses in computer vision: A survey, Sept. 2021. URL https://arxii orgg httg//bb//2100 00401.
[19] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 284-293, 2018.
[20] Buddareddygari, T. Zhang, Y. Yang, and Y. Ren. Targeted attack on deep rl-based au- tonomous driving with learned visual patterns. In 2022 International Conference Robotics and Automation (ICRA), pages 10571-10577, 2022.
[21] A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, Xu, T. Simon, M. Nießner, E. Tretschk, L. Liu, B. Mildenhall, P. Srinivasan, R. Pandey, S. Orts-Escolano, S. Fanello, M. Guo, G. Wetzstein, Y.. Zhu, Theobalt, M. Agrawala, B. Goldman, and M. Zollhöfer. Advances in neural rendering. ACM SIGGRAPH 2021 Courses, SIGGRAPH '21, 2021.
[22] A. Byravan, J. Humplik, L. Hasenclever, A. Brussee, F. Nori, T. Haarnjja, B. Moran, S. Bohez, F. Sadeghi, B. Vujatovic, and N. Heess. Nerf2real: Sim2real transfer of vision-guided bipedal motion skills using neural radiance fields, 2022. URL https:/ //arxi////iii// org/ abs 22220221 C 04932.
[23] Fridovich- Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radi- ance fields without neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5501-5510 2022.
[24] L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651-15663, 2020.
[25] A. Chen, Xu, A. Geiger, Yu, and H. Su. Tensorf: Tensorial radiance fields. European Conference on Computer Vision (ECCV), 2022.
[26] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with mul- tiresolution hash encoding. ACM Transactions on Graphics, 41(4):1-15, July 2022.
[27] N. Hanselmann, K. Renz, K. Chitta, A. Bhattacharyya, and A. Geiger. King: Generating safety-critical driving scenarios for robust imitation via kinematics gradients, 2022. URL org. abs 2204. 13683.
[28]
L. S. Pontryagin. Mathematical theory of optimal processes. CRC press, 1987.
[29]
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035 Curran Associates, Inc., 2019.
10

---

[30] K. M. Jatavallabhula, M. Macklin, F. Golemo, V. Voleti, L. Petrini, M. Weiss, B. Con- sidine, J. Parent-Levesque, K. Xie, K. Erlehen, L. Paull, Shkurti, D. Nowrouzezahrai, and S. Fidler. gradsim: Differentiable simulation for system identification and visuomo- tor control. International Conference on Leurning Representations (ICLR), 2021. URL
https: https://openreview..u net/ forumfirrooooooo Leg. BBK
[31] A. Dosovitskiy, Ros, F. Codevilla, A. LoLoz and V. Koltun. CARLA: An open urban driving simulator. In Proceedings of the st Annual Conference Robot Learning, pages 1 1-16, 2017.
[32] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8660 8669, 2019.
[33] K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger. Transfuser: Imitation with transformer-bant sensor fusion for autonomous driving. Paltern Analysis and Machine Intel- ligence (PAMI), 2022.
[34] A. Tampuu, T. Matiisen, M. Semikin, D. Fishman, and N. Muhammad. A survey of end-to- end driving: Architectures and training methods. IEEE Transactions on Neural Networks and Learning Systems, (3(4):1364-11) 2020.
[35] M. Tancik, V. Casser, X. Yan, S. Pradhan, Mildenhall, P. Srinivasan, J. Barron, and H. Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8248 8258, 2022.
[36] Z. Xie, J. Zhang, W. Li, F. Zhang, and L. Zhang. S-neRF: Neural radiance fields for street views. In International Conference on Learning Representations, 2023.
[37] A. R. Kosiorek, H. Strathmann, D. Zoran, P. Moreno, R. Schneider, S. Mokrá, and D. J. Rezende. Nerf-vae: A geometry aware 3d scene generative model, 2021. URL https: //ar //arxiv.org/abaa........ /ttpr///aba////// 04. 00587.
[38] M. Niemeyer and A. Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
[39] B. Yang, Zhang, Xu, Y. Li, Zhou, Bao, G. Zhang, and Z. Cui. Learning cbject-compositional neural radiance field for editable scene rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13779-13788, 2021.
[40] S. Benaim, Warburg, P. E. Christensen, and S. Belongie. Volumetric disentanglement for 3d scene manipulation, 2022. URL http:///wwwww...oa///a///a/saa// htwpg//bb///2/0 02776.
[41] A. Mirzaei, Т. Aumentado K. G. Derpanis, J. Kelly, M. A. Brubaker, I. Gilitschen- ski, and A. Levinshtein. Spin-nerf: Multiview segmentation and perceptual inpainting with neural radiance fields, 2022. URL https: / /ttp/vawivvv///a///vv///////aaal wrg. abs 2211. 12254.
[42] P. P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron. Nerv: Neu- ral reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7495-7504, 2021.
[43] W. Ye, S. Chen, C. Bao, H. Bao, M. Pollefeys, Z. Cui, and G. Zhang. Intrinsicnerf: Learning intrinsic neural radiance fields for editable novel view synthesis, 2022. URL https:// arxiv. org. abs /2t../211..0 00647.
[44] Y. Xu, M. Chai, Z. Shi. S. Peng, S. Ivan, Aliaksandr, C. Yang, Y. Shen, H.-Y. Lee, Zhou, and T. Sergy. Discoscene: Spatially disentangled generative radiance field for controllable 3d-aware scene synthesis, 2022. URL pattts::/aaxxie .ogg/oorg/ooooo abs/22l2 11984.
11

---

[45] A. Kundu, K. Genova, X. Yin, A. Fathi, C. Pantofaru, L. J. Guibas, A. Tagliasacchi. F. Del- laert, and T. Funkhouser. Panoptic neural fields: A semantic object-aware neural scene rep- resentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12871 12871-, 2022.
[46] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin. Inerf: Inverting neural radiance fields for pose estimation. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), page 1323-1330, 2021.
[47] M. Adamkiewicz, T. Chen, A. Caccavale, Gardner, P. Culbertson, J. Bohg, and M. Schwa- ger. Vision-only robot navigation in neural radiance world. IEEE Robotics and Automation Leiters, 7(2):460-44613, 2022.
[48] L. Cleac"h, H. Yu, M. Guo, T. A. Howell, R. Gao, J. Wu, Z. Manchester, and M. Schwager. Differentiable physies simulation of dynamics-augmented neural objects, 2022. URL https: //arxiv. org/ abs 2210. 09420.
[49] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics with compositional neural radiance fields, 2022. URL RtLhttps:// /tatttps///axii...om arxiv. org/ abs/ 2202. 11855.
[50] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects, 2022. URL
[51] B. O. Community. Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http: www.bblender... org.
[52] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp. P. Goyal, L. D. Jackel, M. Monfort, U. Muller, Zhang, et al. End to end learning for self-driving cars, 2016. URL https: //axxiv// org/ abs 604.04440 07316.
[53] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[54] F. Codevilla, M. Müller, A. Dosovitskiy, A. M. López, and V. Koltun. End-to-end driving via conditional imitation learning. CoRR, abs/1710.02410, 2017. URL WRL.http///axxi..rre abs/ / 11100.1100 02410.
[55] L. Schönberger and J.-M. Frahm. Simettueefrrom-motion revisited. In Conference on Com- puter Vision and Pattern Recognition (CVPR), 2016.
[56] J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016.
[57] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson, and G. Gkioxari. Ac- celerating 3d deep learning with pytorch3d. arXiv:20077.0850/ 2020.
[58] Shirobokov, V. Belavin, M. Kagan, Ustyuzhanin, and Baydin. Black-box optimiza- tion with local generative surrogates. Advances in Neural Information Processing Systems, 33: 14650-14662, 2020.
[59] Z. Wang, M. Zoghi, F. Hutter, D. Matheson, N. De Freitas, et al. Bayesian optimization in high dimensions via random embeddings. In IJCAI, pages 1778-1784, 2013.
[60] J. Wu, M. Poloczek, A. G. Wilson, and P. I. Frazier. Bayesian optimization with gradients, 2018.
12

---

[61] H. J. T. Suh, M. Simchowitz, K. Zhang, and R. Tedrake. Do differentiable simulators give better policy gradients?, 2022.
[62] P. Vicol, L. Metz, and Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies, 2021.
[63] Schulman, N. Heess, Weber, and Abbeel. Gradient estimation using stochastic compu- tation graphs. Advances neural information processing systems, 28, 2015.
[64] R. Astudillo and P. I. Frazier. Thinking inside the box: A tutorial on grey-box bayesian opti- mization. In 202/ Winter Simulation Conference (WSC), pages 1-13. IEEE, 2021.
[65] M. Vlastelica, A. Paulus, V. Musil, G. Martius, and M. Rolínek. Differentiation of blackbox combinatorial solvers. arXiv preprint 1rXiiv1112.02/111 2019.
[66] A. Agrawal, B. Amos, S. Barratt, S. Boyd, Diamond, and Z. Kolter. Differentiable convex
optimization layers. Advances neural information processing systems, 32, 2019.
13

---

A Appendix: Additional Background
A.1 Neural Rendering
Differentiable rendering. NeRFs represent scenes as emissive volumetric media [26]. Unlike surface rendering. volumetric rendering does not suffer from explicit hard discontinuities, which difficult to handle for traditional surface rendering methods[21]. We exploit the differentiable volume rendering of NeRFs, to robustly compute efficient gradients for arbitrary geometries.
Complex scene reconstruction. Works such as BlockNeRE [35] and S-NeRF [36] show great po- tential for automatically capturing street-level scenes relevant to autonomous vehicle simulation. Unlike traditional simulators, these neural representations are directly trained on raw sensor cap- tures, thereby obtaining high-fidelity visual reconstruction without laborious asset creation.
Composition and editing. Recent works have extended the static single scene setting of NeRF to composition of NeRFs, scene disentanglement, as well as editing and relighting. Specifically, [37] encodes scenes with latent codes from which new scenes can be generated. [38]. [39] and [40] intro- duce compositional radiance fields to represent different objects and realize scene decomposition. [41] utilizes segmentation information to perform 3D scene inpainting. [42] and [43] decompose color into different illumination components. [44] [45] learn priors from big datasets of images to disentangle existing scenes.
Control and neural rendering models. Neural rendering has seen utility in a few different opti- mization tasks for robotics such as pose estimation [46]. NeRFs have also seen direct application to trajectory optimization, including utilizing NeRF's density as approximate occupancy [47], col- lision, and friction constraints [48], effectively allowing NeRF act as differentiable physics simulator. On the other hand, [49] learns additional latent dynamics of NeRF objects. In [22], physics simulations of known objects are combined with their learned NeRF to create high fidelity dynamic environments for training deep reinforcement learning agents, that can transfer to the real world. We use composition of NeRFs in our work to insert and optimize adversarial objects. This is shown in Fig. 3, and the details are Sections 4.1 and 4.2.
B
Appendix: Method Details
B.1 NeRF
Volume Rendering A neural radiance field consists of two fields, (x(x), (I) that encode the density a at every location x and the outgoing radiance at that location the direction w. In NeRFs, both of these functions are represented by parameterized differentiable functions, such as neural networks. Given a radiance field, we are able to march rays through an image plane and reconstruct a camera image from given camera pose and intrinsic matrix using the rendering function reiterated here for clarity:
((x = ("
o(t) o(1))x exp (.. ((o()))) L(t, -w)dt
(B.1)
Where I(I,,) -) and ol) are shorthands for Tllw + .) and a alw x), and T(x, w) is the intensity at location J given in world space in the direction w.
Compositing For our adversarial attacks to contain 3D semantics, it is crucial to insert the pertur- bation in 3D aware manner. For this we utilize another feature of neural radiance fields, which is to output opacity values. Specifically, in Eqn. (1) we can extract the transmittance component, which acts measure of the pixel transparency
a(x,w) = exp (% a(aat)
(B.2)
14

---

Furthermore, we can replace the radiance term with distance in (1) to extract the expected termina- tion depth of ray 2:
x(x,w)= J.' tt(())(tt)dt
(B.3)
We consider the case of two radiance fields, the object radiance field oo, Lo and the background radiance field o,, L. We use a transformation matrix to correspond ray coordinates between the scene and the object radiance field.
By applying equations (1), (B.2), (B.3) single ray that corresponds both the base scene and the object radiance field, we obtain the values Co. Coa. Zoi Ca. as. 2, respectively, where is the opacity and 2, is the depth along the ray. We denote the foreground and background values at pixel as
arg min( 20)
(B.4)
0,
6 = arg max( Zar 2o) 0,8
(B.5)
The final blended color is then given by:
agey + 1 a1l-a))ac
(B.6)
af ab(l az)
In the case of multiple object NeRFs, we simply repeat the alpha blending for each object com- posite them all into the same scene.
B.2 Vehicle Dynamics
The dynamics in equation (5) can take multiple forms, for the CARLA experiments, we choose the simplest kinematic model of car, a Dubin's vehicle:
c cos 0
i
U sin 0
(B.7)
For the purposes of the CARLA deployment environment, we find that is sufficient to consider the kinematic model with fixed velocity, and only angular control. Thus, our imitation learning policy in Eqn. (3) only outputs steering commands. We note that approach is applicable any dynamics model, long is differentiable.
For the real world experiments, we opted for fixed velocity Ackerman steering model:
0 $ = v sin 0 y lan(0)
(B.8)
where is the robot wheelbase.
B.3 Optimization Details
As described in Section 4.1, following prior work, we do not propagate gradients of camera param- eters through the sensor model function. Specifically, we set,
0₂ = h, (tttopggradeet((:)))
(B.9)
Thus gradients of the observation will only be taken with respect the adversarial object parameters 0 and not the state of the car. The gradient with respect x, corresponds exploiting higher order effects of how the observation would change if the car was looking in slightly different direction due to previous steps of the attacks, and leads to very non-smooth loss objective that is not useful for finding practical attacks.
For experiments in the real world, we found the attacks were sometimes very sensitive to the robot's pose. To alleviate this issue, we chose optimize multiple randomly sampled initial poses simul- taneously. The samples were normally distributed around the nominal car starting location, with standard deviation of 0.1.
15

---

B.3.1 Optimization parameters
In all our experiments, our optimization parameters 0 correspond to values on the NGP voxel grid. Since we have removed the decoder, the grid values directly correspond to the color for given position in the volume. Due to this, the parametrization even for small models can get quite large, the order of million for the hydrant.
C Appendix: Experimental Details
C.1 NeRF Models
When training the surrogate NeRF models of the background scene and objects, we the default Instant-NGP hyperparameters and optimize over 50 epochs using the Adam optimizer.
The source 3D assets for our objects were obtained from the Objaverse dataset [50] and posed images produced by rendering with Blender[51]. For our object models, we choose to use Instant- NGP without a decoder, instead directly encoding the colour values in the feature grid. Furthermore, remove view dependence for better multi-view consistency. Finally, we use lower resolutions for the object feature grids compared to the scene feature grids. The object feature grids contain resolutions up to 128 and 64 features for the car and hydrant, respectively. Since adversarial objective does not have any smoothness constraint, we found it critical to use lower resolution grids and remove the positionally encoded feature decoders to avoid aliasing effects.
C.2 Driving Policy.
We train our own policy on which the attack will be performed. Our policy is an end-to-end RGB image neural network policy and the architecture is taken from [52]. We make a slight addition to goal condition the policy by adding a goal input to the linear component and increasing the capacity of the linear layers. The policy is trained via imitation learning, specifically DAgger [53], [54].
Expert actions are given by lane following controller tuned for the simulator that gets access to the ground truth state, unlike the policy. The expert queried from various states random distances from the center of the road to recover from. Furthermore, random noise augmentation is used on the images during training to make the policy more robust to noisy observations.
C.3 CARLA
We fit the background scene model using a dataset of 1800 images and their corresponding camera poses, which provide a dense covering of the CARLA scene.
When transferring attacks back to the deployment scene, opacity values usually not available. In order to evaluate our attacks, we assume that objects are opaque (a 1), and thus our method of blending in Equation B.3 can be calculated using just the depth and color values. We observe from experiments on the CARLA simulator that this type of composition sufficient for the evaluation in the deployment environment.
Driving Policy. For our driving policy the initial training dataset of images is collected from the intersection in CARLA. We further fine-tuned the policy with some additional data collected from our surrogate simulator to ensure that our policy is not trivially failing due to slight visual differ- ences. We use a total dataset of 120000 images CARLA and 60000 images in the surrogate simulator in order to train the policy. We validated our policy on hold out validation set consisting of 12000 images captured purely from the surrogate simulator. All data were collected by running the expert on the reference trajectories. The policy was trained using behaviour cloning, where we gave examples of recovery from deviation by collecting data from random start locations around the nominal trajectory.
16

---

Figure C.I: Picture of driving area for the real world scenario experiments.
C.4
Real World
We fit the background to a room in the real world using a dataset of 2161 images captured from an iPhone camera at 4K resolution. We collect data covering the room by walking around, then attach the iPhone to the robot to collect further data from the driving view points. The captured videos are processed using COLMAP [55, 56] for both camera intrinsic and the poses.
Driving Policy. We train driving policy to track square track in the room marked by green tape, this policy was trained using an expert PID) controller with global positioning supplied by the VICON system providing 9584 images. We further augment this again with 12000 images from driving data in the NeRF scene. An overview of our working area is given in Figure C.I.
For all real world attacks we optimize the color of cube in the surrogate NeRF scene, placed at one of the corners such that the camera will encounter this cube as the car takes the turn.
C.4.1 Robot
We carry out experiments using the RACECAR/J platform. The robot equipped with a ZED stereo camera, of which we only utilize the RGB data from the left sensor, which has been configured resolution 366x 188 at 10 frames per second. We operate the robot inside VICON system that positions the robot rate of 50Hz streaming through a remotely connected computer that runs policy well the image processing for some of the attacks.
C.4.2 Green Screen Attack
For the green screen attack, we utilized a VICON system to accurately position both our robot and the green screen target. Using the green screen target position, well as the camera parameters, we project one face of the cube on the input image to the policy. We opt overlay the cube in such manner keep the policy driving in real time and to ensure that there is penalty control frequency. The image compositions is done the remote computer where the controls are computed, which are then sent wirelessly to the robot to execute.
C.4.3 Monitor Attack
To replace the green screen with a physical object, we place a monitor and display the same attack above the monitor. We place the monitor in location such that it is visually consistent with the NeRF and green screen attacks. For the monitor attack, utilize 27-inch monitor with 16:9 aspect ratio. Since the adversarial objects optimized in earlier examples are cubes we only use the center of the monitor to display the attack.
'bttss//tacecarj.com/
17

---

Figure D.1: The performance of the driving policy before (left) and after (right) retraining the discovered adversarial scenarios.
CARLA
Attack Transfer in CARLA CARLA Attack After Retraining
Scenario
Unperturbed
Random
Gradient
Gradient
Straight 1166
1193 + 19
1702. + 160
1250
Right 1315
1476 + 12
2101. 75
1307
Left
1448
1158 + 163
2240. 574
1419
Table 2: Comparison of the total cross-track error for the retraining experiment over the different trajectories. Results are extending the results from the main paper Table1 shown for the following cases: (1) attack in CARLA (unperturbed), (2) an attack in the CARLA scene, (3) attack in the CARLA scene after the driving policy is retrained using adversarial data.
D Appendix: Additional Experimental Results
D.1 Incorporating Discovered Adversarial Scenarios in the Training Set
Our primary focus this paper was to discover adversarial attacks for the evaluation of pretrained self-driving policy. Here we perform some preliminary investigations on fine-tuning our self-driving policies. on the old data and the adversarial attacks we found. Specifically, we take the attacks discovered by the gradient-based optimization and use them collect additional imitation learning data. The collection is performed in the CARLA simulator using the depth compositing approach to insert the adversarial objects, as was done for the evaluation in the main paper. Apart from the object compositing, the data is collected in the same way as the original CARLA data used to train the base policy. We collect 24000 total frames over three trajectories with two different starting points. After fine-tuning our policy on the combination of the original dataset and the new adversarially augmented dataset, we evaluate the fine-tuned agent in the same scenario. We visualize the trajectories of the fine-tuned policy in Figure D.I and report the total deviation compared to before fine-tuning in Table 2. We find that the policy is no longer susceptible to the adversarial attacks, even though the initial starting position for evaluation unseen during training.
18

---

Figure D.2: Sample renderings the left turn trajectory with the adversarial perturbations in CARLA from the ego vehicle's point of view. Four different snapshots from the evolution of the trajectory are shown.
D.2 CARLA Visualizations
We show first person visualizations of our discoverered adversarial attacks inserted back into the CARLA deployment simulator in Figure D.2. We note the smoothness of the texture discovered by our method. Purely perceptual single-frame attacks typically exhibit much higher frequency texture.
We show additional overhead trajectory views of adversarially attacked trajectories from one CARLA scene in Figure D.3.
D.3 Object Translation Attacks
We find in practice that the loss landscape with respect to the poses of inserted objects are extremely non-smooth, as seen in Fig. D.4 We therefore investigated mixed attack in NeRF where use multi-start (multi-seed) gradient optimization for the object poses and single-seed gradient optimiza- tion for the adversarial textures. For this experiment, we used more robust policy trained with the data from our color attacks, as well as additional examples of the adversarial objects in random poses. We report the results in Table 3. During the course of optimization we randomly sample set poses within the constraints, then we perform 10 gradient descent iterations to refine the candidate solutions. We keep the best solution across 50 total evaluations. We see that even with more robust policy, by combining gradients and multi-seed sampling significant and transferable adversarial scenarios that incorporate both textures and object poses. We note that the straight trajectory is particular robust after the retraining, which causes the new gradient attacks to be less effective. We observe that modifying the poses of adversarial objects in addition to their textures allows the attacks to transfer even better to the deployment CARLA scene.
able discover some
CARLA
NeRF
Attack Transfer in CARLA
Scenario
Unperturbed
Multistart Gradient
Random Only
Multistart Gradient
Random Only
Straight
1248
1377 + 87
1265 + 21
1194 + 10
1164 + 23
Right
1648
2682 + 175
2529 169
2707. + 342
1981 170
Left
1353
1523 + 27
1476 + 27
1808 + 401
1792 + 447
Table 3: Total cross track error for attacking CARLA policy with random and gradient combined.
19

---

(c) Transferred Figure D.3: Overhead views of three distinct trajectories driven by the policy. (a) shows the policy driving behavior in CARLA when no adversarial perturbation introduced. (b) shows the policy driving behavior in the surrogate simulator with the discovered adversarial perturbation. (c) shows the same perturbation transferred to the deployment scene.
(a) Unperturbed
(b) Attacks in NERF
Creas Track Deviation Object Poation
3000
1m
1300
3200
1000
1000
-0.25
0.00
0.05
0.10
Object Normalaaed Yspcattonn
Figure D.4: Loss landscape as a function of car position for the straight scenario
20

---

CARLA
Scenario
Differentiable Rasterization
Straight
1647 = 38
Right
1727 + 41
Left
3397 + 180
Table 4: Results of optimizing the texture of an adversarial mesh directly using PyTorch3D differ- entiable rasterization
Figure D.5: Overhead views of the driving with adversarial PyTorch3D objects.
D.4 Differentiable Rasterization for Adversarial Textures
In this section, perform investigation methods for the inserted objects. Instead of carrying volume rendering, to carry out differentiable rasterization for our adversarial objects. We optimize the textures of car and hydrant models as in our experiments on Instant-NGP, however, we now render the objects us- ing PyTorch3d directly on top CARLA background. We report the mean and standard deviation from random seeds table 4. Figure D.5 shows selected overhead views of the adversarial tra- jectories. We observe that using differentiable rasterization we are able obtain results similar to volume rendering. However, there still needs be work done determine how to differentiable rasterization to generate adversarial attacks that transferable the real world.
method using different differentiable rendering
PyTorch3D[57]
D.5 Real-world Visualizations
We show aligned visualizations of the same adversarial real-world monitor attack in Figure D.6.
E
Limitations
E.1 Non-differentiable policies
In our work, we indeed assume that the visual driving policy is end-to-end differentiable. Recent work has shown the potential of modular end-to-end learned policies that can optimize the interac- tion between perception and planning [32, 34].
(a) Surrogate Simulator
(b) First person view Figure D.6: Real-world adversarial monitor attack visualizations.
(c) Third person view
21

---

However, our current method does have clear limitation when it comes to non-differentiable poli- cies. Here we outline a few approaches to address this issue. the policy entirely black-box, then just as we have learned a differentiable surrogate scene to approximate the true simulator, may possible to also learn differentiable surrogate policy to approximate non-differentiable policy [58]. Otherwise, we may need resort to zeroth order optimization. This would prove computationally challenging for high-dimensional parameter spaces, but methods such as sparse Bayesian optimization could be applicable [59]. We can also leverage gradient information for Bayesian Optimization [60], recent work on combined zeroth-order and first-order gradient estimators, such as [61] or [62].
behavior of
If we consider the structure of the policy, most modern driving pipelines will still contain large chunks of end-to-end differentiable components, such perception modules, together with differentiable components. In these cases, the policy would be mixed computation graph where some parts are differentiable and others are not. Gradient estimation in mixed stochastic compu- tation [63] graphs has been explored by many works in the contexts of probabilistic programming, variational inference and RI,, and could be adapted for stochastic optimization methods. Gray-box bayesian optimization [64] also considers splitting up functions into constituent parts, and tech- niques for leveraging gradient information for BO exist as well.
Although parts of the driving policy may not be trivially differentiable via standard backpropaga- tion, there are also many techniques (such as implicit differentiation and informed perturbations) proposed for obtaining gradients of algorithms including combinatorial [65] and convex optimiza- tion [66] that may be present in the self-driving policy. These could account for many of the "non- differentiable" components of modern driving policies, and is worth exploring the applications of such techniques to them.
22


Reviews:
[{"review_id": "ndr8mvn7S6", "reviewer": "Reviewer_Hzbf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper takes recent work on differentiable neural rendering, and applies it to the task of generating adversarial scenarios for self-driving cars.\n\nIt does so by taking an differentiable image-based control policy, feeding it image observations from a neural rendering model and generating a trajectory via a simulator. The realized trajectory is then compared to a reference trajectory, and an optimization step attempts to adversarial change the parameters of the neural renderer such that the cross-track error between the reference and realized trajectories is maximized.\n\nTheir experiments (On both simulated CARLA deployments and a real-world RC track) show that their system is able to consistently find stronger adversarial examples than random-search (in terms of maximizing cross-track error), though attack strength diminishes when transferring results to a real-world monitor. ", "summary_of_recommendation": "This paper shows a sensible application of NeRF technologies to a downstream AV-robustness task. While each of the individual components of their pipeline are not necessarily novel in isolation, and while the adversarial images produced are not \"realistic\" in the sense of what might be seen on a real highway, I think this is a decent first step in showing how such neural renderers have promise in terms of improving robustness for image-based control.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Listed above in the review. The paper is promising, and my main questions generally ask the authors to discuss how they see they work as fitting in to the broader landscape of safety-testing and assurance within the Automated-Driving domain (rather than solely \"Neural Network Robustness\").", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well written and easy to follow - the figures and equations are clearly laid out and allow the reader to understand quickly their pipeline.\n\nWhile none of the individual components are technically \"novel\" in isolation, the combination of recent neural surrogate techniques, adjoint gradient computation, and adversarial simulation are combined in a way that I think makes a sensible contribution to assessing image-based AV safety.\n\nThe related literature does a decent job placing the work within the context of adversarial sensor attacks, and general adversarial robustness. However, I wonder if the scope is a little too narrow? Ultimately this work is about safety and risk, in the sense that it is concerned with how an upstream image/sensor perturbation can affect downstream trajectory control. For this reason, I would have liked to have seen how the authors envision the work as fitting in to the wider story of perceptual-control safety and testing. For example, with respect to [Perception Error Models](https://ieeexplore.ieee.org/abstract/document/10161501), [Adaptive Stress Testing](https://ieeexplore.ieee.org/abstract/document/9981724), [Cyber-Physical System Falsification](https://arxiv.org/abs/1703.00978) etc.\n\nThe authors differentiate their work from other forms of adversarial attacks by the fact that their perturbations are:\n\n- Temporally consistent\n- Transferable\n- Object Centric\n\nThis is commendable. However, I was left wondering if all this is really a sufficient description for \"Realistic\" attacks? Ultimately, from the images provided, it seems like the attacks that their neural renderer produces take the form of rainbow-colored plasma-like billboards. How do the authors see their work in the context of papers which attempt to make AV-perception systems more robust to types of corruption that are [actually likely to manifest in deployment](https://users.ece.cmu.edu/~koopman/pubs/pezzementi18_perception_robustness_testing.pdf)?\n\nSimilarly in the experiments - I think the work here is cleanly laid out and does a good job demonstrating the merits of their approach. However, in a broader context, is \"maximizing cross-track error\" the same as \"making the trajectory unsafe\"? The author's somewhat allude to this as the \"true\" goal when they talk about the significance of perturbations and whether the attack \"causes to vehicle to under-steer and exit the track\". I wonder if the work could benefit from relating it to work on more [Risk-Aware](https://papers.nips.cc/paper_files/paper/2022/hash/40739b3bb584c117b3e2f418d17f63a1-Abstract-Conference.html) notions of attack?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "iqNQQ9moZk9", "reviewer": "Reviewer_i9eu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a pipeline to generate adversarial examples for end-to-end self-driving policy. Specifically, it first builds a neural radiance field of the streets as a surrogate, differentiable, photorealistic simulator. Then, it solves an optimal control problem in order to solve the perturbation that could lead the policy to deviate from its original route. In the experiments, the authors demonstrate that the proposed attack can be transferred to the real-world environment.\n", "summary_of_recommendation": "I think the originality of the paper is good and hence recommend a weak accept. I believe a rewriting of section 3 and clarification of the limitation will make the paper a good contribution to the community.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Does the proposed method assume an end-to-end, differentiable, neural network policy?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n### Significance & Originality\nI like the idea of using NeRF as a differentiable renderer for improving the robustness of self-driving policies. To the best of my knowledge, this is a novel application of NeRF. In the future, this method can potentially become very practical as more and more teams are using neural rendering to build photorealistic simulators.\n\n### Figures\nI like Figures 1-3. They illustrate the results, method, and system clearly.\n\n### Quality\nI think the proposed method is technically sound and claims are supported with experiments.\n\n## Weaknesses\n### Limitation\n- Does the proposed method assume an end-to-end, differentiable, neural network policy? If so, I think this is a major limitation that is not mentioned in section 6. Lots of established methods for self-driving are not necessarily fully differentiable.\n\n### Writing\nI think the writing can be drastically improved. \n- In section 3, line 101 - line 132 can be safely skipped and instead, just say we use instant-NGP with equation A.1 for rendering. Also, authors should consider moving equation A.1 to the main text as it is later referenced in line 188. This way, readers don’t need to go to the appendix to search for the equation. \n- Lines 134-140 can be moved to section 4.3 when it is actually about to be used. Also, by deleting lines 101-132, authors can move more details of the implicit differentiation from the appendix to the main text so that readers can actually understand the mechanics behind the scene instead of just having rough ideas.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "59-SGwa_av", "reviewer": "Reviewer_qikM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper addresses the issue of adversarial scenario generation in the context of autonomous driving. The approach taken involves optimizing the textures and poses of objects in the scene to maximize the deviation of the ego vehicle from desired trajectories. Instead of developing a differentiable simulator, the work leverages NeRF as a neural renderer/simulator. The authors validate their method by testing it in CARLA and also on a toy car, demonstrating the transferability of the generated attacks to target domains.", "summary_of_recommendation": "The interesting direction of work and demonstration on a real robotic platform make me inclined toward accepting the paper.\nNonetheless, I admit that the paper shows a pretty limited scope and demonstration of its potential, hence I would not strongly object to rejection either.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- The major assumption of having differentiable dynamics, policy, and objectives may limit the applicability of the method to a narrow range of modern autonomous vehicle (AV) stacks. \n\n- Using deviation from the desired trajectory as the sole objective may be misleading. For instance, if a human driver encounters a car flashing extreme colors, deviating from the initial desired trajectory to create a safe distance from the object might be the appropriate response. Under such a loss function, good behavior could become indistinguishable from poor one. This situation is complicated by the differentiability of the objective in my previous point, i.e. expressing more complex objectives could be a challenge.\n\n- The scope of the work is somewhat limited, as acknowledged by the authors. It only focuses on manipulating object textures and poses. It would be beneficial to see the capability to manipulate the scene in a more sophisticated manner. \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses an important problem in autonomous driving, specifically adversarial scenario generation. The application of NeRF for adversarial rendering is an interesting concept that will be widely applied in the future. It is interesting to see that authors even tried to reproduce (to the best of their abilities) generated adversarial examples on the real scene. Overall, the presence of robotics experiments is very appreciated.\nNonetheless, the paper has its limitations. I will mention them in the \"Questions for Rebuttal\" section. ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "-ggM_zaW07R", "reviewer": "Reviewer_8f1g", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "End2end driving policies, like black-box, lack interpretability, and thus methods for discovering the failure cases are important for improving their safety. This paper proposes a promising method to attack a well-trained policy and efficiently discover object textures that confuse the policy. Concretely, the gradient can be found to maximize the error between the generated trajectory and the reference trajectory. After that, this gradient is fed into a differentiable renderer to modify the scene representation to the extent that policy won't work when taking this new adversarial representation as input. The experiments on both the simulator and the real robot show the effectiveness of the method. ", "summary_of_recommendation": "A potentially impactful work. Recommend for acceptance.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "N/A", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I enjoy reading this pioneering work. It is well-motivated and easy to understand from the teaser to the experiments. In addition, the method is well-explained in Fig.3, which clearly describes how the method works. The experiments provide strong evidence that it can work on real robots. \n\nThe only suggestion I have is to add some details and move some content in the appendix into the main paper. For example:\n1. Please mention that the system dynamic is differentiable by using Dubin's vehicle in the caption of Figure.3.\n2. It would be better to include Appendix C3.1 in the main paper to show an important downstream application.\n\n\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel and promising approach to generating adversarial examples for end-to-end self-driving policy. The method first builds a neural radiance field (NeRF) of the streets as a surrogate, differentiable, photorealistic simulator. Then, it solves an optimal control problem to find the perturbation that can lead the policy to deviate from its original route. The method is evaluated on both simulated and real-world environments, the results demonstrate that the proposed attack can be transferred to the real-world environment. 

Most reviewers agree that the paper is well-written and presents a novel approach to generating adversarial examples for self-driving policies. Specifically the strengths pointed out include
- The use of NeRF as a differentiable renderer for improving the robustness of self-driving policies.
- The figures and equations are clearly laid out and allow the reader to understand quickly their pipeline.

The reviewers also brought some weakness including
- The assumption of having differentiable dynamics, policy, and objectives may limit the applicability of the method to a narrow range of modern autonomous vehicle stacks.
- Using deviation from the desired trajectory as the sole objective may be misleading.
- The scope of the work is somewhat limited, as acknowledged by the authors. It only focuses on manipulating object textures and poses.

During the rebuttal phase, the reviewers  acknowledged the authors did a thorough job with clarifying and improving the paper in line with reviewer concerns.

For the above reasons, we recommend accepting this paper as poster. The authors should address the limitations of their method, such as the assumption of differentiable dynamics, policy, and objectives in the camera-ready version.
Decision: Accept (Poster)