Adv3D: Generating Safety-Critical 3D Objects through Closed-Loop Simulation
Authors: Jay Sarva
Jingkang Wang
James Tu
Yuwen Xiong
Sivabalan Manivasagam
Raquel Urtasun
Venue: CoRL 2023 Poster (2023)
Keywords: closed-loop simulation
adversarial robustness
self-driving

Abstract: Self-driving vehicles (SDVs) must be rigorously tested on a wide range of scenarios to ensure safe deployment. The industry typically relies on closed-loop simulation to evaluate how the SDV interacts on a corpus of synthetic and real scenarios and to verify good performance. However, they primarily only test the motion planning module of the system, and only consider behavior variations. It is key to evaluate the full autonomy system in closed-loop, and to understand how variations in sensor data based on scene appearance, such as the shape of actors, affect system performance. In this paper, we propose a framework, Adv3D, that takes real world scenarios and performs closed-loop sensor simulation to evaluate autonomy performance, and finds vehicle shapes that make the scenario more challenging, resulting in autonomy failures and uncomfortable SDV maneuvers. Unlike prior work that add contrived adversarial shapes to vehicle roof-tops or roadside to harm perception performance, we optimize a low-dimensional shape representation to modify the vehicle shape itself in a realistic manner to degrade full autonomy performance (e.g., perception, prediction, motion planning). Moreover, we find that the shape variations found with Adv3D optimized in closed-loop are much more effective than open-loop, demonstrating the importance of finding and testing scene appearance variations that affect full autonomy performance.

ADV3D: Generating Safety-Critical 3D Objects through Closed-Loop Simulation
Jay Sarval.3t
Jingkang Wang! James Tu Yuwen Xiong!
Sivabalan Manivasagam 1,2 Raquel Urtasun
Waabi
University of Toronto² Brown University
jay.sarva@browm.cor edu [jwang,jtu, ftu. smanivasagam, aurtasun)wwaabll ai
Abstract: Self-driving vehicles (SDVs) must be rigorously tested wide range of scenarios to ensure safe deployment. The industry typically relies on closed-loop simulation to evaluate how the SDV interacts on a corpus of synthetic and real scenarios and verify it performs properly. However, they primarily only test the system's 's motion planning module, and only consider behavior variations. It is key to evaluate the full autonomy system in closed-loop, and to understand how variations in sensor data based on scene appearance, such as the shape of actors, affect system performance. In this paper, we propose framework, ADV3D, that takes real world scenarios and performs closed-loop sensor simulation to evaluate autonomy performance, and finds vehicle shapes that make the scenario more challenging, resulting autonomy failures and uncomfortable SDV maneuvers. Unlike prior works that add contrived adversarial shapes to vehicle roof-tops roadside to harm perception only, we optimize a low-dimensional shape representation to modify the vehicle shape itself in realistic manner to degrade autonomy performance (c.g., perception, prediction, and motion planning). Moreover, we find that the shape variations found with ADV3D optimized in closed-loop than those in open-loop. demonstrating the importance of finding scene appearance variations that affect autonomy in the interactive setting. Please refer to our project page ht attps: /tt////wabii.. ai/adv3d. for more results.
much more effective
Keywords: Closed-loop simulation, Adversarial robustness, Self-driving
Digital
Modified Digita/ Twins
Original
Adv3D
Oripinal
1.0s
5.0g
Modifying Actor Shapes in Digital Twins
Closed-Loop Simulation with Full Autonomy
Figure I: ADV3D is framework that evaluates autonomy systems in closed-loop complex real- world scenarios, and alters the scene appearance via actor shape modification (left) to create more challenging scenarios that cause autonomy failures such inaccurate detections, incorrect predictions and strong decelerations or swerving, resulting in uncomfortable and dangerous maneuvers (right).
Introduction
Most modern autonomy systems in self-driving vehicles (SDVs) perceive the agents in the scene, forecast their future motion, and then plan a safe maneuver [1, 2, 3, 4]. These tasks are typically
1 Work done while a research intern at Waabi.
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

referred to as perception, prediction and motion planning respectively. To deploy SDVs safely, we must rigorously test the autonomy system on a wide range of scenarios that cover the space of situations we might see on the real world, and ensure the system can respond appropriately. Two strategies are employed to increase scenario coverage; either synthetic scenarios are created or situations are retrieved from a large collection of logs captured during real-world driving.
The self-driving industry then relies on closed-loop simulation of these scenarios test the SDV in a reactive manner, so that the effects of its decisions are evaluated in a longer horizon. This is important, as small errors in planning can cause the SDV to brake hard or swerve. However, coverage testing is typically restricted to the behavioral aspects of the system and only motion planning is evaluated. This falls short, as does not consider scene appearance and ignores how perception and prediction mistakes might result in safety critical errors with catastrophic consequences, e.g., false positive detections or predictions might result in a hard break, false negatives could cause collision.
In this paper, we are interested in building a framework for testing the full autonomy system. We focus on a LiDAR-based stack as it is the most commonly employed sensor in self-driving. Naively sampling all the possible variations is computationally intractable, as we need to not only cover all the possible behavioral scenarios but also scene appearance variations, such as actor shape, the primary factor for LiDAR point clouds. Towards this goal, we propose novel adversarial attack framework, ADV3D, that searches over the worst possible actor shapes for each scenario, and attacks the full autonomy system, including perception, prediction and motion planning (Fig. 1). This contrasts with existing approaches that focus on building universal perturbation to a template mesh that is added into all scenes as roof-top or road-side object, and only attacks the perception system [5, 6, 7, 8).
We leverage a high-fidelity LiDAR simulation system that builds modifiable digital twins of real- world driving snippets, enabling closed-loop sensor simulation of complex and realistic traffic scenarios at scale. Thus, when modifying the scene with a new actor shape, we can see how the autonomy system would respond to the new sensor data as if it were in the real world (e.g.. braking hard, steering abruptly) as the scenario evolves. To ensure the generated actor shapes are realistic, during optimization we constrain the shape to lie within a www-dimensional latent space learned over a set of actual object shapes. This contrasts with existing approaches that search over the vertices of the mesh directly [5, 6, 7, 81, resulting in unrealistic shapes that need to be 3D-printed to exist.
In our experiments, find ADV3I can generate challenging actor shapes on over 100 real-world highway driving scenarios and on multiple modern autonomy systems and attack configurations, resulting in perception and prediction failures and uncomfortable driving maneuvers. Importantly, we show that by attacking in closed-loop we can discover worse situations (i.e, more powerful attacks) than attacking in open loop. We believe this finding is very significant and will spark a change in the community to find scenario variations that are challenging to every aspect of autonomy.
2 Related Work
Self-Driving Systems: One of the earliest approaches to self-driving autonomy was an end-to-end policy learning network [9] that directly outputs control actuations from sensor data. This autonomy approach has significantly evolved due to advancements in network architectures, sensor inputs, and learning methods [9, 10, 1, 11, 12], but lacks interpretability. Most modern self-driving autonomy systems in industry typically break down the problem into three sequential tasks: instance-based object detection [13, 14, 15, 16), motion forecasting [17, 18, 19, 20]. and planning [21, 22, 23, 24).
Some works conduct joint perception and prediction (P&P) first [25, 26]. Other works leverage shared feature representations for simultancous perception, prediction and planning learning [27, 28]. Lately, interpretable neural motion planners have emerged, enabling end-to-end learning while ensuring modularity and interpretability [27, 2, 4]. Recently, another autonomy paradigm is instance- autonomy that estimate spatial-temporal occupancies [2, 29, 30, 4]. To demonstrate generalizability, this work evaluates an instance- based [26] and instance- [31] autonomy system.
Ad Adversarial Robustness for Self-Driving: Research on adversarial robustness has obtained sub- stantial attention [32, 33, 34, 35, 36, 38, 39, 40, 41, 42, 43], particularly in safety-critical domains such as self-driving. Recent works primarily consider individual subsystems in self-driving. Specif- ically, researchers have focused on generating undetectable or adversarial objects [5. 7, 8, 44], spoofing LiDAR points [45, 46) and creating adversarial vehicle textures within simulation en- vironments [47, 48]. Most recently, some works focus prediction robustness, exploring data-
2

---

Latent rep.
Generated Object
UIAA simulation
Closed-Loop Simulation
Adv. Objactive 6u Serffue
LOVK
En Cr
Z
paint clouds
Autonomy system
$ DP-LCS
PCA
AcOM lorory
location Eyp1
Black box optimization
Figure 2: Overview of ADV3D for safeta-critical 3D object generation in closed-loop. Given a real-world scenario, Adv3D modifies the shapes of selected actors, and runs LiDAR simulation and full autonomy stack in closed-loop to optimize the shape representation with black-box optimization.
driven trajectory prediction models and proposing a series of adversarial attack and defense mech- anisms [49, 50, 51]. Other works evaluate planners given the ground-truth perception informa- tion [52, 53, 54, 55] or consider simplified image-based imitation learning systems [56, 57J. Finally, some works consider the entire end-to-end autonomy stack [58, 59] by modifying the actor trajectories to induce poor downstream planning performance. However, all these attacks are performed in the open-looy setring that are not suitable for real-world autonomy testing.
The gap between open-loop and closed-loop testing for deep neural networks has been studied in [60], showing the results of open-loop testing does not necessarily transfer to closed-loop environments. Existing closed-loop adversarial attacks [61, 56, 62, 63] are performed in game-engine environments like CARLA [64]. Specifically, [61] perform adversarial attacks on an image-based autonomy by painting black lines on the road. 156, 62) attack the entire autonomy stack by modifying the trajectories of traffic participants. [63] take simple parameterized scenarios and create ad adversarial scenarios by optimizing the parameters. However, these works only consider few hand-crafted synthetic scenarios with limited realism and diversity, which generalizes poorly to the real world [65]. In contrast, ADV3D generates adversarial actor shapes with a realistic closed-looy loop sensor simulator on over 100 real-world scenarios for HiDAR-based autonomy.
Closed-Loop Adversarial Attacks:
3 Generating Safety-Critical 3D Objects via Closed-Loop Simulation
We aim to extend closed-loop autonomy system testing to not only include behaviors, but also scene appearance, such as actor shape variations. Towards this goal, we conduct black-box adversarial attacks against the full autonomy stack to modify actor shapes in real-world scenarios to cause system failures. To efficiently find realistic actor shapes that harm performance, we parameterize our object shape as a low-dimensional latent code and constrain the search space to lie within the bounds of actual vehicle shapes. Given a real-world driving scenario, we select nearby actors and modify their actor shapes with generated shapes. We then perform closed-loop sensor simulation and see how the SDV interacts with the modified scenario over time and measure performance. Our adversarial objective consists of perception, prediction, and planning losses to find errors in every part of the autonomy. We then conduct black-box box optimization per scenario to enable testing of any autonomy system scale. Fig. 2 shows an overview of our approach.
In what follows, we define the closed-loop simulation setting and our attack formulation (Sec 3. We then describe how we build realistic scenes from real world data, parameterize the adversarial actor geometries, and carry out realistic LiDAR simulation for autonomy testing. (Sec 3.2). Finally, present the adversarial objective and the black-box optimization approach for ADV3D (Sec 3.3).
3.1 Problem Formulation
Closed-loop Autonomy Evaluation: We now review closed-loop evaluation of an autonomy sys- tem. A traffic scenario S, at snapshot time is composed of static background B and N actors: S. = ([A) A?, AA ), B). Each actor A, consists of {G, where G and & represent the actor's geometry and its pose at time i. Given S, and the SDV location &,, the simulator vé gen- erates sensor data according to the SDV's sensor configuration. The autonomy system F then consumes the sensor data, generates intermediate outputs O such detections and the planned
3

---

Digital Twins
Simulated LIDAR
Digital Twins
Simulated LIDAR
Digital Twins
Simulated LIDAR
Figure 3: Reconstructed digital twins and simulated LiDAR for real-world highway scenarios.
trajectory, and executes a driver command (e.g. Steering, acceleration) via the following mapping F: (., + D x O, The simulator then will update the SDV location given the driver command E,, D + E as well as update the actor locations to generate the scenario snapshot S2+1- This loop continues and we can observe the autonomy interacting with the scenario over time.
now select actors in the scene to modify their shapes and optimize to find autonomy failures. For simplicity, describe our formulation with single modified actor, but our approach is general and also demonstrate modifying multiple actors in Sec 4. Let Gldv denote the safety-critical 3D object geometry for the interactive actor AA'. In this paper, our goal is to generate gudv that is challenging to the autonomy system F. We define cost function C2 that takes the current scene and autonomy outputs to determine the autonomy performance at time t. We accumulate the cost over time to measure the overall closed-loop performance, resulting in the following objective:
Ad Adversarial Shape Attacks in Closed Loop: Given closed-loop formulation,
T
Gadv = arg max
Cee(s, (S., (v(S..9,E)) G
where the sensor simulator / takes the original scenario S, but replaces the geometry of actor A with the optimized shape G and simulates new sensor data.
3.2 Realistic Sensor Simulation and Adversarial Shapes
We now describe how we build digital twins from real world scenarios to perform realistic closed-loop sensor simulation (via 2) for autonomy testing. We then explain how we parameterize our actor shape G" to enable realistic and efficient optimization.
Building Digital Twins for LiDAR Simulation: Data-driven sensor simulation [66, 67, 68, 69, 70, 71] has witnessed great success in recent years. Following [72, 73], we leverage real-world LiDAR data and object annotations to build aggregated surfel meshes (textured by per-point intensity value) for the virtual world. We manually curated a set of actor meshes that have complete and clean geometry, together with CAD assets purchased from TurboSquid [74] to create rich asset library to model actor shapes for benign actors in the scene. During simulation, we query the actor geometries from the curated set based the actor class and bounding box. We can then place the actor geometries according to their pose & in the scenario snapshot at time f based on S,, and then perform primary raycasting [72, 73, 75, 76] based the sensor configuration and SDV location to generate simulated LiDAR point clouds for the autonomy system via tb. Fig. 3 shows examples of reconstructed digital twins and simulated LiDAR point clouds.
Adversarial Shape Representation: With our digital twin representation that reconstructs the 3D world, we can now model the scenario snapshot S,. We now select an actor to modify its shape. To ensure the actor shape affects autonomy performance, SDV as ' and modify its shape G to be adversarial. To ensure the actor shape is realistic and is not contrived and non-smooth mesh, representation z with realistic constraints. Specifically, for vehicle actors, the primary actor in driving scenes, we take inspiration from [77] and learn shape representation over large set of CAD vehicle models that represent a wide range of vehicles (e.g.. city cars, sedans, vans, pick-up trucks). For each CAD vehicle shape, first compute a volumetric truncated signed distance field (SDF) to obtain a dense representation. We then apply principal component analysis [78] (PCA) over the flattened volumetric SDF &b E TR. L|x1 of the whole CAD dataset to obtain the latent representation:
select the closest actor in front of the
parameterize the geometry G using low-dimensional
W (Φ- 14),
(z)) = MarchingCubes (W... z + 1))

---

where E R|| is the mean volumetric SDF for the meshes and W is the top K principle components. We then extract the explicit mesh using marching cubes [79]. Note that each dimension in controls different properties of the actors (e.g., scale, width, height) in controllable and realistic manner [77, 80). To ensure during optimization the latent code lies within the set of learned actor shapes and is realistic, we normalize the latent code to lie within the bounds of the minimum and maximum values of each latent dimension over the set of CAD models.
3.3 Adversarial 3D Shape Optimization
With the simulation system vp and a black-box autonomy system F under test, we can perform closed-looy testing (Sec.3.1). Given the selected actor A'' and oiwddimesioonll shape representation the next step is to optimize z such that the overall autonomy performance drops significantly. We now introduce the adversarial objective and the search algorithms to produce safety-critical shapes.
Adversarial Objective: We consider an adversarial objective that accounts for the entire autonomy stack, aiming to concurrently reduce the performance of each module, Taking instance-based modular autonomy example, the adversarial objective C₁ combines the perception loss Eett. prediction loss fpred and planning comfort cost Cplan to measure the overall autonomy performance. We increase perception errors by decreasing the confidence / Intersection of Union (IoU) for the true positive (TP) detections and increasing the confidence/ / loU for false positive (FP) proposals. For motion forecasting, compute the averaged displacement error (ADE) for multi-modk trajectory predictions. As the modified actor shape can also affect the perception and motion forecasting of other actors, consider the average objective for all actors within the region of interest across all the frames. In terms of planning, we calculate two costs that measures the comfort (jerk and lateral acceleration) of the driving plan each snapshot time l. Formally, we have
Ce =
=-:ies ldeet + Apread pred + AplanC
(3)
Cdct Σ
I Co) + Σ IoU((", Conf(BB),
(4)
TP
FP
K
@t pred
9
95 c
1 Pj
plan = Geek + ciac>
(5)
where B², B² are the ground truth and detected bounding boxes at time & and a, 3 are the coefficients to balance TP and FP objectives. 3j 1.h t.h .Pj" are the hith ground truth and predicted waypoints for actor j at time i and H is the prediction horizon. Lastly, Gjeck and clat represent the jerk (m/s²) and lateral acceleration (m/s²) cost at time t. We aggregate these costs over time Σ C. to get the final closed-loop evaluation cost. Note that our approach is general and can find challenging actor shapes for any autonomy system. We detail in supp. the objective function for instance-free autonomy.
Black-box Search Algorithm: We apply black-bos box optimization since we aim to keep ADV3D generic to different modern autonomy systems (including non- non-differentiable modular autonomy systems). Inspired by existing works [56, 58), we adopt Bayesian Optimization [81, 82) (BO) as the search algorithm with Upper Confidence Bound [83] (UCB) as the acquísition function. Since the adversarial landscape is not locally smooth, we use standard Gaussian process with a Matérn kernel. We also compare BO with the other popular search algorithms including grid search [84] (GS), random search [85, 86, 87] (RS) and blend search (BS) [88]. We also compare to a baseline that conducts brute-force search (BF) over the curated asset library.
Experiments
We showcase applying ADV3D to generate safety-critical 3D actor shapes for autonomy system testing on real-world scenarios. We first introduce the experimental setting in Sec 4.1. Then in Sec 4.2, we demonstrate the importance of adversarial optimization through closed-loop simulation for the whole autonomy stack. We further show the realism of our adversarial shape representation. Finally, we investigate different attack configurations in closed-loop including attacks with multiple actors reactive actors, and benchmark various black-box optimization algorithms.
5

---

Perception and Prediction
Plancing
Execuibea Driving Comfort J
Closed-Loop Test
Recall (%)
ADE
Planning Comfort
AP
Recall
minADE:
mEInADE
Tat. (mx/5*)
Jark (mn/s")
at. (m//*))
Jenk (m//""
Autonomy-A: Instance-based [26] [24]
Original
88.2
89.4
2.14
4,90
0.203
0.336
0. 1.194
0.331
open-loop
88.3
2.08
4.87
0.214
0.207
0.337
Adv. closed-loop
80.1
84.8
2,40
5.09
0.263
0.427
0.265
0.401
Looe Test
mAP
Occupancy (%) +
Soft-ToU
mAP
Flow Grounded f
Soft-ToU
Lat. (m/s*)
Planning Comfort 1
Jerk (m//s"
Lar. (m/s")
Driving Comfort 1
Jerk (m/s*)
Autonomy-B: Instance-free [31] [24]
Original
83.1
50.4
94.6
61.2
0.256
0.319
0.263
0.315
Adv. open koop
85.7
53.2
96.3
65.5
0.260
0.451
0.279
0.424
Adv. closed-loop
78.8
45.9
90.1
55.7
0.302
0.456
0.308
0.431
Table I: Evaluation of adversarial objects in the closed-loop setting.
Original
Adv. Closed-Loop
Adv. Open-Loop
Original
Adv. Closed-Loop
Adv. Open-Loop
11-2.
37.5
0.0.1
r-03
Uncommon shape (short pickup truck) degraded detection
Problematic perception prediction to unconfortable driving
Figure 4: Qualitative examples of adversarial shape generation closed loop vs open loop. We highlight the modified actors using orange bboxes and show generated adversarial shapes aside. The perception and prediction failures are pointed out by green and blue arrows. Specifically, the green arrows point the green detected bounding boxes, and blue arrows point to the predicted light blue trajectories. The numbers on top of green predicted bounding boxes are the detection confidence scores. The numbers beside the pink ego-truck denote the current velocity and acceleration.
4.1 Experimental Setup
Dataset: We evaluate our method on a real-world self self-driving dataset HighwaySSenarios, which contains 100 curated driving snippets captured US highway each with a duration of 20 seconds (200 frames, sampled at 10hz). The dataset is collected with multiple LiDARs that we then annotate spatial-temporal actor tracks and leverage build the digital twin backgrounds and reconstructed assets. It covers wide variety of map layouts, traffic densities, and behaviors.
Autonomy Systems: We evaluate state-of-the-ard interpretable multi-LiDAR based autonomy systems. The first (Autonomy-A) uses an instance-based joint perception and prediction (P&P) model [26] which outputs actor bounding boxes and trajectories in birds-eye-view (BEV). The second (Autonomy-B) uses an instance-free foccupancy-based) P&P model [31] that predicts the implicit BEV occupancy and flow for queried spatial points at current (detection) and future (prediction) timestamps. Both systems use sampling-based motion planner [24] which samples lane-relative trajectories and chooses the best trajectory with the lowest combined safety and comfort costs. Specifically, We report evaluation results for two autonomy systems in Table 1. For the rest experiments, we only study the instance-base based Autonomy-A.
Metrics: To investigate the effectiveness of ADV3D on downstream tasks, we measure various metrics to evaluate the module-Ievel performance, including detection, prediction and planning, For Autonomy-A, we use Average Precision (AP) and Recall to measure the detection performance. For prediction, we measure Average Displacement Error (ADE) quantify the performance of trajectory forecasting. For Autonomy-B -B, we follow [31] to report mAP and Ioft- to measure the performance of occupancy and flow prediction. All metrics are averaged across the simulation horizon (T = 5s) within ROI. We report the planning comfort metrics including lateral acceleration (Lat.) and Jerk, which assess the smoothness of the proposed ego plan. To evaluate how the SDV executes in the closed loop, we evaluate leste metrics driving comfort (i.e., lateral acceleration and jerk on the executed 5s SDV trajectory). Finally, we report SeonennShhanme divergence [89] (JSD) to measure the realism of the generated actor shapes.

---

Perception Σ fere
Prediction Σ, Speed
Planning Σ plan
AP (%, @0.5)
Recall f (%, 80.5)
minADE
meanADE 4 L2 error
Lat. 4 (m/s²)
Jerk
#ID
L/2 error
(m//²)
Original
88.7
89.4
2.51
4.99
0.261
0.294
M1
69.6
71.4
5.02
0.239
0.310
M2
83.1
89.1
2.92
6.34
0.254
0.412
Ma
86.7
88.3
2.94
6.03
0.324
0.434
Ours
75.4
76.4
2.82
6.21
0.411
0,410
Table 2: Adversarial optimization for the full autonomy system. We compare ADV3D with three baselines that each only attack one module: detection (M1), prediction (M2) and planning (Mg). Each baseline adopts the same pipeline as ADV3D except the adversarial objective is changed. ADV3D generates actor shapes that are challenging to all downstream modules. Interestingly, results in more uncomfortable driving maneuvers (i.e. worst lateral acceleration). We mark the methods with best performances using gold silver and bronze medals.
Algorithms
AP (% 110.5)
Rocall (K, (50.5)
minADE: Lg erroe
(an/a")
Adv3D
VD (5 cm)
VD (10 cm)
VD (20 om) VD (50
VD (100
Original
98.7
99.6
4,70
0.050
VD: 0.05m
98.7
99.6
5.07
0.050
0.061
VD:
99.6
VD: 0.1m
98.7
99.6
5.73
0.090
0.125
VD: 0.2m
98.7
99.6
5.78
0.050
0.285
VD: 01.mm
80.2
83.0
0.090
0.688
(ouns)
0:175
Figure 5: Qualitative comparisons with VD,
Table 3: Compare with vertex deformation (VD).
4.2
Experimental Results
ADV3D finds challenging actor shapes: We report autonomy performance metrics on the 100 traffic scenarios with and without our adversarial shape attack Table 1. For each scenario, we set the attack search query budget to 100 queries. Our approach can degrade the subsystem performance and execution performance significantly. We further provide qualitative examples of optimized adversarial shapes together with autonomy outputs in Fig. 4. In Fig. 4 left, ADV3D successfully creates a uncommon actor shape (short and wide truck) that degrades the detection performance (low confidence mis- mis-detection). Fig. 4 right shows another example where the closed-loop attack finds a tiny city-car causes inaccurate detection and prediction for an actor behind the SDV and results in the SDV applying a strong deceleration. Note that the modified actor shape alters the simulated LiDAR such that perception and prediction outputs are harmed even for other actors in the scene.
Importance of Closed-Loop Simulation: We also compare approach, where find chal- lenging actor shapes during closed-loop autonomy evaluation, against optimizing actor shapes in the open-loop setting. In the open-loop shape attack, the ego vehicle follows the original trajectory in the recorded log, and we optimize the actor shape with the same objective in Eq. 3. We then test the optimized actor shapes in closed-loop simulation where the ego vehicle is controlled by the autonomy model. Generating adversarial objects in closed-loop simulation yields substantially worse performance compared to open-loop. This indicates that it is insufficient study adversarial robustness in open-loop as the attacks do not generalize well when the SDV is reactive. In Fig. 4, the optimized shapes in the open-loop attack do not harm autonomy performance significantly.
Attacking Full Autonomy Stack: Our adversarial objective takes the full autonomy stack into account. To demonstrate the importance of attacking the full system, we choose 10 logs from HighwayScenario and compare with three baselines inspired by existing works that each only attack one module (detection: [5, 7], prediction: [49, 50], and planning: [59, 90]) for Autonomy-A. To reduce performance variability, for each scenario, we perform 3 attacks, each with budget of 100 queries, and modify the closest 3 actors in front of the SDV individually and report the worst-case performance. As shown in Table 2, attacking each downstream module produces challenging objects that are only contrast, our model effectively balances all tasks to generate worst-case 3D
risky to that module. objects that challenge the entire autonomy stack, serving holistic tool for identifying potential system failures. We report additional objective combinations in supp.
Latent Asset Representation: We also demonstrate that optimizing with our latent asset represen- tation is more realistic than prior-works that perform per-vertex deformation. We select a scenario where the modified actor mesh from ADV3D has strong attack against the autonomy. We initialize
7

---

Perception t (%)
Prediction
Planning 4 Planning Comfon
IOU/ / Confidence
AP Recall (80.5)
ADE
ToU
Conl.
AP
Recall
minADE
meanADE
Lat. (m/s*)
Jerk (m/s")
Original
69.2
88.8
92.7
88.7
2.51
4.99
0.261
0.294
m =
62.9
71.1
76.5
79.3
2.83
6.33
0.312
0.405
m 2
61.7
70.1
71.8
75.1
2,96
6.30
0,352
0.460
m 3
55.7
77.0
70.2
73.7
3.37
6.27
0.360
0.475
5
50.1
70.9
64.5
67.7
3.39
6.34
0.377
0.535
Table 4: Adversarial optimization with multiple actors. denotes the number of modified actors.
AP (%/ (20.5)
minADE. Zg ervor
Jerk (m/s")
GPU
Algorithms
#Query
Hour
AP (%,, (50.5)
minADE 1 Ty error
Jerk (m/s*)
Settings
Original
98.7
4.70
0.090
0.2
GS
RS [85, 86, 87]
52.4
6.10
0.090
500
98.3
Original
98.7
5.91
0.168
BS [88]
52.4
6,09
0.080
100
19.6
Adv. open-loop
59.3
5.23
0.140
BO (82)
19.6
Adv. closed-loop
52.1
6.86
0.335
Brute-Foree
52.7
7.91
0.090
746
146.6
Table 6: Attacks with reactive actors.
Table 5: Different black-box search algorithms.
the mesh by decoding the latent z to generate a mean shape with 500 faces and 252 vertices, and add perturbations vertices. The perturbations the vertex coordinates are constrained by an &oo norm, and we experiment with 4 variations [0.1m, 0. 0.2m, 0.5m, 1.0m]. Results in Tab. 3 show that very loose constraints of Imn are needed achieve the strong attack performance. However, the generated assets with loose constraints are unrealistic compared to ADV3D 3DV meshes. We hypothesize this is because is difficult to optimize such nigh-dimensional vertex-based shape representations, as black- box optimization methods suffer from curse of dimensionality.
Multiple Adversarial Actors: In Table
study how ADV3D can easily scale to modify the shapes of multiple actors in the scenario. We use the same 10 logs as Table 2. We set m = [1, 2, and sample the closest actors in front of the ego vehicle. Modifying multiple actors' shapes simultaneously yields stronger adversarial attacks.
Attack Configurations: We benchmark other black-box search algorithms and brute-force search in Table For brute-force baseline, we iterate over all shapes in the asset library and select the one that results in the strongest attack. Table 5 shows that Bayesian Optimization (BO) leads to the strongest adversarial attack while also using the least compute. We further investigate different selected log in Table 6. To increase the realism of our setting, reactive actors by using a traffic model [91] to control their behaviors. As shown in Table 6, ADV3D) also generates challenging actor shapes in this setting and the results demonstrate the importance of closed-loop simulation.
attack configurations
adopt
Limitations and Conclusion
ADV3D's main limitation is that we do not optimize the actor behaviors like prior work [58, 59, 56] to allow for more diverse adversarial scenario generation. Moreover, how to incorporate ADV generated safety-critical objects to create new scenarios for robust training remains future study. While our shapes are more realistic than prior work, we also observe occassionally convergence to shapes that have artifacts oblong wheels. Better shape representations (including for non- vehicle classes) and optimization approaches (e.g., multi-objective optimization), can help create higher-fidelity and more diverse adversarial objects more efficiently.
In this paper, we present closed-loop adversarial framework to generate challenging 3D shapes for the full autonomy stack. Given real-world traffic scenario, our approach modifies the geometries of nearby interactive actors, then run realistic LiDAR simulation and modern autonomy models in closed loop. Extensive experiments on two modern autonomy systems highlight the importance of performing adversarial attacks through closed- loop simulation. We hope this work can provide useful insights for future adversarial robustness study in the closed-loop setting.
8

---

Acknowledgement
We sincerely thank the anonymous reviewers for their insightful suggestions. We would like thank Yun Chen, Andrei Bårsan and Sergio Casas for the feedback on the early results and proofreading. We also thank the Waabi team for their valuable assistance and support.
References
[1] A. Kendall, Hawke, D. Janz, Mazur, D. Reda, J.-M. Allen, V.-D. Lam, A. Bewley, and
A. Shah. Learning to drive in day. arXiv preprini arXiv: 00088807000100 2018.
[2] A. Sadat, S. Casas, M. Ren, X. Wu, Dhawan, and R. Urtasun. Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. CoRR, abs/2008.05930,, 2020.
[3] M. H. Danesh, P. Cai, and D. Hsu. Leader: Learning attention over driving behaviors for planning under uncertainty. In Conference on Robot Learning, pages 199-211. PMLR, 2023.
[4] Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, Chai, S. Du, T. Lin, W. Wang. et al. Planning- oriented autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17853-17862, 2023.
[5] Y. Cao, C. Xiao, D. Yang, J. Fang, R. Yang, M. Liu, and B. Li. Adversarial objects against lidar-based autonomous driving systems. arXiv preprint arXiv: 1907.0541 2019.
[6] Y. Cao, C. Xiao, B. Cyr, Y. Zhou, W. Park, S. Rampazzi, Q A. Chen, K. Fu, and Z. M. Mao. Adversarial sensor attack on lidar-based perception in autonomous driving. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pages 2267-2281, 2019.
[7] J. Tu, M. Ren, S. Manivasagam, M. Liang, B. Yang, R. Du, F. Cheng, and R. Urtasun. Physically realizable adversarial examples for lidar object detection. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pages 137255,, 2020.
[8] J. Tu, H. Li, X. Yan, M. Ren, Y. Chen, M. Liang, E. Bitar, E. Yumer, and R. Urtasun. Exploring adversarial robustness of multi-sensor perception systems in self driving. arXiv preprint arXiv:2101.06784, 2021.
[9] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In NIPS, 1989.
[10] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp. P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, al. End to end learning for self-driving cars. arXiv preprint 164.:7160..07316, 2016.
[11] F. Codevilla, M. Miller, A. López, V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In ICRA, 2018
[12] A. Hu, G. Corrado, N. Griffiths, Z. Murez, C. Gurau, H. Yeo, A. Kendall, R. Cipolla, and Shotton. Model-based imitation learning for urban driving. Advances Neural Information Processing Systems, 35:20703-20711 2022.
[13] B. Yang, W. Luo, and R. Urtasun. Pixor: Real-time 3d object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7652-7660, 2018
[14] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12697-12705, 2019.
[15] T. Yin, X. Zhou, and P. Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 11784-11795 2021.

---

[16] S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li. Pv-renn: Poxnt feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10529-10538, 2020.
[17] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11525-11533 2020.
[18] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++:: Dynamically-fessbble trajectory forecasting with heterogeneous data. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 683-700. Springer, 2020.
[19] W. Zeng, M. Liang, R. Liao, and R. Urtasun. Lanerenn: Distributed representations for graph- centrie motion forecasting. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 532-539. IEEE, 2021.
[20] A. Cui, S. Casas, K. Wong, S. Suo, and R. Urtasun. Gorela: Go relative for viewpoint-invarian motion forecasting. arXiv preprint arXiv:2211.02544 2022.
[21] M. McNaughton, C. Urmson, J. M. Dolan, and J.-W. Lee. Motion planning for autonomous driving with conformal spatiotemporal lattice. In 2011 IEEE International Conference on Robotics and Automation, pages 4889-4895, IEEE, 2011.
[22] H. Fan, F. Zhu, C. Liu, L. Zhang, L. Zhuang, D. Li, W. Zhu, J. Hu, H. Li, and Q. Kong, Baidu apollo em motion planner. arXiv preprint arXiv:1807.0800 2018.
[23] N. Rhinehart, K. M. Kitani, and Vernaza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), pages 772-788, 2018
[24] A. Sadat, M. Ren, A. Pokrovsky, Y.-.. Lin, E. Yumer, and R. Urtasun. Jointly learnable behavior and trajectory planning for self-driving vehicles. IROS, 2019.
[25] W. Luo, B. Yang, and R. Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 3569--3777 2018.
[26] M. Liang, B. Yang, W. Zeng, Y. Chen, R. Hu, S. Casas, and R. Urtasun. Pnpnet: End-to-end perception and prediction with tracking in the loop. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11553-11562, 2020.
[27] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In CVPR, 2019.
[28) W. Zeng, S. Wang, R. Liao, Y. Chen, B. Yang, and R. Urtasun. Dsdnet: Deep structured self-driving network. CoRR, abs/2008.060044 2020.
[29] S. Casas, A. Sadat, and R. Urtasun. Mp3: A unified model to map, perceive, predict and plan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14412, 2021.
[30] R. Mahjourian, J. Kim, Y. Chai, M. Tan, B. Sapp, and D. Anguelov. Occupancy flow fields for motion forecasting in autonomous driving. IEEE Roboties and Automation Letters, 7(2): 2022.
[31] B. Agro, Q. Sykora, S. Casas, and R. Urtasun. Implicit occupancy flow fields for perception and prediction in self-driving. In CVPR, 2023.
[32] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arxiv preprint arXiv: 14l.:1122.222.2. 2014.
[33] A. Madry, A. Makelov, I.. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1701037706000083, 2017.
10

---

[34] T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer. Adversarial patch. abs/1712.09665, 2017.
CoRR,
[35] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. In ICML, 2018.
[36] J. Wang, T. Zhang. S. Liu, P.-Y. Chen, J. Xu, M. Fardad, and B. Li. Adversarial attack generation empowered by min-max optimization. Advances in Neural Information Processing Systems, 34: 16020-16033, 2021.
[37] K. Xu, G. Zhang, S. Liu, Q. Fan, M. Sun, H. Chen, P. Chen, Y. Wang, and X. Lin. Evading real-time person detectors by adversarial t-shirt. CoRR, 0tss1910.11199, 2019.
[38] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song. Robust physical-world attacks on deep learning visual classification. In CVPR, 2018.
[39] J. Wang, Y. Liu, and B. Li. Reinforcement learning with perturbed rewards, In Proceedings of the AAAI conference on artificial intelligence, number 04, pages 6202-6209, 2020.
[40] J. Wang, H. Guo, Z. Zhu, and Y. Liu. Policy learning using weak supervision. Advances in Neural Information Processing Systems, 34: 14:19990-19973, 2021.
[41] C. Xiao, D. Yang. B. Li, Deng, and M. Liu. Meshadv: Adversarial meshes for visual recognition. In CVPR, 2019.
[42] Tu, T. Wang. Wang. S. Manivasagam, M. Ren, and R. Urtasun. Adversarial attacks on multi-agent communication. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7768-7777, 2021.
[43] N. Vadivelu, M. Ren, J. Tu, J. Wang, and R. Urtasun. Learning to communicate and correct pose errors. In Conference on Robot Learning, pages 1195-1210, PMLR, 2021.
[44] H.-T. D. Liu, M. Tao, C.-L. Li, D. Nowrouzezzhraai and A. Jacobson. Beyond pixel norm-ball ball: Parametric adversaries using an analytically differentiable renderer. International Conference on Learning Representations, 2019.
[45] Y. Cao, C. Xiao, B. Cyr, Y. Zhou, W. Park, S. Rampazzi, Q. A. Chen, K. Fu, and Z. M. Mao. Adversarial sensor attack on lidar-based perception in autonomous driving. In CCS, 2019.
[46] J. Sun, Y. Cao, Q. Chen, and Z. M. Mao. Towards robust lidar-based perception in autonomous driving: General black-box adversarial sensor attack and countcrmcasures. In USENIX Security Symposium, 2020.
[47] Y. Zhang, H. Foroosh, P. David, and B. Gong. CAMOU: Learning physical vehicle camouflages to adversarially attack detectors in the wild. In ICLR, 2019.
[48] Wu, X. Ning, W. Li, Huang, H. Yang, and Y. Wang. Physical adversarial attack on vehicle detector in the carla simulator. CoRR, abs/2007.161 18, 2020.
[49] Y. Cao, C. Xiao, A. Anandkumar, D. Xu, and M. Pavone. Advdo: Realistic adversarial attacks for trajectory prediction. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V. pages 36-52. Springer, 2022.
[50] Q Zhang, S. Hu, J. Sun, Q. A. Chen, and 7. M. Mao. On adversarial robustness of trajectory prediction for autonomous vehicles. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 15159-15168, 2022.
[51] Y. Cao, D. Xu, X. Weng, Z. Mao, A. Anandkumar, C. Xiao, and M. Pavone. Robust trajectory prediction against adversarial attacks. In Conference on Robot Learning, pages 128-137. PMLR, 2023.
[52] B. Chen and L. Li. Adversarial evaluation of autonomous vehicles in lane-change scenarios. CoRR, abs/2004.06531, 2020.

---

[53] A. Wachi. Failure-scenaric maker for rule-based agent using multi-agent adversarial reinforce- ment learning and its application autonomous driving. In I/CAI, 2019.
[54] W. Ding, M. Xu, and D. Zhao. Learning to collide: An adaptive safety-critical scenarios generating method. CoRR, abs/2003.01197, 197, 2020.
[55] M. Klischat and M. Althoff. Generating critical test scenarios for automated vehicles with evolutionary algorithms. In /V, 2019.
[56] Y. Abeysirigonnawarden,, Shkurti, and Dudek. Generating adversarial driving scenarios in high-fidelity simulators. In ICRA, 2019.
[57] Norden, M. O' O'Kelly, and Sinha. Efficient black-box assessment of autonomous vehicle safety. arXiv preprint arXiv:9912.06618 2019.
[58] J. Wang, A. Pun, J. Tu, S. Manivasagam, A. Sadat, S. Casas, M. Ren, and R. Urtasun. Advsim: Generating safety-critical scenarios for self-driving vehicles. In CVPR, 2021.
[59] D. Rempe, J. Philion, I. J. Guibas, S. Fidler, and O. Litany. Generating useful accident-prone driving scenarios via learned traffic prior. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17305-17315, 2022.
[60] Haq. D. Shin, S. Nejati, and L. Briand. Can offline testing of deep neural networks replace their online testing? a case study of automated driving systems. Empirical Software Engineering. 26(5):90, 2021.
[61] A. Boloor, K. Garimella, X. He, C. Gill, Y. Voroheychik, and X. Zhang. Attacking vision-based perception in end-to-end autonomous driving models. Journal of Systems Architecture, 110: 101766, 2020.
[62] N. Hanselmann, K. Renz, K. Chitta, A. Bhattacharyya, and A. Geiger. King: Generating safety- critical driving scenarios for robust imitation via kinematics gradients. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVIII, pages 335-352. Springer, 2022.
[63] S. Ramakrishna, B. Luo, C. B. Kuhn, G. Karsai, and A. Dubey. Anti-carla: An adversarial testing framework for autonomous vehicles in carla. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pages 2620-2627. IEEE, 2022.
[64] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. Conference on robot learning, 2017.
[65] S. R. Richter, H. A. AlHaija, and V. Koltun. Enhancing photorealism enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1502):1700 2022.
[66] 7. Yang, Y. Chen, J. Wang, S. Manivasagam, W.-C. Ma, J. Yang, and R. Urtasun. Unisim: A neural closed-loop sensor simulator. In CVPR, 2023.
[67] Y. Chen, F. Rong, S. Duggal, S. Wang, X. Yan, S. Manivasagam, S. Xue, E. Yumer, and R. Urtasun. Geosim: Realistic video simulation via geometry-aware composition for self- driving. CVPR, 2021.
[68] Wang. S. Manivasagam, Y. Chen, Z. Yang, I. A. Bårsan, J. Yang, W.-C Ma, and R. Urtasun. Cadsim: Robust and scalable in-the-wild 3d reconstruction for controllable sensor simulation. In 6th Annual Conference on Robot Learning. 2022. URL attps:///i http:::://////eevi//eeee net/forum? id=Mp3Y5jd7mm.
[69] Z. Yang, S. Manivasagam, Y. Chen, J. Wang. R. Hu, and R. Urtasun. Reconstructing objects tn-the-wild for realistic sensor simulation. In ICRA, 2023.
[70] J. Y. Liu, Y. Chen, Z. Yang, J. Wang, S. Manivasagam, and R. Urtasun. Neural scene rasterization for large scene rendering in real time. The IEFE International Conference on Computer Vision (ICCV), 2023.
12

---

[71] S. Manivasagam, I. A. Bärsan, J. Wang, Z. Yang, and R. Urtasun. Towards zero domain gap: A comprehensive study of realistic lidar simulation for autonomy testing. In ICCV, 2023.
[72] S. Manivasagam, S. Wang, K. Wong, W. Zeng. M. Sazanovich, S. Tan, B. Yang. W.-C. Ma, and R. Urtasun. Lidarsim: Realistic lidar simulation by leveraging the real world. CVPR, 2020.
[73] Z. Yang, Y. Chai, D. Anguelov, Y. Zhou, P. Sun, D. Erhan, S. Rafferty, and H. Kretzschmar. Surfelgan: Synthesizing realistic sensor data for autonomous driving. CVPR, 2020.
[74]
TurboSquid. https///ww..urrbosuui.dcoo Access date: 2023-05-17.
[75] J. Fang, D. Zhou, F. Yan, T. Zhao, F. Zhang, Y. Ma, L. Wang, and R. Yang. Augmented lidar simulator for autonomous driving. IEEE Robotics and Automation Letters, 2020.
[76] J. Fang, X. Zuo, D. Zhou, S. Jin, S. Wang, and L. Zhang. Lidar-aug: A general rendering-basee augmentation framework for 3d object detection. In CVPR, 2021.
[77] F. Engelmann, J. Stückler, and B. Leibe. SAMP: shape and motion priors for 4d vehicle reconstruction. In WACV, 2017.
[78] K. Pearson. Liii on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559-572, 1901.
[79] W. F. Lorensen and H. E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163-169 1987.
[80] F. Lu, Z. Liu, X. Song, D. Zhou, W. Li, H. Miao, M. Liao, I.. Zhang, B. Zhou, R. Yang, et al. Permo: Perceiving more once from single image for autonomous driving. arXiv preprint arXiv:2007.08) 16, 2020.
[81] J. Snock, H. Larochelle, and R. P. Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
[82] B. Ru, A. Cobb, A. Blaas, and Y. Gal. Bayesopt adversarial attack. In International conference on learning representations, 2020.
[83] N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995, 2009.
[84] P. Liashchynskyi and P. Liashchynskyi. Grid search, random search, genetic algorithm: big comparison for nas. arXiv preprint arXiv:1912.060594 2019.
[85] C. Guo, J. Gardner, Y. You, A. G. Wilson, and K. Weinberger. Simple black-box adversarial attacks. In International Conference on Machine Learning, pages 2484-2493, PMLR, 2019.
[86] M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein. Square attack: a query-efficient black-box adversarial attack via random search. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII, pages 484-501. Springer, 2020.
[87] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2). 2012.
[88] C. Wang, Q. Wu, S. Huang, and A. Saied. Economic hyperparameter optimization with blended search strategy. In International Conference on Learning Representations, 2021.
[89] V. Zyrianov, X. Zhu, and Wang. Learning to generate realistic lidar point clouds. In 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIII, pages 17-35. Springer, 2022.
Computer
[90] W. Ding, B. Chen, B. Li, J. Fun, and D. Zhao. Multimodal safety-critica scenarios generation for decision-making algorithms evaluation. IEEE Robotics and Automation Letters, 6(2):1551- 1558, 1.21.
13

---

[91] S. Suo, K. Wong, J. Xu, J. Tu, A. Cui, S. Casas. and R. Urtasun. Mixsim: A hierarchical framework for mixed reality traffic simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9622-9631, 2023.
[92] H. Thomas, B. Agro, M. Gridseth, J. Zhang, and T. D. Barfoot. Self-supervised learning of lidar segmentation for autonomous indoor navigation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 14047-14053. IEEE, 2021.
[93] P. Polack, F. Altché, B. A'Andréa Novel, and A. de La Fortelle. The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles? In 2017 IEEE intelligent vehicles symposium (/V), pages 812-818. IEEE, 2017.
[94] Y. Xiong, W.-C Ma, Wang, and R. Urtasun. Learning compact representations for lidar completion and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1074-1083, 2023.
[95] Kim, Mahjourian, Ettinger, M. Bansal, B. White, B. Sapp, and D. Anguelov. Stop- net: Scalable trajectory and occupancy prediction for urban autonomous driving. In 2022 International Conference on Robotics and Automation (ICRA), pages 8957-8963, IEEE, 2022.
[96] Auer. Using confidence bounds for exploitation-exxllraaion trade-offs. Journal of Machine Learning Research, 3(Nov):397-423 2002.
[97] R. Liaw, E. Liang, R. Nishihara, Moritz, J. E. Gonzalez, and I. Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv: 1807.05 18, 2018.
14

---

Appendix
We provide additional details our method, implementation and experimental setups, and then show additional quantitative / qualitative results. We first detail how we implement ADV3D including how to build digital twins for realistic LiDAR simulation (Sec A.I), how to create low-dimensional representation space (Sec A.2) for adversarial shapes, the details of two modern autonomy models (Sec A.3), the adversarial optimization procedure (Sec A.4) and more experimental details. Finally then provide additional results and analysis in Sec B including full closed-loop and open-loop results for major tables in the main paper and additional experiments and additional qualitative examples. Additionally, we include a supplementary video in https : //wabbi....///// a ai/adv3d/, providing an overview of our methodology, well video results generated adversarial shapes and how it affect the autonomy performance in different scenarios.
A ADV3D Implementation Details
A.1 Realistic LiDAR Simulation
Following [72, 73], we leverage real-world LiDAR data and object annotations to build surfel meshes (textured by per-point intensity value) for the virtual world. For complete background coverage, we drove through the same scene to collect multiple sets of driving data and then unify multiple LiDAR sweeps to standard map coordinate system. We then aggregate LiDAR points from all frames and apply a dynamic point removal algorithm [92] to keep only static points and reconstruct the background B. For dynamic actors, we aggregate the LiDAR points within object-centric coordinate bounding boxes for each labeled driving snippet. We then symmetrize the aggregated points along the vehicle's 's heading axis for more complete shape. Given the aggregated points, then estimate per-point normals from 200 nearest neighbors with a radius of 20cm and orient the normals upwards for flat ground reconstruction, outwards for more complete dynamic actors. We downsample the LiDAR points into 4cm voxels and create per-point triangle faces (radius 5cm) according to the estimated normals. Due to sparse observations for most aggregated surfel meshes, manually curated a set of actor meshes that have complete and clean geometry, together with CAD assets purchased from TurboSquid [74] for a larger asset variety.
A.2 Adversarial Shape Representation
To ensure realism and watertight manifolds, we use all the CAD cars (sedan, sports car, SUV, Van, and pickup trucks) to build the low-dimensional representation. We first rescale all actors to be in a unit cube with a pre-computed scaling factor (1.1x largest dimension for all actors). Then we convert the input meshes to volumetric signed distance fields (SDF) with a resolution of 100 (iee, [L = 1003) using a open-source library?. Then we apply principal component analysis [78] on the flattened SDF values & E IR LL x1 to obtain the latent representation. Specifically, we use K principle components for constructing the latent space. In practice, we find the larger number use K, the high-frequency details can be captured but the interpolated shapes can be less realistic. This is because the non-major components usually capture the individual details instead of shared properties across all vehicles.
During optimization, we first obtain the minimum and maximum latent range 2min E RR3 7.nax E TR3 where the minimum and maximum value for each latent dimension is recorded. Then we optimize a unit vector E IR5, where the first three dimensions are for PCA reconstruction, and the last two dimension indicates the scale value for width and length (range from 0.8 to 1.3). Then we normalize this first three dimension to [imin maax] Given the optimized latent z. we apply Equation (2) in the main paper to get the generated 3D SDF volumes and then extract the meshes using marching cubes [79] algorithm. The extracted meshes are then scaled to the woal-world size and placed in the virtual world for simulation.
'nttps:/ggithub http::///iituuu..o////gegge
15

---

A.3 LiDAR-Basee -eaReB Autonomy Details
Both autonomy systems tested consist of two parts, where the first part uses different joint perception and prediction models, and the second part share the same ule-based planner [24]. We assume perfect control where the ego vehicle can follow the planned trajectory (kinematic bicycle model) exactly. In other words, the driver command policy brings the ego vehicle to the next state (10Hz) by executing current planning trajectory for 0. Taking instance-based autonomy for example, given the LiDAR points, the perception and prediction models output the actor locations at current and future timestamps. Then the sampling-based motion planner [24] produces the optimal plan that follows the kinematic bicycle model [93]. All the models are trained on highway driving datasets.
Autonomy-A (instanc--Base)): We implement a variant of joint P&P model [26] to perform instance-based joint detection and trajectory prediction. For the 3D object detection part, we use modified two-stage PIXOR [13] model following [94] which takes voxelized LiDAR point clouds as input and outputs the BEV bounding box parameters for each object. For the trajectory prediction part, we use a model that takes lane graph and detection results as input and outputs the per-timestep endpoint prediction for each object. The prediction time horizon is set to 6 seconds.
Autonomy-B (Instance-Free): We also verify our method on an instance-fres autonomy sys- tem [31] for joint detection and motion forecasting to show its generalizability. Specifically, replace the P&P model used in Autonomy-A A with the occupancy-basee model, which performs non-parama non-parametric binary occupancy prediction
perception results and flow prediction motion forecasting results for each query point on a query point set. The occupancy and flow prediction can serve the input for the sampling- based planner to perform motion planning afterwards.
A.4 Adversarial Optimization Details
Adversarial Objectives: The adversarial objective is given in Eqn. (3-5) in the main paper, where we set Xpred = 0.1 and Aplan = 0.5 for Autonomy-A. The adversarial objective for Autonomy-B also includes three terms: Ldet. fred and éplan- and we keep lpann as is since the PLT model we used is the same Autonomy-A. However, the ImplicitO model in Autonomy-B does not have instance-level bounding box results, and the confidence score as well as IoU terms are no longer applicable. We thus follow [31, 95] and use the Soft-IoU metric to assess occupancy predictions. Similarly, use the foreground mean end-poin error (EPE) to measure the average L.2 flow error at each occupied query point as done in [31], the instance-based trajectory prediction is not available. Formally, the adversarial objective for Autonomy-B is defined as:
C. Elee + Apred pred + Aplan
(6)
Eneo p
" Eqco(o(a) + o(q) )())(()))
(pred Eqee o(q)
Σ ol - f(g)lla, catmn =feek + clue> q
(8)
where Q is the query point set, o(q) and ō(q) are ground truth and predicted binary occupancy value E [0. 1] on the query point q. respectively. Flow vector f R - IR2 and the corresponding
prediction f specifies the BEV motion of any agent that occupies that location. We set Apred 1.0 and Aplan = 0.5 for Autonomy-B.
Black-box Optimization Details: To handle different modern autonomy systems and the non- differentiable LiDAR simulator, we adopt the black-box optimization in ADV3D. Inspired by existing works [56. 58, 61], we adopt the Bayesian Optimization [81, 82] (BO) as the search algorithm, which maintains a surrogate model and select the next query candidate based on historical observations and acquisition function. Specifically, we use a standard Gaussian process (GP) model with Upper Confidence Bound [96, 83] (UCB) as the acquisition function. We set the exploration multiplier 1.0 to balance exploitation and exploration. Since the adversarial landscape is not locally smooth, we use the Matérn 3/2 kernel (product over each dimension with a length scale of 0. 1) for the GP model. Unless stated otherwise, we set the total query budget as 100 and the first 11 queries are used for the initialization.
16

---

We also benchmark the other popular black-box algorithms including grid search [84] (GS), random search [85, 86. 87] (RS) and blend search (BS) [88]. For GS, we set search points per dimension thus in total 3 = 243 queries. For a random search, we set the query budget as 500 to achieve better performance. BS is an economical hyperparameter optimization algorithm that combines local search and global search. We adopt the official implementation' We also compare baseline that conducts brute-forcing (BF) over the curated asset library with 746 vehicles and find the worst-case actor shape. Our optimization pipeline is built on the Ray Tune framework [97].
A.5 Additional Experimental Details
Realism Evaluation for Generated Shapes: We evaluate the realism of ADV3D using divergence [89] (JSD) between generated shapes by ADV3D and vertex defor- mation (VD). Specifically, we calculate JSD by uniformly sampling point clouds of 1000 points from the optimized shapes with the birds-eye-view 2D histogram of all CAD models in our asset sets (resolution of 100 x 100).
A.6 Additional Discussions
Offroad or Collision Evaluation for Planning? We do not adopt offroad and collision as adver- sarial objectives or evaluation metrics. This because the planner [24] leverages map information to prevent offroad trajectories to ensure safety driving. Moreover, adding a collision loss requires careful formulation to provide stable supervision (such as through actor-ego distance or toee-oo-ollissoon, as the vanilla implementation results in sparse and noisy objective function to optimize. To ease optimization stability and enable fast convergence, losses that can be easily computed for perception, prediction, and planning at each timestep over the full scenario. We also do not modify actor behavior, making challenging generate collision outcomes in the SDV on real-world highway driving scenarios.
designed our objective function leverage
Gradient- attacks for Ady3D? sradient-based attacks require fully differentiable autonomy and simulation systems. Since the goal of Adv3D is to cater to any autonomy system, not just those that are fully differentiable, we chose the black-box algorithm BO to ensure broader applicability. For systems that are completely differentiable, one may pursue gradient- based white-box attacks with differentiable simulator, which can be stronger and more efficient.
B Additional Results and Analysis
To take into account the full autonomy stack, we find it is important to use an adversarial objective that takes a combination submodule costs. Tab, A7, we provide the full results for Tab. 3 in the main paper, including missing combinations M1 and M3. Moreover, we also compare with the results closed loop using shapes generated by open-loop attack.
Attacking Full Autonomy Stack:
Latent Asset Representation: We repeat the experiment from Tab. 3 but now using lower density 100 triangle mesh which has a lower dimension thus can be optimized with BO. Results in Tab. A8 show that large vertex deformation is required to achieve similar attack strength as Ady3D. Moreover, the vertex deformed actors are overly simplified and unrealistic with noticeable artifacts.
httt::///gitabb.co//nicccoots//
17

---

#ID
Opt. Settings
Perception Σ, Aa
Prediction
E pood
Planning X Saam
(%. (50.5)
AP1
(%, 800.5
minADE Ta ermr
meanADE To error
(m//a))
(vn/a")
Original
88.7
89.4
2.51
4.99
0.261
0.294
Lpep
80.4
87.6
2.01
4.95
0.256
0.301
Closed-Loop
69.6
71.4
1.97
5.02
0.239
0.310
Loop-Loop
83.5
88.5
2.52
5.39
0.223
0.341
Closed-Loop
89.1
2.92
6.34
0.254
0,412
Open-Loop
87.2
88.8
2.57
5.38
0.305
0.352
M,
Closed-Loop
88.3
2.94
6.03
0.434
M4
Closed-Loop
Open-Loop
79.9
85.9
2.57
5.35
0.231
0.353
M:
Closed-Loop
Open-Loop
81.2
84.3
2.57
5.60
0.333
0.401
0.253
Mo
Open-Lnop
85.5
87.7
2.73
5.99
0.262
0.372
Closed-Loop
Table A7: Full table of adversarial optimization for the full autonomy stack. Unlike existing works that consider sub-modules, ADV3D generates actor shapes that are challenging to all down- stream modules.
AP f (%, @0.5)
Recall f (%, @0.5)
minADE +
Jerk 4 (m/s²)
Algorithms
JSD
L2 error
Original
98.7
99.6
4.70
0.090
VD: 0.05m
98.7
99.6
5.04
0.090
0.057
VD: 0,1m
98.7
99.6
5.10
0.090
0.137
VD: 0.2m
98.7
99.6
5.16
0.090
0.253
VD: 0.5m
78.3
81.2
5.77
0.103
0.745
VD: 1.0m
45.5
48.5
5.79
0.155
0.758
ADV3D (ours)
50.3
55.8
7.87
0.111
0.175
Table A8: Compare with vertex deformation.
Adv3D
VD (5 cm)
VD (10 cm)
VD (20 cm)
VD (50 cm)
VD (100 cm)
Figure A6: Qualitative comparisons with Vertex-Deformation (VD). Top and bottom show side and top-down views respectively.
Additional Qualitative Examples: We provide more qualitative examples in Figure A7, A8 and A9 to show that ADV3D able to generate safety-critical actor shapes for autonomy testing with appearance coverage.
18

---

Original
Adv. Closed-Loop
Original
Adv. Closed-Loop
Missed detections of occluded vehicle reactive actors)
Figure A7: Qualitative examples of adversarial shape generation (non-reactive actors vs reactive actors). ADV3D is able to generate adversarial actors that cause detection failures due occlusion.
Original
Adv. Closed-Loop
Adv. Open-Loop
31.7
Missed detections of occluded vehicle
Figure A8: Qualitative examples of adversarial shape generation in closed loop open loop. ADV3D able generate adversarial actors that cause detection failures due occlusion but the open-loop counterpart fails.
Original
Adv. Closed-Loop
Multi-actor detection failure, planned trajectory affected
Figure A9: Qualitative examples of adversarial shape generation with multi actor attacks. ADV3D
creates three safety-critical shapes that cause detection failures for all front actors.
19


Reviews:
[{"review_id": "xDgxo8ZsAb", "reviewer": "Reviewer_xR6w", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a novel framework for adversarially attacking self-driving vehicle autonomy systems, ADV3D. It focuses on LiDAR-based systems and tried to attack and evaluate the full stack of self-driving autonomy systems, including perception, prediction, and planning. It attacks the system by generating realistic adversarial vehicle shapes around the autonomous agent. The adversarial vehicle shape is optimized in the latent space through a simple PCA-based vehicle shape encoder-decoder framework. Unlike previous methods, the proposed optimization takes into account the so-called closed-loop performance loss of the autonomous vehicle simulation. The experiments demonstrate its effectiveness in finding worse failure cases than baseline approaches in a captured dataset including 100 real autonomous driving scenes.", "summary_of_recommendation": "I am slightly on the negative side of the recommendation due to the weaknesses and questions I have raised above. The major concerns are the significance of the contribution and the justification of the algorithm design choice.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See the questions raised in the above section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I summarize the strengths and weaknesses of the paper below.\n\n**Strength**:\n\n1. The paper investigates an important problem (i.e. adversarial attacking) in self-driving to evaluate the robustness of autonomous driving systems. \n\n2. The idea of evaluating the performance of the system in a closed-loop manner is insightful.\n\n3. The experiments are well designed to demonstrate the effectiveness of various components of the method.\n\n**Weaknesses** and **Questions**:\n\n1. The main contribution of the paper seems to be incremental: a) a simple PCA-based latent space. b) a loss function taking into account the future trajectory of SDV induced by the modified vehicle shape.\n\n2. The definition of $W$ in Eq (2) is unclear. I suppose it is the PCA components computed from the whole CAD dataset?\n\n3. The selection of the PCA-based latent space of the vehicle shape needs more justification. The encoder/decoder in Eq (2) seems to not guarantee obtaining the same/similar output shape from a given input shape since $W$ is not invertible. Furthermore, how does this simple encoder/decoder approach compare to more sophisticated neural network based models (VAE)? It is necessary to include such experiments to validate the selection of the PCA-based latent space.\n\n4. The reason for selecting the closest actor in front of the SDV to be modified needs more discussion. The agents aside the SDV might also have a significant impact in the system's performance. Optimizing the selection of the modified actor(s) can be a valuable option.\n\n5. Figure 4 is hard to interpret. It is unclear how the green and blue arrows related to perception and prediction failures. And the numbers in the figure are also not defined.\n\n6. The comparison against vertex-wise noise baseline would be more meaningful if the average performance over the whole dataset is reported. Currently, it is biased since it only compares the methods on the ADV3D's best data. Also, did the authors also apply optimization over the vertex-wise noise for the baseline?\n\n7. The advantage of the proposed approach over the existing method is unclear. From Table 2, while the proposed method is averagely good across metrics, it is not the best one in most of them. Can we achieve better results by simply combining existing methods?\n\n8. How is the scalability of the method to multiple modified actors? In other words, hoe many optimization iterations are required to achieve a good result (compared to the baseline) when the number of modified actors increases?\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "wQaPPzHWFV", "reviewer": "Reviewer_htaN", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes Adv3D, a framework to stress-test autonomous driving approaches by generating challenging shapes of the road user immediately in front of the autonomous vehicle in a simulator. In doing so, Adv3D challenges LIDAR-based perception stacks as it manipulates the appearance towards shapes that harm the performance of the autonomy stack. \n\nThe shape manipulation is performed using adversarial shape optimization, where a learned latent 3D encoding of a vehicle is iteratively updated using Bayesian optimization such that the overall autonomy performance drops significantly in simulation.\n\nThe proposed approach is evaluated on a dataset containing 100 driving sequences, each 20s in duration. Each segment is replayed in a simulator while the LIDAR data is modified. To measure the impact of this altered sensor data, two perception systems are tested, both showing degraded performance when deployed on manipulated data.", "summary_of_recommendation": "The basis for my recommendation is comprised of the following observations:\n- The paper provides a limited interpretation of the presented results. Especially if the results are contradicting (such as Table 2), the paper should provide more interpretation and analysis of the obtained results. \n- The paper makes the misleading claim of attacking the entire autonomy stack, even though the presented adversarial attacks are limited to perception.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- The adversarial attacks proposed in this paper are focused on the perception system of a LIDAR-based autonomous vehicle. While the paper mentions the importance of closed-loop evaluation, it is not fully clear to me why this is the case. Why can the perception system not be evaluated on a dataset without closing the loop? Where does the difference between open-loop evaluation and closed-loop evaluation come from? One hypothesis might be that closing the loop results in novel viewing angles that result in poor performance when combined with adversarial shapes. It would be interesting to better understand why closed-loop perception is important in this application.\n\n- SOTA autonomous driving systems combine multiple sensor modalities for perception, such as cameras, radar, and LIDAR. One of the challenges that these systems face are for example challenging edge cases (for example: driving on a highway behind a car that is carrying a bike). Could the proposed approach be extended to generate LIDAR readings of such challenging scenarios?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strenghts\n=================\n- The use of a low-dimensional latent code for object shape parameterization and restricting the search space to realistic vehicle shapes is an elegant approach to mining difficult examples for stress-testing autonomous vehicles.\n- In general, the paper is written clearly and reads well.\n- The figures are well made and illustrate the generated adversarial shapes well. \n\nConstructive Criticism\n====================\n- The paper states that their method attacks the full autonomy stack. This statement is misleading in my opinion. The presented approach attacks the perception stack and closes the loop in simulation. Just closing the loop does not mean it attacks the full autonomy stack. Attacking the full autonomy stack would also include alterations to simulation such as the behavior of other road users or the dynamics of the simulated vehicle.\n\n- In Table 2, what method is being tested in this table? From my understanding, each row corresponds to a method that manipulates the LIDAR data, but what approach is used for perception and planning? Why do some rows contain contradicting results? For example: the second row (M1) results in very low AP and Recall, but performs best in lateral acceleration and second-to-best in jerk. It would be interesting to understand where this discrepancy is coming from. Extending the caption of table 2 to be more self-contained would help the reader understand the presented data. In general, more interpretation of the presented results would greatly improve the paper IMO.\n\n- The adversarial shape modification is based on a dataset of real-world LIDAR data and CAD assets. It would be interesting to understand how the performance obtained from the generated adversarial shapes compares to the performance obtained when using the original assets. Do the adversarial shapes degrade performance more than any shape existing in the dataset?\n\n- This is a very minor comment, but I wouldn't call 2000s or highway data a large-scale dataset.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "sVTseh-PST", "reviewer": "Reviewer_nxxv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a framework (ADV3D) that performs closed-loop sensor simulations for evaluating self-driving vehicles (SDVs). This framework focuses on LiDAR-based data. The proposed framework utilizes a low-dimensional shape representation to modify the vehicle shape. The experiments show that ADV3D is able to generate challenging actor shapes. Moreover, it is shown that the closed-loop is more powerful than attacking in the open-loop. ", "summary_of_recommendation": "Please refer to my review comments above.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please refer to my review comments above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The problem studied in this paper is interesting. The supplementary materials and the video are helpful to understand the proposed algorithm. \n\nSome comments are as follows:\n- In Sec. 3.2, the actor is placed using the pose based on S_t. Does this pose is known or it needs to be optimized?\n- Table 2 and Table 3 are a bit confusing. It seems that the proposed method does not outperform existing methods in general.\n- Typo in Sec. 3.2 paragraph 3. ", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "b8CgNYPIJfh", "reviewer": "Reviewer_BYfr", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an adversarial shape attack pipeline that targets the whole autonomous vehicle stack consisting of perception, prediction, and planning. The adversarial shape is searched by mapping a suite of common shapes into a latent space and performing a search there. Specifically, the authors starts from  CAD assets and turn them into SDFs, then apply PCA on the SDFs to obtain a latent representation together with the principal axes.\n\nThe simulation environment uses raycasting to simulate Lidar point cloud, then the AV either takes an instance-based stack to predict the environment motion and plan its own motion, or takes an implicit occupancy-based approach to plan its motion. The other agents follow a traffic model.\n\nThe adversarial search uses a Bayesian optimization approach that targets the whole stack and the result shows that the proposed approach is able to generate more realistic adversarial shapes than the baseline, and closed-loop training is superior. ", "summary_of_recommendation": "I think this paper proposes an interesting approach to adversarial attacks on AV. Particularly, the PCA approach that uses actual CAD assets is novel and significantly improves the realism of the adversarial shape attack. The engineering effort is commendable given that the authors built a functional closed-loop simulator with sensor simulation, a complete AV stack, and a traffic model for other agents. The results are nicely explained and make sense to me. Overall, I find this paper well-written, the research of high quality, and suitable to be published in CoRL.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. There are two autonomy stacks mentioned, yet in the result section they are not discussed separately, what is the actual setup?\n2. In table 1, it seems that open-loop attack sometimes result in better score than the baseline, can you explain why?\n3. When evaluating planning, my intuition is to use offroad or collision as the primary metrics, can the authors explain the reason behind their choice?\n4. BO can be applied to any black-box system, yet it is not scalable, can the authors comment on the practical use of BO v.s. gradient-based adversarial attack approaches?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall I think the paper is solid and well-written. The approach makes sense and the engineering effort is impressive. While closed-loop simulation remains challenging for a gradient-based approach to train on, the BO approach can certainly work. The SDF-then-PCA approach for generating adversarial shape is interesting and novel albeit simple. The experiments are well-designed with a high level of engineering implementation. The result shows the benefit of the approach, although I do have some questions (see next section).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews for this paper were a bit diverging. The reviewers generally agree that this paper addresses a timely problem (namely, that of stress-testing AV stacks) and is well executed, but also raise a number of concerns, in particular in terms of technical novelty. Upon reading the paper, the AC believes that the "system-level" contribution of this paper is indeed notable and outweighs its rather limited technical contribution. Thus, the AC recommends acceptance. In the final version of this paper, the authors should carefully revise the paper to address all reviewers' comments / suggestions.
Decision: Accept (Poster)