CAT: Closed-loop Adversarial Training for Safe End-to-End Driving
Authors: Linrui Zhang
Zhenghao Peng
Quanyi Li
Bolei Zhou
Venue: CoRL 2023 Poster (2023)
Keywords: Safety-Critical Scenario Generation
Adversarial Training
End-to-End Driving

Abstract: Driving safety is a top priority for autonomous vehicles. Orthogonal to prior work handling accident-prone traffic events by algorithm designs at the policy level, we investigate a \textbf{C}losed-loop \textbf{A}dversarial \textbf{T}raining (CAT) framework for safe end-to-end driving in this paper through the lens of environment augmentation. CAT aims to continuously improve the safety of driving agents by training the agent on safety-critical scenarios that are dynamically generated over time. A novel resampling technique is developed to turn log-replay real-world driving scenarios into safety-critical ones via probabilistic factorization, where the adversarial traffic generation is modeled as the multiplication of standard motion prediction sub-problems. Consequently, CAT can launch more efficient physical attacks compared to existing safety-critical scenario generation methods and yields a significantly less computational cost in the iterative learning pipeline. We incorporate CAT into the MetaDrive simulator and validate our approach on hundreds of driving scenarios imported from real-world driving datasets. Experimental results demonstrate that CAT can effectively generate adversarial scenarios countering the agent being trained. After training, the agent can achieve superior driving safety in both log-replay and safety-critical traffic scenarios on the held-out test set. Code and data are available at: https://metadriverse.github.io/cat

CAT: Closed-loop Adversarial Training for Safe End-to-End Driving
Linrui Zhang Zhenghao Peng', Quanyi Lis, Bolei Zhou' Tsinghua University, 3 UCLA, The University of Edinburgh
Driving safety is a top priority for autonomous vehicles, Orthogonal to prior work handling accident-proom pronn traffic events by algorithm designs at the policy level, we investigate a Closed-loop Adversarial Training (CAT) framework for safe end-to-end driving in this paper through the lens of environment augmen- tation. CAT aims continuously improve the safety of driving agents by training the agent on safety-critical scenarios that are dynamically generated over time. A novel resampling technique is developed to turn log-replay real-world driving scenarios into safety-critical ones via probabilistic factorization, where the ad- versarial traffic generation is modeled as the multiplication of standard motion prediction sub-problems. Consequently, CAT can launch more efficient physi- cal attacks compared to existing safety-critical scenario generation methods and yields significantly less computational cost in the iterative learning pipeline. We incorporate CAT into the MetaDrive simulator and validate approach on hundreds of driving scenarios imported from real-world driving datasets. Experi- mental results demonstrate that CAT can effectively generate adversarial scenarios countering the agent being trained. After training, the agent can achieve superior driving safety in both log-replay and safety-critical traffic scenarios on the held- out test set. Code and data are available at https//mettadriverse github.io/cat.
Abstract:
Introduction
While end-to-end driving has achieved promising performance in urban piloting [1] and track rac- ing [2], safely handling accident-prone traffic events is one of the crucial capabilities to achieve for autonomous driving (AD). Benchmarking the safety and performance of an Al driving agent in simulation is a stepping stone for the real-world deployment [3]. However, it is insufficient to train or evaluate an end-to-end driving agents on traffic scenarios only retrieved from real-world traffic datasets [4, 5] since accident-prone events are extremely rare and difficult to collect in practice [6, 7].
Prior work improves the driving agent against safety-critical scenarios through various methods such as buse- reasoning [8], motion verification [9], and constrained reinforcement learning [10]. Or- thogonal to the elaborate algorithm designs at the policy level, recent studies obtain robust driving policies at the environment level by creating a set accident- scenarios before hand as aug- mented training samples [11, 12]. Nevertheless, the learned policy may still easily overfit the fixed set of training samples thus fail to handle unknown hazards [13].
An alternate approach is to dynamically generate challenging scenarios that match the current pability of the driving agent being trained in a closed-loop manner. However, the state-of-the-an safety-critical scenario generation methods [11, 12, 14] are not yet applicable for that purpose due to the following issues: (i) Scene generalizability probabilistic graph methods like CausalAF [11] require human prior knowledge of each scene graph and thus cannot scale to large and complex driving datasets; (ii) Model dependency: kinematics gradient methods like KING [12] relies on the forward simulation of the running policy and the backward propagation based the environmental transition, which might not be accessible in the model-free end-to-end driving; (ii) Time efficiency:
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Closed-loop Adversarial Training (CAT)
Ego Vehicle
Opponent Vehicle
Iinelevant Vehick
DRIVE SAVE
Environment
Safity-Crtticll
Driving Policy
Inirialization
Resampling
Oprimization
Scenario
Traflic
Traffic Scenario
Aware Driving
Safe AD Policy
Figure 1: CAT iterates over safety-critical scenario generation and driving policy optimization in closed-loop manner. In this example, the safety-critical resampling technique alters the behavior of the opponent vehicle (blue car) such that it suddenly cuts into the lane of the ego vehicle (red car), enforcing the agent to learn risk-aware driving skills such as deceleration and yielding.
autoregression-basee generation methods like STRIVE [14] take minutes to optimize the adversarial traffic per scenario, which is time prohibitive for large-scale training with millions of episodes.
In this paper, present the Closed-loop Adversarial Training (CAT) framework for safe end-to-end driving. As shown in Fig. CAT imports driving scenarios from real-world driving logs and then generates safety-critical counterparts adversarial training environments tailored to the current driving policy. The agent continuously learns to address emerging challenges and improves risk awareness in a closed-loop pipeline. CAT directly launches physical attacks against the estimated ego trajectory, the proposed framework is thus agnostic to the driving policy used by the agent and compatible with a wide range of end-to-end learning approaches, such as reinforcement learning (RL) [15], imitation learning (IL) [16], and numan-in-the-lll feedback (HF) [17].
One crucial component of the proposed framework is a novel factorized safety-critical resampling technique that efficiently turns logged driving scenarios into safety-critical critical ones during training. Specifically, we cast the safety-critical traffic generation as the risk-conditioned Bayesian probability maximization and then decompose into the multiplication standard motion forecasting sub- problems. Thus, we can utilize off-the-shelf motion forecasting models [18. 19] as the learned prior generate adversarial scenarios with high fidelity, diversity, and efficiency. Compared to previous safety-critical traffic generation methods, the proposed technique obtains competitive attack success rate while significantly reducing the computational cost, making the CAT framework effective and efficient for closed-loop end-to-end driving policy training.
To demonstrate the efficacy of our approach, we incorporate the proposed CAT framework into the MetaDrive simulator [20] and compose adversarial traffic environments from five hundred complex driving scenarios in closed-loop manner to train RL-based driving agents without any ad-hoc safety designs. Experimental results show that CAT generates realistic and challenging physical attacks, and the resulting agent obtains superior driving safety in both log-replay and adversarial traffic scenarios on the held-out test set. The contributions of this paper are summarized as follows:
i) We propose an efficient safety-critical scenario generation technique by resampling the learned traffic prior, which improves attack success rate and lowers computation cost compared prior work, making continuous adversarial scenario generation viable in closed-loop AD training.
ii) We present closed-loop adversarial training framework for end-to-end safe driving based on the above technique and demonstrate the proposed framework substantially improves AI driving safety complex testing scenarios imported from the real world.
2 Related Work
Adversarial Training for Autonomous Driving, Deep neural networks (DNNs), pervasively used in barning-based AD systems, are found vulnerable to adversarial attacks [21, 22]. Recent stud-

---

ies tend to manipulate the physical environment to generate realistic yet adversarial observation sequences from LiDAR inputs [23], camera inputs [24), and other physical-world-rsiliee ob- jectives [25]. Compared to the above work focusing on perception, adversarial training for AD decision-making is much less explored. Ma et al. [26] first investigate the adversarial RL on a single autonomous driving scenario. Wachi [27] employs the multi-agent DDPG algorithm [28] to enforce the competition between player and non-player vehicles. In addition to algorithmic level designs, more natural but less explored approach is to iteratively propose challenging scenarios during train- ing [29]. There is line of works on evolving training environments in RL [30, 31J. However, existing approaches are evaluated only in simplified environments like bipedal walker and heuris- tically modify the terrain or static barriers, which is not sufficient for complex AD tasks. In this work, we focus on generating realistic and safety-critical traffic scenarios to facilitate closed-loop adversarial training for end-to-end driving.
Safety-critical Traffic Scenario Generation. Safety-critical traffic scenario generation is of great value in adaptive stress testing [32] and corner case analysis [33] for the research and development of autonomous vehicles. 1.2C [34] learns to place and trigger a cyclist to collide with the target vehicle via RL algorithms, but it goes far to model complex vehicle interactions in real-world scenes. For robust imitation learning, kinematics gradients [12] and black-box optimization [23] can be used to magnify traffic risks. However, relies on the forward simulation of the running policy and the backward propagation based the vehicle kinematics, which might not be accessible in model-free end-to-end driving. CausalAF [11] builds scenario causal graphs to uncover behavior of interest and generates additional training samples to improve the robustness of driving policies. Nevertheless, the evaluations are limited to three scenarios since it requires human prior knowledge of each scene and thus hardly scale to larger dataset. STRIVE [14] constructs a latent space to constrain the traffic prior and searches for the best responsive mapping via gradient- representation. Despite its impressive results on realistic traffic flows, the autoregression on raster maps takes several minutes to optimize the adversarial traffic for each scene, which brings about costly computational burden for periodic policy optimization. We refer to the survey [35] for more detailed safety-critical scenario generation methodologies. Different from the above literature, we propose a novel adversarial traffic generation algorithm for real-world scenarios with an admissible time consumption, making viable for large-scale policy iterations involving millions of episodes.
optimization on that dense
3 Method
In this section, we first formulate the closed-loop adversarial training (CAT) for safe end-to-end driving as min-max problem in the context of RI., and then introduce the factorization of the learned traffic prior so as to generate adversarial driving scenarios efficiently in practice.
3.1 Problem Formulation
End-to-end driving directly uses raw sensor data as the inputs and outputs the low-level control command. Safe end-to-end driving incorporates risk-awareness into the above end-to-end pipeline and aims to minimize traffic accidents while maintaining the performance of route completion. We focus on reinforcement learning bRL)-based driving policy in this work, though the proposed CAT can be extended to accommodate range of end-to-end driving policies. In our scope, the driving task can be formulated as Markov Decision Process (MDP) [36] in the form of (S, R. f). S and A denote the state and action spaces, respectively. S includes maps sensor readings such as camera images or LiDAR point cloud, high-level navigation commands and vehicle states. A consists of low-level control commands like steering, throttle and brake. The reward function can be defined as R = 77, wherein is the displacement toward the destination, c is a boolean value indicating collision with other objects and 1) hyper-parameter for the reward shaping. f is the transition function to describe the dynamics of the traffic scenario. The goal is to maximize the expected return J (TT, f)) Eraz [Σt= R(st, ae) the driving policy π receives within the time horizon T, where is short handed for a (.1ss), St+1 f(-lst,;;) CAT aims to enhance the robustness of
3

---

the learning agent via the following adversarial optimization:
max min [ f fAdvey
(1)
Here, the adversarial transition function fAdo must be within the feasible set J that is aligned with realistic traffic distribution, otherwise the learned driving policy is not applicable in practice.
The fundamental problem is to construct f by generating compliant future traffic trajectories that are prone to collisions with the agent's rollouts. To formalize the traffic collision, we denote the vehicle controlled by the learning agent as the ego vehicle (Ego) and other vehicles as opponent vehicles (Op) and represent traffic scenario as tuple (M, S S
1:rr) with duration 7 time steps. Here, the High- High-Definition (HD) road map M consists of road shapes, traffic signs, traffic lights, etc. SE denotes the past states of the ego vehicle. 5 an N-element array [ ..... SPPy wherein denote X (M, S lit as the information cutoff by step and E St. Yop S are the future trajectories of ego and opponent starting from 4, respectively. Y is conditioned on the RL agent л. The cutoff step t is fixed. We define binary random variable Coll = rTuree, F False] to denote whether yEgo collides with Y Considering that the opponent vehicle must launch effective attacks based on the potential ego behavior which is responsive to the Yop, the opponents' trajectories yOp and the ego vehicle's trajectory yT are thus not independent. Therefore, we model OP and y EEgo jointly and the safety-critical scenario distribution is expressed as:
each element stands for the past states of the corresponding opponent. For simplicity,
sEgo
Pay Y (Coll True, X)
Proposition 1 further shows that the construction of fAdd can be cast marginal probability max- imization of opponent trajectories Y based on the above joint posterior distribution, where assume that E generated by the current driving policy is sampled from ((x).
Proposition 1. Suppose that forces the agent to approach the destination and the episode termi- nates when any traffic collision happens, then we have
f [(, Ado) + max
min
Σ
P(
y yOolCool= True, X).
Y
YE
3.2 Factorized Cafety-Critical Resampling
The joint distribution in Eq. (3) is still intractable. However, under the assumptions that the ego hicle's reactions are unidirectionally based on the future traffic, can factorize it with the Bayesian formula as shown in Proposition 2.
Proposition 2. Suppose that Ese depends Op unidirectionally: then we have Es y = True, X) x IP Op (Y y X)PP(Col = True Y (4)
After the factorization, we can search the best responsive "YOpp to magnify the probability of traffic collisions with the ego agent as possible through the marginal probability maximization given as:
ERo Y True, X)
(5)
P(YOpp
Σ
P(y Op X) P(Coll = True yop)
= max Yo
1st Term
2nd Torm
3rd Term
is beneficial to perform the above safety-critical traffic probability factorization since each term in Eq. (5) features a specific meaning and tractable to handle. They are interpreted as follows:
i) Traffic prior. The 1st term the standard motion prediction problem in which we can lever- age arbitrary probabilistic traffic models [18, 37, 38, 39] to portray the multi-modal trajectory distribution. Taking the pre- treitriined model as the traffic prior enables the attack plausibility in complex scenarios without human specifications.

---

Ego Vehicle
Ego Trajectory
Opponent Vchicle
Oppenent Trajectory
Figure 2: Illustration of Factorized Safety-Critice Resampling- (A) We initialize Is traffic history with the dense map representation. (B) We then predict the traffic prior as well as the agent's reaction. (C) The most accident-proct trajectory of the opponent vehicle is selected. (D) The generated scene is thus expected to be safety-critical.
ii) Ego estimation. The 2nd term denotes the interactive ego trajectory yielding to the current state and upcoming traffic flow. The transition can be deterministic the world model learned or accessible under model-based settings [12]. As for the inference of real-world-complian traffic flows, we can employ an interactive motion predictor [19] conditioned on known surrounding vehicles' trajectories better reflects the ego compliance under risky interactions.
iii Collision likelihood. The 3rd term reflects the likelihood of a collision in the compositional future, which can be simulated directly or treated as a binary classifier to fit [40].
As shown in Fig. 2, it is possible to approach the near-optimal adversarial trajectory via numerical optimization after each term is calculated.
3.3 Practical Implementation
We summarize the overall implementation of the CAT framework for safe end-to-end driving in Algorithm 1. Recalling the training objective of CAT in Eq. (1), we need to perform iterative opti- mization of policy learning and adversarial environment generation synchronously The policy optimization can be achieved by arbitrary end-to-end driving policy learning approaches, e.g., vanilla RL algorithm. Below, we focus the adversarial environment generation, where we utilize the proposed factorized safety-critical resampling in Eq. (5). Note that make a sim- plification in CAT by enforcing a single rival to launch the attack in each generated scene while simply maneuvering the other vehicles to avoid self-collisions. This is reasonable since most traffic accidents are caused by two traffic participants rather than involving multiple vehicles.
closed loop.
We first predict the traffic prior PPY using a pre-trained probabilistic traffic forecasting model G. Considering the strong performance and the ease of sampling, anchor-free goal-based motion predictor, in this work. Specifically, we propose M possible candi- dates {(Y } in parallel. The component i.A in the k-th time step consists of the predicted position and yaw of the opponent vehicle. The probability of the trajectory p.u coincides with the probability of the corresponding destination goal.
adopt DenseTNT [18), an
We then tackle the ego estimation term poy XX). Considering the non-stationary policy during training, notice that the ego behavior does not necessarily match the logged behavior in the dataset. Consequently, directly utilizing the pre-trained traffic estimator derived from natural traffic flows [19] to provide ego trajectory probability has severe bias. Alternatively, we record the latest rollouts the ego vehicle each scenario formed as { _ } wherein we derive
the likelihood of visited state sequences deduced by the current policy w: plgg -PEgo
I (aa (sx)-
j,k+1 j.k
At last, we empirically estimate the collision likelihood PPCool]|yEgg, yoP). Given the specific compositional future of y and YOp, we compute the minimal distance between their bounding boxes in the following steps and set the collision likelihood as pColl a if the closest gap at timestep k. If the collisions happen multiple step, the earliest & will be used. Here, (0,1) is a heuristic decay factor to reflect the increasing uncertainty of the traffic model.
5

---

Algorithm 1: Closed-loop Adversarial Training (CAT) for Safe End-to-End Driving.
Input:
Initial driving policy и, learning algorithm T, trajectory predictor G. the simulator.
Output: Robust driving policy n².
Initialize the scenario pool D {Xx,, X2. from real-world datasets.
Initialize the ego trajectory buffer for each scenario.
while is not converged do
Randomly sample logged traffic X from the scenario pool D. Retrieve the ego trajectory bufter for this scenario {(Y, _. ) {[x,0) )
((X)
Generate the traffic prior, M Op's trajectories. For each Op candidate. > For each Ego candidate.
for 1, 2,,, M do
for j in 1, 1,2,... N do
a
Fay collides with BBox(X, (2)0) at step k; if BBox j.k
PC
0
otherwise.
P(Y TT, Coll, X) P.p E P
Compute the posterior probability.
11
yup arg y P(V [,, X)
Select the best Op's trajectory. Reset sim replay the adversarial scenario. Initialize F (). P = for / in 1,2, do Rollout the policy against the
obs simulator.reset(X yopr)
13
adversarial scenario.
14
act ~ 7 (-|obs))
15
obs = sitelaton.step(acct))
16
y + y U@x
b Update Ego trajectory. b Update Ego probability.
17
pF PF (act)obb))
+ T(x) 18
b Policy optimization.
Add (YFs), pFen) to the ego trajectory buffer for this scenario.
19
4 Experiments
4.1 Experiment Setup
We import 500 real-world traffic scenarios involving complex vehicle interactions from the Waymo Open Motion Dataset (WOMD) [4] as the raw data. Each scene WOMD contains a traffic partic- ipant labeled as Object of Interest regarding the ego car, which is also designated as the opponent vehicle in our experiments. All the experiments are conducted in MetaDrive [20], an open-souree and lightweight AD simulator. The specific state, action and reward function in policy training and detailed hyper-parameter settings in safety-critical scenario generation and D. Here, we point out some pivotal parameters, Each scene lasts 9s, in which take the first Is traffic history as X and manipulate the following 8s to generate the adversarial trajectory yOp, We set 32 as the number opponent trajectory candidates, N = as the length of ego rollout queue and a = 0.99 to penalize the uncertainty motion forecasting.
placed in Appendix C
4.2 Evaluation of Safety-critical Traffic Generation in CAT
The factorized safety-critical resampling is the crucial component of CAT to generate adversarial training samples. We provide qualitative and quantitative comparisons with the following baselines: (A) Raw Data: Replaying the recorded real-world traffic. (B) M21 (adv) [19]: The interactive traffic motion prediction similar to our factorized formulation and thus can be modified as an adversar- ial scenario generator. (C) STRIVE [14]: The state-of-the-art safety-critical scenario generation methods performing gradient-basee optimization on latent variables.
Qualitative analysis. Fig. 3, we present 9 different types of safety-critical scenarios that CAT generates from raw scenes, according to the pre-crashed traffic categorized by the National Highway Traffic Safety Administration (NHTSA). It can be concluded that CAT is able to generate adversarial traffic given arbitrary real-world scenes. Meanwhile, the generated trajectories human driver behavior, even though we don't specify prior knowledge of that scene. In Fig. 4,
in line with

---

Barw
Adv
Rano
Ad:
Raw
Adv
(1) Right Turn
(2) Left Turn
(3) U-Tum
Raow
Adc
Rink
Adk
Raw
Adv
(4) Rear-Eine
(5) Emergent Brake
(6) Lane Change
Raw
Adl
Raw
Rawn
skdee
(7) Cross Paths
(8) Run-OfT-Rewe
(9) Opposite Direction
Figure 3: Qualitative results on the diversity of safety-critical scenarios generated by CAT, In each subfigure, the left and right are the raw scene and the adversarial counterpart. The ego and adversar- ial trajectories are highlighted with red and blue arrows, respectively.
Ego Vehicle
Opponent Vehicle
Vehicle
Triffic Light
No Accident
Off the Driveway
Elective Attack
(A) Raw
(B) M21
STRIVE
CAT
Figure 4: Qualitative results on the plausibility of safety-critical scenarios generated by CAT. The attack is regarded as effective only if leading traffic accidents are consistent with real-world events.
compare the generated adversarial traffic of the four methods on the same intersection. In the raw scene, the leading vehicle turns preferentially and does not cross the path of the ego vehicle. The opponent attempts to collide with the agent at the intersection through the safety-criticll generation. However, M2I (adv) has a bias in estimating the reaction of the ego vehicle, which does not cause the expected accident. STRIVE finds the solution to enforce crash, but it is still cumbersome to tweak the multinomial loss function balance the goal of colliding as soon as possible and reasonable driving behavior, like keeping the vehicle in the driveway. By contrast, our factorized safety-critical critical resampling leverages the learned motion prior to regularize the opponent's 's trajectory, magnifying the traffic risk while preserving its plausibility. More visualization can be found in Appendix E.
Quantitative analysis. In Tab. 1, we com- pare adversarial traffic generation methods on 100 test scenes, focusing on two met- rics. The first metric of interest is the attack success rate as the driving policies are re- sponsive and even defensive to the traffic flow. We adopt three kinds of agents with fixed policies to validate: (i) Replay Agent: Replay the original trajectory of the ego ve-
Table I: Comparing adversarial generation methods.
Attack Success Rate f
Per Scene Generation Time L
Methods
Replay
IDM
Pretrnined
34%
14%
M21 (adv)
47%
41%
19%
0.41 + 0.03s 153. = 47.33s
STRIVE
85%
82%
60%
CAT (N 1)
91%
71%
62%
0.66 + 0.09s
CAT
91%
86%
69%
3.34 + 0.41s
hicle logged in real-world data-set. (ii) IDM Agent: A heuristic controller well-adopted in AD tasks [41]. (iii) Pre-trained Agent: A pre-trained RL policy on WOMD. We find that M2I (adv) is insufficient for ego prediction and attacks less effeetively especially against low-level policy, which fatal for end-to-end driving. CAT collects ego rollouts to enhance the confidence of ego estima- tion during training (N 5) and testing (N which significantly improves the attack success rate and is competitive with the SOTA method STRIVE. The second metric of interest is the time consumption per scene, which is neg-nggli considering the large number of scenario iterations
7

---

Completion (Log
Crash replay)
Complete (
Rate (Safety-critical)
75%
30%
75%
50%
25%
45%
65%
63%
40%
20%
35%
55%
15%
55%
30%
50%
10%
Total Interactions
105
105 Total Interactions
105
Interactions
105
No Adv/ Replay
busee Adv
Open-loop Adv
Closed-loop -oop Adv
Figure 5: The learning curves of the policies trained with different pipelines.
Table 2: Performance of driving policies with different training pipelines on the held-out test set.
Log-replay Scenarios
Safety-critical Scenarios
Methods
Route Completion
Crush Rate
Route Completion
Crush Rate
No Adv/ Replay
72.91% + 2.05%
19.89% 1.95%
63.48% 1.46%
43.33% + 1.13%
Rule-based Adv
62.42% + 3.99%
15.61% + 1.98%
56.68% + 4.66%
30.31% + 3.33%
Open-loop Adv
68.89% + 1.05%
17.15% 1.80%
63.48% 1.46%
36.96% + 1.00%
Closed-loop Adv
72.47% + 2.04%
13.43% 0.88%
67.62% + 1.89%
28.15% 1.63%
during training. We find that STRIVE generally requires 2-3 minutes to process single scene due to its autoregression procedure the raster map, which means it takes days train the agent in closed loop involving thousands of episodes. By contrast, approach best balance the attack success rate and computational time compared and admits privileged advantage in closed-loop adversarial training for end-to-end driving.
4.3 Evaluation of Closed-loop Adversarial Training in CAT
We show how the driving agent improves its safety performance within CAT framework. We split the 500 raw scenes into 400 training and 100 testing scenarios. We train TD3 [42] driving policy from scratch with types of training pipelines:
(A) No Adv/ Replay: The raw driving scenarios (B) Rule-based Adv: We implement rule-based system that overwrites the trajectories in data to generate physical attacks (see the Appendix F for details). (C) Open-loop Adv: We generate the opponent trajectories that collide with the ego trajectories against the log-replayed ego rollout before training.
used as the training environments.
(D) Closed-loop Ady: We use CAT to generate adversarial scenario on-the-fly against the ego trajectories generated by the learning agent.
We evaluate the driving policies trained from different pipelines with metries. The first metric is the route completion rate, which measures the progress the agent makes; The second metric is the crash rate, the ratio of episodes that the ego vehicle crashes into others. We first evaluate the policy on the held-out testing scenarios with logged traffic (Log-replay Scenarios). Then we run CAT against the policy to generate adversarial traffic. Finally, we run the policy in the testing scenarios with CAT-generated traffic (Safety-critical Scenarios). As shown in Table and Fig. 5, find that CAT substantially enhances safety performance compared with vanilla RL training, reducing crash rate by 6.46% in log-replayed scenarios and 15.18% in safety-critical ones with competitive route completion. More qualitative results can be referred in Appendix G. Besides, demonstrate that generating adversarial environments against current policy on-the-fly makes the trained policy performs better. At last, factorized safety-critical resampling can preserve the realistic traffic distribution so the learned policy has competitive route completion rate. On the contrary, the rule-based attacks lead to over-conservative driving policy that has inferior route completion.
5
Conclusion
In this paper, we investigate how to improve the safety of end-to-end driving through the lens of safety-critical traffic scenario augmentation. Empirical results demonstrate that the proposed closed- loop adversarial training (CAT) framework can provide realistic physical attacks efficiently during training and enhance AI driving safety performance in the test time.
8

---

Acknowledgments
This work was supported by the National Science Foundation under Grant No. 2235012 and the Cisco Faculty Award.
References
[1] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. Conference on robot learning, pages 1-16. PMLR, 2017.
[2] J. Herman, J. Francis, S. Ganju, B. Chen, A. Koul, A. Gupta, A. Skabelkin, I. Zhukov, M. Kum- skoy, and E. Nyberg. Learn-to-race: A multimodal control environment for autonomous rac- ing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9793-9802, 2021.
[3] C. Xu, W. Ding, W. Lyu, Liu, S. Wang, Y. He, H. Hu, D. Zhao, and Li. Safebench: benchmarking platform for safety evaluation of autonomous vehicles. arXiv preprint 2022.
[4] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9710-9719, 2021.
[5] H. Caesar, J. Kabzan, K. S. Tan, W. K. Fong, E. Wolff, A. LLngg L. Fletcher, O. Beijbom, and S. Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint urXiv:2106. 1810, 2021.
[6] M. Favarò, Nader, O. Eurich, M. Tripp. and N. Varadaraju. Examining accident reports involving autonomous vehicles in california. PLoS one, 12(9))e0184952, 2017.
[7] A. Sinha, S. Chand, V. Vu, H. Chen, and V. Dixit. Crash and disengagement data of au- tonomous vehicles on public roads in california. Scientific data, 8(1):298, 2021.
[8] B. Mirchevska, C. Pek, M. Werling, M. Althoff, and Boedecker. High-level decision making for safe and reasonable autonomous lane changing using reinforcement learning. In 2018 2 International Conference on Intelligent Transportation Systems (ITSC), pages 2156-2162. IEEE, 2018.
[9] D. Isele, A. Nakhaei, and K. Fujimura. Safe reinforcement learning on autonomous vehicles. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-6. IEEE, 2018.
[10] L. Wen, J. Duan, S. E. Li, Xu, and H. Peng. Safe reinforcement learning for autonomous vehicles through parallel constrained policy optimization. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1-7. IEEE, 2020.
[11] W. Ding, Lin, Li, and D. Zhao. Causalaf: Causal autoregressive flow for safety-critical driving scenario generation. In Conference on Robot Learning, pages 812-823 PMLR, 2023.
[12] N. Hanselmann, K. Renz, K. Chitta, A. Bhattacharyya, and A. Geiger. King: Generating safety-critical driving scenarios for robust imitation via kinematics gradients. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Pro- ceedings, Part XXXVIII, pages 335-352. Springer, 2022.
[13] D. Katare, N. Kourtellis, S. Park, D. Perino, M. Janssen, and A. Y. Ding. Bias detection and generalization in ai algorithms edge for autonomous driving. In 2022 JEEE/ACM 7th Symposium Edge Computing (SEC), pages 342-348. IEEE, 2022.

---

[14] D. Rempe, J. Philion, L. J. Guibas, S. Fidler, and O. Litany. Generating useful accident-promm driving scenarios via a learned traffic prior. In Proceedings of the IEEFJCVF Conference on Computer Vision and Pattern Recognition, pages 17305-17315 2022.
[15] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Alllab, S. Yogamani, and P. Pérez. Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions on Intelligent Transportation Systems, 23(6))4909-4999
2021.
[16] Z. Zhu and H. Zhao. A survey of deep rl and il for autonomous driving policy learning. IEEE Transactions on Intelligent Transportation Systems, 1303):1403314465,, 2021.
[17] Z. Peng, Q Li, C. Liu, and B. Zhou. Safe driving via expert guided policy optimization. In Conference on Robot Learning, pages 1554-1563. PMLR, 2022.
[18] J. Gu, C. Sun, and H. Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 15303- 15312, 2021.
[19] Q. Sun, X. Huang, J. Gu, B. C. Williams, and H. Zhao. M2i: From factored marginal trajectory prediction to interactive prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6543-6552, 2022.
[20] Li, Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning, IEEE transactions paltern analysis and machine intelligence, 2022.
[21] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposimm security and privacy (sp), pages 39-57. Icce, 2017.
[22] Q. Zhang, Hu, Sun, Q. A. Chen, and Z. M. Mao. On adversarial robustness of trajectory prediction for autonomous vehicles. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15159-15 2022.
[23] Wang, A. Pun, Tu, Manivasagam, A. Sadat, S. Casas, M. Ren, and R. Urtasun. Advsim: Generating safety-critical scenarios for self-driving vehicles. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9909-9918, 2021.
[24] A. Boloor, X. He, C. Gill, Y. Vorobeychik, and X. Zhang. Simple physical adversarial examples against end-to-end autonomous driving models. In 2019 IEEE International Conference Embedded Software and Systems (ICESS), pages 1-7. IEEE, 2019.
[25] Z. Kong, J. Guo, Li, and Liu. Physgan: Generating ahysical-worldrreslieen adversarial examples for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14254-14263, 2020.
[26] X. Ma, K. Driggs-Campbell, and M. Kochenderfer. Improved robustness and safety for autonomous vehicle control with adversarial reinforcement learning. In 2018 IEEE Intelligent Vehicles Symposium (/V), pages 1665-1671. IEEF, 2018.
[27] A. Wachi. Failure-scenario maker for rule-based agent using multi-agent adversarial reinforce- ment learning and its application to autonomous driving. arXiv preprint urXiv:/903 10654, 2019.
[28] R. Lowe, Y. I. Wu, Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor- critic for mixed sooperative-compettiiy environments. Advances in neural information pro- cessing systems, 30, 2017.
[29] L. Anzalone, P. Barra, S. Barra, A. Castiglione, and M. Nappi. An end-to-end curriculum learning approach for autonomous driving scenarios. IEEE Transactions Intelligent Trans- portation Systems, 23(10):19817-1922 2022.
10

---

[30] R. Wang, J. Lehman, J. Clune, and K. O. Stanley. Poet: open-ended coevolution of environ- ments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Compu- tation Conference, pages 142-151, 2019.
[31] R. Wang, J. Lehman, A. Rawal, J. Zhi, Y. J. Clune, and K. Stanley, Enhanced poet: Open- ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning, pages 9940-9951. PMLR, 2020.
[32] Z. Zhong, Y. Tang, Y. Zhou, V. d. O. Neves, Y. Liu, and B. Ray. A survey on scenario- based testing for automated driving systems in high-fidelity simulation. arXiv preprint arXi::21 2021.
[33] S. Riedmaier, Ponn, D. Ludwig, B. Schick, and F. Diermeyer. Survey on scenario-based safety assessment of automated vehicles. IEEE access, 8:87456-87477, 2020.
[34] W. Ding, B. Chen, M. Xu, and D. Zhao. Learning to collide: An adaptive safety-critich sce- narios generating method. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2243-2250. IEEE, 2020.
[35] W. Ding, C. Xu, M. Arief, H. Lin, B. Li, and D. Zhao. A survey on safety-critical driving scenario generation- a methodological perspective. IEEE Transactions on Intelligent Trans- portation Systems, 2023.
[36] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 1998.
[37] Gilles, Sabatini, D. Tsishkou, Stanciulescu, and Moutarde. Home: Heatmap output for future motion estimation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 500-507, IEEE, 2021.
[38] B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cornman, K. Chen, B. Douillard, C. Lam, D. Anguelov, al. Multipath++:: Efficient information fusion and trajectory aggregation for behavior prediction. In 2022 International Conference on Robotics and Automation (ICRA), pages 7814-7887 7821. IEEE, 2022.
[39] S. Shi, L. Jiang, D. Dai, and B. Schiele. Motion transformer with global intention localization and local movement refinement. arXiv preprint arXiv:2209. 2022.
[40] X. Wang. J. Liu, T. Qiu, C. Mu, C. Chen, and P. Zhou. A real-time collision prediction mecha- nism with deep learning for intelligent transportation system. IEEE transactions on vehicular technology, 6999)999779958, 2020.
[41] M. Treiber, A. Hennecke, and D. Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical review E, 62(2):1805 2000.
[42] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[43] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 1.

---

A Proof of Proposition
Proposition. Suppose that x forces the agent to approach the destination and the episode terminates
when uny traffic collision happens, then we have
min f
[ f Ado) max Y
Σ
P(YEgg
y FpplColl -ruue X).
(A.1)
YE
Proof. According to the definition of return (T) and reward function R = d - ac, we have Edd-
min [(T, f. min IAdvEF
(A.2)
JA
Since æ forces the agent to approach the destination and the episode terminates when any traffic collision happens, is minimized when encountering collisions; otherwise = Σ reaches its upper bound. Considering that the construction of fād is to maneuver the surrounding vehicles when the map is given, it equals that we search the best constraint-satisfying Y in the prior trajectory distribution. Thus, we have
min J(,, f Adv) + max P(Y X) s.t Ego controlled by a collides with Op. (A.3)
I AACUEF
you
We then rewrite Eq. (A.3) in the form of posterior probability distribution maximization as
min EF [zz fādv) max Y
PP-YOP| Coll True,X)
(A.4)
Suppose that Eg generated by the current driving policy be sampled from (x)), Eq. (A.4) is equivalent to marginal maximization over the joint trajectory distribution, which follows as
min fadee
J(, Ado)
+ max y
Σ
TTue,
(A.5)
(x
The proof of Proposition 1 is completed.
B Proof of Proposition 2
Proposition. Suppose that Fgo depends on Y unidirectionally, then we have Es Y (Coll- True, X) x P X)P(E X)PP(Col Truee E Y
(B.1)
Proof. According to Bayes theorem, we have
P(y yop|Call True, X) x P(Coll True/ytig yO X)P((ylis yOpp x)
(B.2)
Since Coll merely depends on bioo and YOp (B.2) is equivalent to F yOp True, X) & P(Coll Yrue|YEFy YOP)P Fgo yop X)
(B.3)
Since we assume that Egg depends on Op unidirectionally; continuing with Bayes theorem, we have
Ego Op Coll True, X)
P(Coll True Fog Y (PP Up X) X)
(B.4)
P(Coll True Fge y P Fgg Fan Op X)P(P X)P(X)
Since the past state X is given, we can omit the last item P(X) in (B.4). Therefore, it holds that B8 OPolColl X) " or XCPl(Coll TruelyEss Op
(B.5)
The proof of Proposition 2 is completed.
12

---

C RL Experimental Settings
We implement CAT' in MetaDrive [20]. MetaDrive simulator provides off-the-self RL. environments for end-to-end driving. We follow the basic setting in MetaDrive'.
In MetaDrive RL environments, the state includes maps sensor readings (Camera or LiDAR), high- level navigation command and self vehicle states. In our experiments, we use 2D LiDAR as the sensor to detect the surrounding vehicles, road boundaries and road lines. The state vector consists of three parts:
Ego State: current states such as the steering, heading, velocity. (ii) Navigation: the nav- igation information that guides the vehicle toward the destination. Concretely, MetaDrive first computes the route from the spawn point to the destination of the ego vehicle.
Navigation: the navigation information that guides the vehicle toward the destination. Con- cretely, MetaDrive first computes the route from the spawn point to the destination of the ego vehicle. Then a of checkpoints are scattered across the whole route with certain intervals. The relative distance and direction to the next checkpoint and the next next checkpoint will be given the navigation information.
Surrounding: the surrounding information is encoded by vector containing the Lidar-like cloud points. We use 72 lasers to scan the neighboring area with radius 50 meters.
The action consists low-level control commands like steering, throttle and brake. MetaDrive receives normalized action as input to control each target vehicle: a = [01, 0g]" E [-,, At each environmental time step, MetaDrive converts the normalized action into the steering u, (degree), acceleration Ua (hp) and brake signal us (hp) in the following ways: (i) U Smaza1, (ii) ua - Beez min(O, 02), wherein Smaz (degree) is the maximal steering angle, Fmaz (hp) is the maximal engine force, and Bmaz (hp) is the maximal brake force.
Frar max (0, (2) (iii)
MetaDrive uses compositional reward function as R Rariving Rerash vehicle penalty + Rout.of of roodd. Here, the driving reward Rdriving de d:-1, wherein the de and di denote the longitudinal coordinates of the target vehicle in the current lane of two consecutive time steps, providing dense reward to encourage agent to move forward. By default, the penalty is -1 if the agent collides with surrounding vehicles, and the penalty is -10 if the agent runs out of the road.
D Hyper-parameter Settings
Table 3: CAT
Table 4: TD3
Table 5: DenseTTNT and M2I
Hyper-parameter
Value
Hyper-parameter
Value
-lpeer-paameter
Value
Scenario Horizon T
9s
Discounted Factor ?
0.99
Train Batch size
256
History Horizon L
1s
Train Batch Size
256
Train Epoches
30
of OV candidates M
32
Critic Learning Rate
3E-4
Sub Graph Depth
3
# of EV candidates N
5
Actor Learning Rate
3E-4
Global Graph Depth
Penalty Factor a
0.99
Policy Delay
2
NMS Threshold
7.2
Policy Training Steps
10E6
Target Network
0.005
Number of Mode
32
13

---

E Qualitative Results of Safety-critical Traffic Generation
Raw
CAT
Raw
CAT
Figure 6: More comparison between the original scenarios raw datasets and the safety-critical scenarios generated by CAT. The red car is the ego vehicle and the blue car is the opponent vehicle.
14

---

82
Raw
M21
STRIVE
CAT
Figure 7: Comparing the different scenario generation methods. The red car is the ego vehicle and the blue is the opponent vehicle.
F Details of the Rule-based Adversarial Traffic Generation
Way-points
Rule-besed Generation
Inttial Condition
Safety-Criticl Traffic
Figure 8: An example of the rule-based adversarial traffic generation.
15

---

Considering the HD-map in Waymo datasets are highly unstructured, thus we design uule-based system as follows:
1. We heuristically take the vehicle labeled as 'Object of Interest' as the adversary.
2. We take some waypoints on the navigation path of the ego vehicle, which will be occupied by the adversary later to minimize the ego vehicle's drivable area.
3. We mix above waypoints with those on the original path of the adversarial vehicle.
4. We fit Bezier curve based all the way-points to derive smooth and feasible path of the rival vehicle.
G Qualitative Results of Safety Improvement after CAT
Case
Case 2
Before CAT
After CAT
Before CAT
After CAT
Case 3
Case
Before CAT
After CAT
Before CAT
After CAT
Figure 9: Driving behaviour before and after CAT. The red car is the ego vehicle and the blue is the opponent vehicle. In case 1, the opponent car makes an unprotected left turn at an intersection; the driving agent learns stay away from potentially dangerous vehicles. In case 2, the leading slows down; the driving agent learns to change its lane and overtake. In case 3, the opponent car cuts into the lane suddenly; the driving agent learns yield and change its lane ahead of time. In case 4, two vehicles traveling in opposite directions meet and the driving agents learns to pass by.
H Further Discussion
Following limitations wait to be addressed in future work: (i) we only consider ad- versarial vehicles in this work but the safety-critical behaviors of pedestrians and cyclists are also of importance for safe driving and yet to be done, it requires the access to different motion fore- casting model; (ii) Experiment on five hundred scenes cannot cover all the accident-prone situa- tions, thus there other possible failure modes in the resulting agent; (iii) only investigate the RL-based driving policy but the adversarial scenarios should also benefit the human-in-thee imitation learning [17, 43).
Limitations:
Transferring to real-world driving: The proposed adversarial training method and the comparison with prior methods are evaluated in the simulation of one hundred complex traffic scenarios imported from real-world driving dataset [4]. Thus, the evaluation contains realistic and complex vehicle interactions and shows promise for transferring real-world settings.
16


Reviews:
[{"review_id": "nyE0643-sQO", "reviewer": "Reviewer_9tii", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "paper presents \"CAT: Closed-loop Adversarial Training for\nSafe End-to-End Driving\", where the goal is to augment training set of replay scenarios with automatically generated safety-critical scenarios in closed-loop simulation to train safe driving policy", "summary_of_recommendation": "I recommend to re-enforce the experimental section with fair train/test/val split and introduction of the rule-based baseline with traffic-priors to confirm the claim of the paper at the rebuttal stage, so that the contribution of the authors could be fairly judged", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "- factorized safety critical resampling is based on hypothesis that safety critical scenario can be identified as  a re-combination of possible, independent predictions (multiple-futures) where some of the re-combinations can be more safety critical than others,  - this doesn't consider scene-wise multimodal prediction, where most probable modalities have to match in realistic cases only (which is done via learning), thus the proposed generated safety-critical scenario can be mostly un-realistic therefore it will bias whole distribution of realistic scenarios in a closed-loop simulation framework  \n\n- closed-loop adversarial training, doesn't have a mechanism to ensure that generated safety-critical scenarios are all meaningful (no distribution shift to unrealistic but safety-critical scenarios learning) and the experimental setup doesn't specify enough details to ensure that CAT produces generalized driving policy , which needs to be validated by ensuring that there is no overlap between training and test data (different locations/maps/traffic conditions/time, etc...)\n\n- not clear, which is of the proposed contributions has biggest impact on the ability of the CAT to generate safe driving policy, need to decompose the key contribution factors and how they improve the metrics/scores\n\n- need to introduce baseline rule-based system to generate safety-critical scenarios, especially since HD-map information is used as a traffic prior, it should generate complete set of safety-critical scenarios given human replay as an input then compare the experiments with the baseline", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- quality: overall paper is well written\n- clarity: ideas expressed clearly\n- originality and significance: I believe that even thought the idea might have originality, the experimental details can't fully confirm that it could actually works on practice for large scale training of autonomous driving, due to the following missing points:\n\nis generated driving policy realistic? - factorized safety critical resampling is based on hypothesis that safety critical scenario can be identified as  a re-combination of possible, independent predictions (multiple-futures) where some of the re-combinations can be more safety critical than others,  - this doesn't consider scene-wise multimodal prediction, where most probable modalities have to match in realistic cases only (which is done via learning), thus the proposed generated safety-critical scenario can be mostly un-realistic therefore it will bias whole distribution of realistic scenarios in a closed-loop simulation framework  \n\ndoes experiment have fair split between train/val/test? - closed-loop adversarial training, doesn't have a mechanism to ensure that generated safety-critical scenarios are all meaningful (no distribution shift to unrealistic but safety-critical scenarios learning) and the experimental setup doesn't specify enough details to ensure that CAT produces generalized driving policy , which needs to be validated by ensuring that there is no overlap between training and test data (different locations/maps/traffic conditions/time, etc...)", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "fu3XUu5_8J", "reviewer": "Reviewer_s5WY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper provides a framework to train RL agents on accident-prone scenarios generated from real datasets and improve the agent's behavior in critical situations. Accident-prone scenarios are rare in real-world data sets. Therefore, CAT samples scenarios and adapts them to provoke accidents. Multiple trajectories of opponent vehicles are generated based on a real-world scenario. Based on these trajectories, the agent's reaction is simulated. In order to improve the agent's policy, the trajectory that is most likely to lead to a collision is chosen to create a scenario that is as critical as possible. The paper claims that the framework is agnostic to the learning method but only demonstrates this for one RL algorithm (TD3).\nExperimental results comparing CAT to state-of-the-art methods show that an agent trained with adversarial scenarios performs better in real-world and accident-prone scenarios. Furthermore, scenario generation is less time-consuming than the SOTA method.", "summary_of_recommendation": "Overall this paper is a relevant contribution toward generating adversarial driving scenarios for autonomous driving. The structure is clear, and it is overall well-written. \n\nHowever, there are some weaknesses regarding key aspects that are insufficiently addressed. My main points are:\n1. I recommend clearly stating and specifying what the paper means by safe end-to-end driving and which setting is assumed. \n2. The problem formulation is unclear, especially regarding the definition of the state and action spaces. \n3. Even though the reinforcement learning algorithm is not a key contribution of this paper, the results are not reproducible without this information. Furthermore, the factorized safety-critical resampling technique, which is a contribution, builds on this formulation and cannot be validated without this formulation. \n\n=========================================\nUpdate after rebuttal\n\nI changed my rating to \"Weak Accept\"  (see \"Questions for rebuttal\" for details). \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "### Suggestions for clarification during rebuttal\nI suggest to reiterate the following points: \n1. The paper should not claim the agnostic setting as a contribution\n2. The paper should explain what is understood as safe end-to-end driving\n3. The state and action spaces should be formulated clearly, and the used reward function should be explained more precisely ($\\alpha$  and destination). \n4. The paper should provide a more detailed proof for equation (2) or weaken the equivalence assumption.\n\n\n### Questions for rebuttal\n1. How would imitation learning work in this setting, and how can IL react to adversarial attacks?\n2. Which setting of end-to-end driving is used? Which (sensor) data is used as an input? \n3. Which state formulation was used? What is the action space?\n4. What is the destination for the reward formulation? Can the reward function be explained more precisely? How can the parameter $\\alpha$ be tuned?\n5. Is it possible to prove that the two optimization problems in equation (2) are equivalent? Which assumptions are required for this?\n\n\n===============================================================\n### Response and update in response to rebuttal\n\nI thank the authors for answering most of my questions and considering the feedback in the revised manuscript. I lifted my rating to \"Weak Accept\" as the criticism regarding the paper's clarity and technical quality were revised. I appreciate that the authors added the state formulation, clarified the safe end-to-end driving setting, removed the overstatement of the first contribution, and added further assumptions to Equation (2). ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## General strengths and weaknesses\n### Strengths\n\nS1. Relevant topic (adversarial training for autonomous driving). The paper presents a useful approach to generate adversarial test scenarios from real data, as these scenarios are rare in real-world data sets. \n\nS2. Video and plots help the reader to understand the framework and illustrate the results well\n\n### Weaknesses\nW1. The paper provides a framework for safe end-to-end driving, but does not explain or specify what safe end-to-end driving is\n\nW2. Lack of a clear formulation of the control problem. In particular, the state and action spaces are unclear. As this is not a general reinforcement learning setting, but an explicit driving use case, these formulations are crucial to understanding the results (Factorized Safety-Critical Resampling) \n\nW3. In the introduction, the paper claims that the setting is planner-agnostic. However, it just demonstrates how the framework can be used for RL. It is not specified how the framework would work for other policies. The Factorized Safety-Critical resampling builds on RL formulation and is not agnostic. Actually, the paper states this as a potential of future work (limitations section), so the setting is not agnostic and, hence, this is not a contribution of this work.  \n\n\n## Specific evaluation of requested categories\n('S' - strength, 'W' - weakness)\n\n### Clarity: \n- S: The claimed contributions are stated clearly\n- S: Overall, well organized\n- W: The reinforcement learning problem formulation is unclear, especially the state formulation is unclear (see comments)\n- W: It is not clear what is understood as safe end-to-end driving and what setting the paper uses \n\n### Originality: \n- The paper combines multiple existing components (MetaDrive, DenseNet, M2i probabilistic formulation) to create a more efficient framework for the generation of adversarial scenarios \n- Related work is adequately cited\n- It is clear how this contribution improves upon the related work\n\n### Quality: \n- W: Contribution 1) is not supported by any evidence\n- W: Equation (2): it is questionable if it holds without further assumptions\n\n### Significance: \n- especially in the context of RL for autonomous driving, a more efficient framework to generate critical scenarios is relevant as these are rare in real-world data sets\n\n\n## Comments: \nThe above evaluation is based on the following comments: \n\n1) Abstract, line 13 \"on hundreds of training scenarios\" As far as I understood, the paper just uses 100 scenarios from the Waymo Open data set\n2)  l. 22 Safe End-to-end driving is not further introduced\n3) l. 50 How would imitation learning work in this setting, and how can IL react to adversarial attacks? In the limitations section the paper states:  human imitation learning in the loop. There is a difference between this and the general concept of imitation learning.\n4) Comment on Problem Formulation/ Experimental results\n   * It is not clear what the state and action spaces are. \n   * In an end-to-end driving algorithm, the state should be raw sensor data (simulation from camera/lidar in Meta Drive (?) and the current position/ego motion data), and the action space should be steering angle and accelerator/ brake pedal position. In the paper, this is not clear. I assume the paper uses a similar setting as [19], Section 5.1. The state formulation should be included in the problem formulation. \n5) Problem formulation: What is the destination?\n6) Reward formulation: $R=d-\\alpha c$ ,  $d$ should have a negative sign as a smaller distance to the destination should give more reward. Furthermore, the value range of $\\alpha$ is not defined. \n7) $\\alpha$ is used twice (?) with different explanations: Problem formulation: \"a hyper-parameter for the reward shaping\" Implementation details: \"a heuristic decay factor to reflect the uncertainty of traffic models with the increasing prediction horizon.\" It is unclear whether this is the same  or different parameters.\n8) Equation (2) It is questionable if this holds, especially without knowing the state formulation. Given the setting, it is not obvious that this equivalence is always true\n9) Evaluation of Closed-Loop Adveserial Training (CAT) (A) No Adv: remove the opponent vehicles: Where do the 7546 Train Attack Num come from if the opponent vehicle is removed?\n10) limitations: the paper provides a framework to train driving agents but does not provide any safety guarantees, as it is agnostic to the planner. Work on the algorithmic side is needed to achieve safe driving and provide guarantees. \n11) Publishing code would make this framework usable for others more easily\n\n### Minor Comments / Typos\n1) l.123 $\\uparrow \\downarrow$   introducing this notation would make it clearer \n2) l. 125 $\\sum R$ instead of  $\\Sigma R$ \n3) l.138 $f^{Adv}$\n4) l.119/ l. 149  $J(\\pi, f^{Adv})$ instead of $J(\\pi)$ \n5) Algorithm 1: BBox should be explained in a comment such that the pseudo-code can be understood by itself. \n6) Table 2: \"Replay\" instead of \"Heuristic\" would be more consistent", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "XsFjMxBj6U", "reviewer": "Reviewer_hXmh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a reinforcement learning agent, that can generate adversarial scenarios during training. This is done by decomposition the scenario generation process in three factors. First the traffic prior which is simply based on a generic motion prediction model. Second the ego vehicle density, which is based on the currently learned policy. And third some safety critical index, for instance linked to geometrical proximity of the agents in the scene. The proposed framework allows to generate scenarios for simulation-based learning that are adequate with respect to computational requirement of reinforcement learning. The authors show in their experiments that the resulting policies are more robust than regular policies.", "summary_of_recommendation": "- Put more emphasis on robustness at convergence", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Could we get the some robustness evaluation of the best policy?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "= Contribution\n\n The paper tackles a research topic which is linked to distributional shift in E2E training and deployment. This is maybe the main challenge in E2E development now. How do we tackle distribution shift to long tail event that typically are rare during training. This contribution is quite straight forward in the way that it provides a simple decomposition mechanism for generate long tail scenarios. The fact that it can generate these scenarios at a rate that is adequate for training makes it quite appealing.\n\n= Research Idea\n\n== Quality\n\nThe paper is very clear and well written. The idea is well explained and well argued. Limitation of the paper are quite well justified. The paper could have had more in depth analysis with more scenarios and checking of the policy robustness after training.\n\n== clarity\n\nThe paper is clear.\n\n== originality \n\nThe paper is rather original. The contribution is well put together.\n\n== significance\n\nThis work is significant and proposes a solid starting point for performing adversarial scenario sampling during trail and error learning in simulation.\nMore could be done in terms of convergence to an optimal policy. The number of crashes in these context continues to grow through training. One would except that with more rollout the agent learns to avoid all corner cases.\n\nStrengths\n-\tGood results\n-\tSome baseline comparison\n\nWeakness\n-\tCould have more indepth statistical analysis\n-\tMore scenarios\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "SJ_SO4xTfvW", "reviewer": "Reviewer_P4zB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposed a novel iterative learning framework called Closed-loop Adversarial Training (CAT) to dynamically generate adversarial safety-critical scenarios for training a driving policy. The proposed method factorizes the safety-critical scenario generation objective into 3 components: 1) traffic prior 2) ego estimation and 3) collision likelihood. The method resamples the trajectories of the surrounding objects based on the whether they collide with the ego vehicle, while training an ego driving policy in an iterative manner. The paper provides both qualitative and quantitative experimental results to show superior performance of the proposed method compared to a few baseline methods on a driving set of 100 from WOMD.", "summary_of_recommendation": "The paper proposed a novel iterative learning framework called Closed-loop Adversarial Training (CAT) to dynamically generate adversarial safety-critical scenarios for training a driving policy. The method resamples the trajectories of the surrounding objects based on whether they collide with the ego vehicle, while training an ego driving policy in an iterative manner. The paper provides both qualitative and quantitative experimental results to show superior performance of the proposed method compared to a few baseline methods on a driving set of 100 from WOMD.\n\nThe paper is well written and easy to follow. The results are interesting and relevant to the field. Experimental setting is reasonable and clearly demonstrates the major claims of the paper. I recommend an accept condition on addressing the few minor concerns raised above.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- The ii) of the main contributions in the Intro says the method \"balances attack success rate and computation cost by resampling ... \". However there is no experiment that demonstrates this point. An ablation study on how the method trades off success rate and compute cost would be helpful to support this claim.\n- The assumption of ego vehicle is unidirectionally conditioned on other vehicles might be too strong. Have the authors thought of relaxing this assumption (e.g. by rolling out the ego policy and other vehicle policies jointly in an autoregressive manner)? It might require reformulating the factorization. \n- Transferability to real-world autonomous driving systems: RL agents might overfit to the specific training environment, in this case the adversarial sim agents. The resultant policy could potentially be overly cautious and not make enough progress. It would be helpful to also plot the progress in Fig 5.\n\nNice to have:\n- The dataset is very small (100 segments). (The authors acknowledge it in the limitation paragraph). It would be nice to demonstrate similar results using the entire WOMD dataset.\n- In Fig 5, the learned policy is only evaluated in the MetaDrive simulator against log playback traffic with ~10% crash rate. This % is very high even given the small training dataset. Curious if using a larger training set would further bring down the crash rate.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Quality\n- The paper is technically sound. The claims are well supported by experimental results. The proposed method is practical and reasonably easy to implement.\n\n#### Clarity\n- The paper is clearly written and well organized. \n\n#### Originality\n- The problem is not new; the method is novel. The proposed method is clearly different from previous methods.\n\n#### Significance\n- The results are interesting, and support the main claims. Others are likely to use the ideas. Not significantly better than previous methods. \n\n#### Strengths\n- The paper is clearly written and easy to follow.\n- The proposed method is practical and effective.\n- Empirical results support the main claims\n\n\n#### Weaknesses\n- The ii) of the main contributions in the Intro says the method \"balances attack success rate and computation cost by resampling ... \". However there is no experiment that demonstrate this point. An ablation study on how the method trades off success rate and compute cost would be helpful to support this claim.\n- The dataset is very small (100 segments). (The author mentioned it in the limitation paragraph).\n- In Fig 5. the learned policy is only evaluated in the MetaDrive simulator against log playback traffic with ~10% crash rate. This % is very high even given the small training dataset. \n- Transferability to real-world autonomous driving systems: RL agent might overfit to the specific training environment, in this case the adversarial sim agents. The resultant policy could potentially be overly cautious and not make enough progress. It would be helpful to also plot the progress in Fig 5. \n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an interest adversarial training scheme for autonomous vehicles, which takes in real-world safe driving trajectories (states, no sensor observations) and modifies them locally to create adversarial behaviors, based on a collision-maximizing objective. The resulting optimized adversarial behaviors are set as the open loop behavior of opposing (non-ego) agents, aiming to stress-test the current ego-vehicle's policy that is being trained. The paper shows that this adversarial training scheme results in slightly more robust ego-policies than training based on logged safe trajectories for opposing agents. The paper also shows that this scheme is computationally much more efficient than other similar techniques in this area, such as STRIVE [13].

Overall, reviewers found the idea interesting, but were concerned about the lack of realism of the attacks, no hardware experiments, as well as the fact that although the attacks are computationally efficient, they are not much more effective at producing adversarial traffic scenarios than [13]. If [13] had used a generative model with cheaper inference than an autoregressive model, reviewers were concerned that the impact of the CAT paper would be relatively small.         

In any case, I think the idea is interesting enough to discuss at the conference. I am recommending acceptance as a poster.

[13]  D. Rempe, J. Philion, L. J. Guibas, S. Fidler, and O. Litany. Generating useful accident-prone driving scenarios via a learned traffic prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17305–17315, 2022.
Decision: Accept (Poster)