Breaking through safety performance stagnation in autonomous vehicles with dense learning

Domain: chemistry
Published: 2026-02-25
Editorial outcome: major_revision

Abstract
Autonomous vehicles remain commercially limited largely due to safety performance stagnation. Existing deep learning, heavily reliant on failure data from rare safety-critical events, suffers from the seesaw effect—improvement in some scenarios causes regression in others. We introduce an innovative dense learning approach that prioritizes both informative failures and successes, informed by theoretical findings. Data is sampled proportionally to its contribution to the policy gradient and exposure frequency, excluding non-informative samples. This densifies the training dataset’s information, significantly reducing learning variance without bias, enabling tasks intractable for existing methods. To validate this, we trained a safety-critical driving agent for a highly automated vehicle using mixed reality on an urban test track. Results demonstrate that our approach breaks the performance stagnation, enhancing the model’s overall safety performance by one to two orders of magnitude. This marks a significant stride towards achieving human-level safety and widespread adoption for autonomous vehicles.

Introduction
The dream of autonomous vehicles (AVs) has been around for approximately a century1. Enormous amounts of capital, exceeding $160 billion, have been invested in this vision over the past twenty years2. Despite witnessing significant advancements in AV technology, no commercially available AVs have yet met the SAE Level 4 standard3. The pace of progress in AV development has been disappointingly slow, particularly when it comes to ensuring their safety, which has remained stagnant in recent times. The situation was further exacerbated by the unfortunate accident involving Cruise Automation in San Francisco on October 2nd, 20234. This safety gap presents a significant obstacle to the deployment and commercialization of AVs, as they struggle to effectively handle a wide array of infrequent yet critical safety events, commonly referred to as the long-tail challenge for AV safety5,6. Existing approaches have proven inadequate in overcoming this challenge, resulting in a noticeable slowdown and even stagnation in the enhancement of AV safety performance. Consequently, the development and deployment of AVs have been severely hindered, calling for an urgent breakthrough.

We formulate the safety challenge for AVs as the Curse of Rarity (CoR)7, which arises due to the rarity of safety-critical events in high-dimensional variable spaces. We recognize that this is a compounding effect resulting from the rarity of events combined with the high dimensionality of related variables. As the utilization of deep learning techniques is typically necessary to address the high dimensionality, the rarity of events dramatically increases the estimation variance of policy gradient, thereby impeding the ability of deep-learning models to learn. This challenge exists in different AV safety-related tasks, and we attacked it in the AV safety testing task in our prior work8. However, the main objective of AV testing is to evaluate the safety performance by estimating the probabilities of rare events (e.g. car collisions) given a specific AV policy, whereas the AV training task is to improve the safety performance by searching for a policy from the AV policy space that can minimize these probabilities. This fundamental distinction makes AV training considerably more challenging than AV testing. Furthermore, during the AV training process, better AV safety performance also means fewer safety-critical events, which, in turn, makes it more difficult to improve the safety performance. These issues pose significant obstacles to the development of AVs to ensure safety performance. It should be emphasized that previous advancements in AI have primarily been focused on non-safety-critical applications such as chatbots and games9–12, where a certain level of failure is deemed acceptable. However, when it comes to safety-critical autonomous systems13 such as AVs, an extremely high level of safety performance is demanded, resulting in the CoR challenge.

Existing approaches that attempted to tackle the CoR challenge primarily focus on learning from the data where AI systems exhibit failures14. For instance, Tesla had reported training their systems on datasets that represent scenarios where AVs struggle or deviate from human driver behavior15. However, it is important to note that these approaches lack a solid theoretical foundation, and our investigations have revealed that they can exhibit significant biases and even be misleading. Consequently, while these approaches may enhance the safety performance of AVs in specific scenarios, they run the risk of compromising performance in other safety-critical situations (Fig. 1a). This phenomenon, known as the seesaw effect in different domains16–18, hinders the improvement of overall safety performance and leads to performance stagnation. That is also a key reason why safety-critical driving situations are seemingly endless, despite years of development in AVs. Alternatively, some researchers have attempted to prevent unsafe behaviors of AVs based on rules or models such as formal methods19–21 and constrained learning22–24. However, this approach faces challenges to handle the variability and complexity in high-dimensional variable spaces, because it is difficult to rely on a predefined set of parameters and assumptions for a wide spectrum of diverse driving situations22.

We address the CoR challenge by developing an innovative dense reinforcement learning approach to overcome performance stagnation and enable AVs to continually improve their safety performance beyond the current state-of-the-art. The key idea is to remove the non-informative data to densify the information from different perspectives including the episodic data densification, state-level densification, and retrospective data densification (see Fig. 1b). For the episodic data densification, we found out that the optimal episodic data distribution for training neural networks to overcome CoR should be based on their contribution to the policy gradient and their exposure frequency, as indicated in Theorem 1 in Methods. Specifically for the AV safe driving task, the training dataset should include data from both avoidable crash episodes and episodes where crashes were successfully avoided, i.e., near-misses. Moreover, as each driving episode can span many time steps, we retained and reconnected the safety-critical states, which can further densify training data within each driving episode from the state-level, as indicated in Theorem 2 in Methods. Lastly, due to the changing AV policy during training, achieving both higher precision and recall rates for identifying informative episodes and states has become difficult. To address this, we designed a learned safety metric that serves as a real-time predictive evaluation tool. It also includes a retrospective evaluation component to re-evaluate all data for the new AV policy through counterfactual simulation.

To demonstrate the efficacy of our methodology, we trained a safety-critical driving agent (referred to as SafeDriver in this study) for a SAE Level 4 (ref. 3) AV with Autoware25, which is widely recognized as the leading open-source automated driving system. As a safety filter system26, SafeDriver overrides Autoware only in safety-critical situations to enhance the overall safety performance of the AV, as shown in Fig. 1c. As SafeDriver is decoupled from Autoware, it is applicable for other AV models including those developed based on large neural network models. We first tested the performance of SafeDriver in multiple simulated naturalistic driving environments (NDEs), then equipped a Lincoln MKZ hybrid with Autoware, and tested the SafeDriver in a physical urban test track with mixed reality. The results obtained from both simulation and field-testing clearly demonstrate the effectiveness of our approach in training a safety-critical driving agent. By employing our methodology, the overall safety performance of the AV models can be significantly enhanced, with improvements ranging from one to two orders of magnitude.

Results
Dense learning approach for AV training
We demonstrated the seesaw effect for AVs (see Supplementary Section 2.3 in Supplementary Information) and then found out that it is essentially caused by the biasedness of learning and the severe variance caused by the CoR (see Eq. (8) in Methods). Here the biasedness indicates that the expectation of the gradient estimation is different from the ground truth, so the learning process could be misled. To address this issue, we develop the dense reinforcement learning approach for AV training. The key is to remove the non-informative data to densify the information from different perspectives including the episodic data densification, state-level densification, and retrospective data densification (see Fig. 1b). For the episodic data densification, we first obtain the optimal training data distribution for AV training based on importance sampling theory27 as

q\pi(X) / ||\nabla \pi P(\omega(X)=1)||^2 \times P\pi(X), (1)

where X denotes each driving episode of the training data, \pi denotes the AV policy, \omega(X)=1 denotes the objective event that is a rare event (for example, crash events), \nabla \pi P(\omega(X)=1) is the policy gradient for the probability of the objective event \omega(X)=1 given X, ||\cdot|| denotes the l2 norm of the vector, q\pi(X) denotes the optimal probability distribution of X for training the policy \pi, and the symbol / means ‘proportional to’. Here an episode is a segment of recorded driving data with a predetermined time duration or distance. It indicates that the optimal training data should contain both avoidable crash events (informative failures) and near-miss events (informative successes) where ||\nabla \pi P(\omega(X)=1)||^2 is non-zero, while the data of those unavoidable crash events and safe situations should be removed.

The primary obstacle in applying Eq. (1) is that ||\nabla \pi P(\omega(X)=1)||^2 for each driving episode of data cannot be calculated practically, exacerbated by the changing AV policy during the training process. We note that the key to dense learning is to remove the non-informative data and keep the informative one, so Eq. (1) can be utilized as a guidance for data densification. Therefore, rather than attempting to compute the precise values of ||\nabla \pi P(\omega(X)=1)||^2, we adopt an approximation through a binary classification task. In this task, ||\nabla \pi P(\omega(X)=1)||^2 is approximated as one for informative data, that is, avoidable crash events and near-miss events. Specifically, we identified a crash event as avoidable if an evasive trajectory is still feasible after the vehicle state is identified as safety-critical and a non-crash event as a near-miss if the minimum relative distance between the AV and background vehicles is below a pre-determined threshold. Although these criteria are not unique, we chose these simple yet effective ones to demonstrate the effectiveness of our approach. By sampling the episodes according to P\pi(X) and rejecting these where ||\nabla \pi P(\omega(X)=1)||^2 is estimated as zero, our approach realizes the training data distribution as in Eq. (1), dramatically reducing variance for rare event learning without compromising unbiasedness, as indicated in Theorem 1 in Methods.

We then conduct the state-level densification and retrospective data densification. As each driving episode may last for many time steps, we retained and reconnected the informative states to densify the data from the state-level. This is challenging in AV training, as different AV policies during the training process could have different safety-critical states and we need to identify them with both high precision and recall rates. To address this challenge, we design a learned safety metric, which obtains better performances than existing approaches (see Supplementary Fig. 1 and Supplementary Video 2). Moreover, as AV policy is changed after training, a large policy gap may exist between the new AV policy and the policy that collected the data. To bridge this gap, we introduce a retrospective evaluation component to re-identify the informative episodes and states for the new AV policy through counterfactual simulation (see Supplementary Sections 2.1.9 and 2.11 in Supplementary Information). Furthermore, to improve the efficiency of data collection, we also utilize the intelligent testing environment (ITE) developed in our previous study8. Utilizing ITE allows us to accelerate the collection of informative training data by multiple orders of magnitude (see Supplementary Fig. 2, Supplementary Video 3, and Supplementary Section 2.7 in Supplementary Information). Figure 2a shows the results of a multi-lane highway environment.

Simulation results
We evaluated the effectiveness of our approach by systematic simulation analysis. To measure the overall safety performances of AVs for quantitative comparisons, we utilized the crash rate per test of AVs in NDE. As NDE is generated based on naturalistic driving data, testing results in NDE can represent the safety performance of AVs in the real world28. Specifically, we selected different types of base AV models, and trained SafeDriver that only takes over from the base AV models in safety-critical states (identified by the learned safety metric), resulting in the integrated AV models. We then compared the safety performances of base AV models and integrated AV models in NDE. Although we refer to the baseline models as ‘base’ AV models, they could be advanced commercial AV models with large-scale neural networks together with their own safety guards. To demonstrate the generalizability and applicability of our approach, we conducted experiments for four types of base AV models in different driving environments including highway, roundabout, and urban environments (Fig. 2). In this study, we only utilized a simple Multilayer Perceptron as backbone to demonstrate the effectiveness of our approach, so the performances could be further improved with more advanced backbones.

We trained a RL-based AV model using the proximal policy optimization (PPO) algorithm28,29 (see Supplementary Section 6.7 in Supplementary Materials) and obtained its crash rate as 1.36 \times 10^-2 in NDE. We further implemented the responsibility-sensitive safety (RSS) model30 as the default safety guard. Through millions of tests of the base AV model with RSS in NDE, we obtained the crash rate as 2.71 \times 10^-5 crash per test. Then, we trained SafeDriver using our approach without RSS (Fig. 2a, red line). The results revealed a crash rate of 3.71 \times 10^-6, making an 86.3% reduction in comparison to the base AV model. When considering only avoidable crashes, our approach demonstrated a remarkable 91.7% reduction. This suggests that our approach significantly enhances the overall safety performance by approximately one order of magnitude. We also evaluated the AV model with SafeDriver and RSS, yielding a crash rate of 7.79 \times 10^-6, representing a 71.3% reduction compared to the base AV model with RSS (Fig. 2a, purple line). While not as optimal as the model without RSS, due to the additional constraints introduced by RSS, our approach still improves overall safety performance. This is significant considering that many AV models may already have their safety guards or constraints in place. To demonstrate the effectiveness of our approach, we also compared our approach with the provably safe RL approach31,32 (see Supplementary Section 2.2.5 in Supplementary Information).

Navigating through roundabouts poses a significant challenge for AVs due to the intricate interactions among AVs and the diverse mix of surrounding vehicles. To evaluate the effectiveness of our approach in roundabout scenarios, we expanded our simulation experiments to a real-world four-armed roundabout33 in Germany, known for its high traffic volume and complex intersections. We generated the realistic driving behaviors of background vehicles in NDE with a Transformer-based learning approach28, leveraging the naturalistic driving data in the RounD dataset33 (see Supplementary Section 2.6 in Supplementary Information). Using an advanced AV model with a sampling trajectory generational algorithm34 as our base AV model, we demonstrated that our approach significantly reduces the AV’s crash rate by 74.5%, along with an 89.5% reduction in avoidable crash rates (Fig. 2b).

We further test the performance of SafeDriver when AVs navigate continuously through urban environments. We utilized the digital twin of the Mcity test track, and selected two AV models as base models, one is the default AV model in SUMO35 (the intelligent driving model (IDM)36 and the SL2015 model35 in SUMO) and another one is Autoware25, which is widely recognized as the leading open-source automated driving system. Results show that our approach can reduce the overall crash rate of the SUMO AV model by 98.0% and reduce the avoidable crash rate by 98.9% (Fig. 2c), representing a nearly two-orders-of-magnitude improvement. We integrated the identical SafeDriver into Autoware without any additional refinement, and results indicate a reduction in the crash rate from 1.07 \times 10^-6 to 1.07 \times 10^-7, equating to a 90.0% enhancement in safety performance. This underscores the efficacy and adaptability of our approach across various AV models.

To further investigate the contributions of our developed techniques, we conducted ablation studies on the multi-lane highway environment including 1) no episodic data densification, 2) no state-level data densification, 3) no near-miss episodes, 4) no retrospective data densification, 5) no near-miss and retrospective data densification, 6) no trajectory resampling, and 7) no state reconnection. With the same training steps such as 780 and 1650 steps, we compared the crash rates of SafeDriver in NDE as shown in Table 1. Results demonstrate the effectiveness of all developed techniques. Specifically, both the episodic data densification and state-level data densification contributed dramatically, while the retrospective data densification together with near-miss scenarios affects the performance significantly. Please see details in Supplementary Section 2.4 in Supplementary Information.

To test the scalability and generalizability of our approach, we implemented a unified SafeDriver on the nuPlan benchmark37, recognized as the world’s first extensive planning benchmark for autonomous vehicles, which contains 1200 h of human driving data from 4 cities across the US and Asia with widely varying traffic patterns (Fig. 3a). The goal of SafeDriver is to utilize a learning-based planner to assist the base model in navigating through safety-critical scenarios encountered in urban driving. We selected the state-of-the-art (SOTA) planner PDM-Hybrid38 as our base model to demonstrate the effectiveness of SafeDriver over the SOTA base model. As shown in Fig. 3b, if the PDM-Hybrid model predicts a collision within 2 s, SafeDriver takes control; otherwise, the base model manages the ego vehicle. Results indicate that SafeDriver reduces the total number of collisions by 21.7% and decreases AV-responsible crashes by 29.2%, compared with the base model. Illustration of two examples can be found in Fig. 3c–f. Please see details in Supplementary Section 2.5 in Supplementary Information.

Field testing results
To demonstrate the effectiveness of our approach for real-world AVs, we outfitted a Lincoln MKZ hybrid with Autoware as the base AV model (Fig. 4a) and conducted the experiments in the physical test tracks at Mcity (Fig. 4b) with a mixed-reality testing platform (Fig. 4c). One additional challenge for real AV training is caused by the so-called simulation-to-reality gap, where AV models developed in simulations may not seamlessly translate to real-world performance. In this work, we bridged the gap by iteratively improving the simulation models. Specifically, we utilized the same Autoware system in simulation as employed in the actual AV, mimicked the delay and latency characteristics of the real AV, and compensated for acceleration and deceleration effects induced by road slopes (Supplementary Fig. 4). Results demonstrated that, with this approach, the SafeDriver effectively reduces the crash rate of the real AV in the test track from 1.44 \times 10^-6 to 1.42 \times 10^-7, making an impressive 90.1% improvement in the AV’s safety performance (see Fig. 4d, e and Supplementary Fig. 5). Recognizing that some crashes are unavoidable due to aggressive behaviors of background vehicles, we further assessed the crash rate of avoidable crashes, revealing a 98.8% reduction through our approach. For a more in-depth understanding, additional case studies are presented in Fig. 4f.

Methods
Formulation of AV training problem
This section describes the formulation of the AV training problem. Denote the variables of the driving environment as X=[S(0), A(0), S(1), A(1), ..., S(T)] \in \Omega, where S(k) denotes the states (position, speed, heading, etc.) of the AV and surrounding background vehicles or other road users at the k th time step, A(k) denotes the maneuvers of surrounding background vehicles or other road users at the k th time step, T denotes the total time steps of each driving episode, and \Omega denotes the space of the variables X. The goal of AV training is to optimize the AV policy \pi \in \Pi as

\pi \in \text{argmax} E P\pi [f(\pi, X)], (2)

where f(\cdot) denotes the objective function of AV training and X follows an underlying joint distribution P\pi(X) in NDE. To keep the notation simple, we leave it implicit in all cases that \pi is a function of neural network parameters \theta. For safety training, the objective could be minimizing the overall crash rate as

\min \pi \in \Pi E P\pi [P(\omega(X)=1)], (3)

where \omega(X)=1 denotes the objective event (e.g., vehicle crash), and P(\omega(X)=1) denotes the event probability of the AV policy \pi in the driving environment X.

Curse of rarity for AV training
To solve the AV training problem, deep learning approaches have been widely applied to handle the high variability and complexity of X. The key is to estimate the policy gradient at each training step for the current policy \pi as

\Psi d = E P\pi [\nabla \pi P(\omega(X)=1)], (4)

where \Psi \in R^d is the dimension of the gradient, and then the policy could be updated accordingly. For non-trivial AV policies, however, since the objective event \omega(X)=1 is a rare event, namely, E P\pi [P(\omega(X)=1)] is a near-zero value, most P(\omega(X)=1) and \nabla \pi P(\omega(X)=1) are near zero. Therefore, estimating the policy gradient is essentially a rare-event estimation problem. If directly using the data collected in NDE, it is essentially a Monte Carlo estimation approach44 as

\Psi b MC d = 1/n \sum i=1 n \nabla \pi P(\omega(X i)=1), X i \sim P\pi(X), (5)

where n is the number of samples used at each training step, and \nabla \pi P(\omega(X i)=1) could be estimated through the policy gradient theorem45. According to the properties of Monte Carlo estimation44, however, the estimator \Psi b MC would suffer from a large variance, which severely hinders the learning effectiveness. Moreover, as \Psi b MC is usually near zero, the signal-to-noise ratio is also low, which makes the problem even worse. We call this problem the CoR7.

Let us elaborate the CoR more rigorously. Without loss of generality, we define the informative samples as

\Psi b in d = 1/n \sum i=1 n \nabla \pi P(\omega(X i)=1) I(X i \in \Phi in), X i \sim P\pi(X), (6)

Then we have the following Lemma 1, and the proof can be found in ref. 7.

Lemma 1
If \Phi non and \Phi in satisfy the following conditions:
(1) \Phi non \cap \Phi in = \emptyset, \Phi non \cup \Phi in = \Omega;
(2) E P\pi [\nabla \pi P(\omega(X)=1) I(X \in \Phi non)] = 0;

then we have the following properties:
(1) E[\Psi b in] = E[\Psi b MC];
(2) \sigma^2(\Psi b(k) in) \le \sigma^2(\Psi b(k) MC), 8k=1, ... d; and
(3) \sigma^2 P\pi (\Psi b(k) in) \ge \rho \Phi in \sigma^2 P\pi (\Psi b(k) MC), 8k=1, ... d, with the assumption

E P\pi [G^2 k(\omega(X)=1) I(X \in \Phi in)] = E P\pi [G^2 k(\omega(X)=1)] E P\pi [I(X \in \Phi in)], 8k=1, ..., d (7)

where \rho \Phi in = E P\pi [I(X \in \Phi in)] \in [0, 1] is the proportion of the informative samples, and G(\omega(X)=1) def = \nabla \pi P(\omega(X)=1) is a random vector with G(\omega(X)=1)=[G 1(\omega(X)=1), ..., G d(\omega(X)=1)] \in R^d, where d is the dimension of the parameters of the policy network \pi.

Remark 1. For AV safety training, the proportion of informative samples in all samples could be very small, in the order of 10^-4 ~ 10^-6 or fewer, due to the rarity of safety-critical events in NDE. As the policy gradient \nabla \pi P(\omega(X)=1) is mainly determined by the parameters of neural networks, it could exhibit a stationary uncertainty that is independent of the set \Phi in. This is particularly true at the beginning of the learning process when the parameters are relatively random. Consequently, the assumption in Property 2 of Lemma 1 could be approximately satisfied, particularly at the beginning of the learning process. Therefore, the estimation variance of the traditional deep learning approaches based on Monte Carlo estimation could be very large.

Remark 2. Lemma 1 also indicates that if the set of informative samples \Phi in could be identified, estimating the policy gradient utilizing informative samples can reduce the learning variance without loss of unbiasedness, thereby overcoming the CoR. However, how to define, identify, and effectively leverage informative samples for AV safety training is challenging. Prior to this work, this process is largely intuitive, for instance, many existing approaches only emphasize the crash event data or falsified cases14, discussed as follows.

Learning from crash only
Many existing approaches used to tackle the CoR challenge primarily focus on learning from the data where AVs fail. To be more specific, the estimator of these approaches can be represented as

\Psi b Fail d = 1/n \sum i=1 n \nabla \pi P(\omega(X i)=1) I(X i \in \Phi F), X i \sim P\pi(X) (8)

where I(\cdot) is an indicator function and \Phi F is the set of data samples where AV fails. However, these approaches are largely intuitive without theoretical foundation. Consequently, the definition of the set \Phi F does not satisfy the conditions of \Phi in in Lemma 1, resulting in a severe learning biasedness, that is,

E[\Psi b Fail] \ne E[\Psi b MC], (9)

the learning process could be misleading. That is the reason why learning-from-crash-only could suffer from the seesaw effect as illustrated in Supplementary Section 2.3 of Supplementary Information.

Dense learning approach for AV training
In this work, we propose the dense learning approach to overcome the CoR challenge for AV training. The key to dense learning is to define and identify the informative data samples for AV safety training, satisfying the conditions of \Phi in in Lemma 1. To achieve this goal, we integrate the Lemma 1 with importance sampling theory to derive the optimal unbiased distribution of data samples. It provides the guidance for defining and identifying the informative data for AV training. Then, we sampled the data based on the distribution, rejected the non-informative samples, and effectively leveraged the informative ones, resulting in the dense learning approach for AV training.

We derived the optimal importance sampling distribution q\pi(X) based on importance sampling theory27,44,46 as

\Psi b IS d = 1/n \sum i=1 n [\nabla \pi P(\omega(X i)=1) / q\pi(X i)], X i \sim q\pi(X), (10)

q\pi(X) / ||\nabla \pi P(\omega(X)=1)||^2 P\pi(X), (11)

where ||\cdot|| denotes the l2 norm of the vector, and the symbol / means ‘proportional to’. Equation (10) provides the insights that the most informative data for AV training should contain all safety-critical events, both successful and failed, with probabilities proportional to their contributions to the AVs’ policy gradient as well as their exposure frequencies in the real world. The major challenge of \Psi b IS is that ||\nabla \pi P(\omega(X)=1)||^2 cannot be accurately calculated for each episode X in practice and is dependent of the policy \pi, which is changing during the training process. Some existing studies tried to calculate \nabla \pi P(\omega(X)=1) using the policy gradient theorem (see ref. 47 for example), which introduces additional constraints and severely limit the effectiveness and applicability of the approach.

To address this challenge, we propose to utilize Eqs. (9) and (10) as a guidance to define and identify the informative samples for the dense learning approach. Therefore, instead of trying to calculate the exact values with severely limited applicability, we choose to estimate ||\nabla \pi P(\omega(X)=1)||^2 as a classification task. Specifically, we estimate ||\nabla \pi P(\omega(X)=1)||^2 as

||\nabla \pi P(\omega(X)=1)||^2 \approx I(X \in \Phi d) = 1, X \in \Phi d, (12)

where an avoidability analysis is conducted to approximate if a crash is avoidable (P(\omega(X)=1)<1), and a safety metric was selected to approximate if a non-crash is a near-miss (P(\omega(X)=1)>0). Specifically, we identified a crash event as avoidable if an evasive trajectory is still feasible after the vehicle state is being identified as safety-critical (see Supplementary Section 2.8 in Supplementary Information). And we identified a non-crash event as a near-miss if the minimum relative distance between the AV and background vehicles is below a pre-determined threshold (that is, 2.5m). A sensitivity analysis of the threshold can be found in Supplementary Fig. 6.

Then, we obtain the estimator according to Eq. (9) as

\Psi b IS d = 1/n \sum i=1 n \nabla \pi P(\omega(X i)=1) / bq\pi(X i), (13)

where

bq\pi(X i) = \sum N i=1 I(X i \in \Phi d) / P\pi(X i) P\pi(X i) : (14)

Here N denotes the total amount of data in the collected dataset.
We then have Theorem 1, and the proof can be found in Supplementary Section 2.1.3 of Supplementary Information.

Theorem 1
The estimator \Psi b IS has the following properties:
(1) E q^ \pi [\Psi b IS] = E P\pi [\Psi b MC];
(2) \Delta^2 q^ \pi (\Psi b IS(k)) \le g N \Delta^2 P\pi (\Psi b MC(k)), 8k=1, ... d; and
(3) \Delta^2 q^ \pi (\Psi b IS(k)) \le g N \rho \Phi in \Delta^2 P\pi (\Psi b MC(k)), 8k=1, ... d, with the assumption

E P\pi [G^2 k(\omega(X)=1) I(X \in \Phi in)] = E P\pi [G^2 k(\omega(X)=1)] E P\pi [I(X \in \Phi in)], 8k=1, ..., d (15)

where g N = \sum N i=1 I(X i \in \Phi) P\pi(X i) \in (0, 1), \Psi b IS(k) is the k th components of \Psi b IS, \Delta^2 q^ \pi (\Psi b IS(k)) = \sigma^2 q^ \pi (\Psi b IS(k)) / E q^ \pi [\Psi b IS(k)] and \Delta^2 P\pi (\Psi b MC(k)) = \sigma^2 P\pi (\Psi b MC(k)) / E P\pi [\Psi b MC(k)] are coefficients of variance, and \rho \Phi in = E P\pi [I(X \in \Phi in)] \in [0, 1] is the sampling distribution P\pi(X).

Remark 4. The property of the gradient estimator is critical for overcoming the seesaw effect. We note that g N is a constant and as the learning rate is usually adjustable, such a constant will not affect the learning process.

Remark 5. The second property indicates that the new estimator \Psi b IS has the same calculation equation as the Monte Carlo estimator \Psi b MC (see Eqs. (4) and (12)), and the only difference is the data sampling distribution. This is significant as our approach could leverage all techniques that can be used for the Monte Carlo estimator (that is widely used for deep learning), without introducing any additional limitations. Therefore, our approach could leverage the advances in deep learning but with a dramatically smaller learning variance, which ensures the effectiveness and applicability of our approach.

Remark 6. The definition and identification of the informative samples \Phi in is the core of the dense learning approach. The estimator \Psi b IS is essentially an approximation of the policy gradient estimator for deep learning approaches associated with rare events. As the parameters of neural networks are usually randomly initialized, the assumption in Eq. (14) could be approximately satisfied, particularly at the beginning of the learning process, as discussed in Remark 1.

Remark 7. While the following investigation focuses on the deep reinforcement learning (DRL) tasks of AV safety training, Theorem 1 is also applicable to more generic deep learning tasks. Therefore, our approach has the great potential to address the CoR challenge in generic deep learning tasks associated with rare events.

To further integrate the estimator \Psi b IS with DRL approaches, we obtain the policy gradient estimator of DRL as

\Psi b DRL d = Q b \pi (S t, A t) \nabla \pi \pi (A t | S t), (S t, A t) \sim P\pi(X), (16)

where S t and A t are samples of the state and action following the policy \pi, i.e., E P\pi [Q b \pi (S t, A t) \nabla \pi \pi (A t | S t)] is the policy gradient. As discussed in our previous study, our approach is compatible with existing techniques that can be used for the Monte Carlo estimator. Therefore, we integrate the estimator \Psi b IS with the dense deep reinforcement learning (D2RL) approach that was developed in our previous study for AV testing8. Although the AV training problem is different from the AV testing problem, the D2RL approach can be applied to the AV training problem by identifying the safety-critical states. We then have Theorem 2, and the proof can be found in Supplementary Section 2.1.7 of Supplementary Information.

The estimator \Psi b Dense has the following properties:
(1) E q^ \pi [\Psi b Dense] = g N E P\pi [\Psi b DRL];
(2) \Delta^2 q^ \pi (\Psi b Dense(k)) \le g N \Delta^2 P\pi (\Psi b DRL(k)), 8k=1, ... d;
(3) \Delta^2 q^ \pi (\Psi b Dense(k)) \le g N \rho \Phi D \Delta^2 P\pi (\Psi b DRL(k)), 8k=1, ... d, with the assumption in Eq. (14);
(4) \Delta^2 q^ \pi (\Psi b Dense(k)) \le g N \rho \Phi D \Delta^2 P\pi (\Psi b DRL(k)), 8k=1, ... d, with the assumption in Eq. (14);

\Psi b Dense d = Q b \pi (S t, A t) \nabla \pi \pi (A t | S t) I(S t \in \Phi D, (S t, A t)) / bq\pi(X), (17)

where \Psi b Dense(k) and \Psi b DRL(k) are k th components of \Psi b Dense and \Psi b DRL, respectively, \Delta^2 q^ \pi (\Psi b Dense(k)) = \sigma^2 q^ \pi (\Psi b Dense(k)) / E q^ \pi [\Psi b Dense(k)], \Delta^2 P\pi (\Psi b DRL(k)) = \sigma^2 P\pi (\Psi b DRL(k)) / E P\pi [\Psi b DRL(k)], \Delta^2 P\pi (\Psi b Dense(k)) = \sigma^2 P\pi (\Psi b Dense(k)) / E P\pi [\Psi b Dense(k)] are coefficients of variance, and \rho \Phi D = E P\pi [I(X \in \Phi D)] \in [0, 1] is the expected proportion of the informative states \Phi D = {S t \in X : S t \in S c and X \in \Phi g in all sampled states with the sampling distribution P\pi(X) and under the policy \pi.

Remark 8. Theorem 2 indicates that \Psi b Dense is an unbiased and efficient policy gradient estimator of the DRL approaches, which is critical for overcoming the seesaw effect and CoR for AV training. As \rho \Phi D is the proportion of the informative states, the variance of the policy gradient \Psi b Dense could be reduced by the factor \rho \Phi D, which is the key to overcoming the CoR for AV training.

Remark 9. As the policy gradient \Psi b DRL d = Q b \pi (S t, A t) \nabla \pi \pi (A t | S t) is mainly determined by the parameters of neural networks, it could exhibit a stationary uncertainty that is independent of the set \Phi D. This is particularly true at the beginning of the learning process when the parameters are relatively random. Therefore, the assumptions in (17) could be approximately satisfied. We note that the assumptions are primarily for the theoretical analysis to be clean and are not strictly required in practice.

Additional data densification approaches
Offline resample mechanism. To realize the estimator \Psi b DRL, we design an offline resampling mechanism to resample and redistribute the collected episodic data according to the distribution bq\pi(X i) as in Eq. (13). An avoidability analysis was developed to help approximate the distribution bq\pi(X i). This mechanism is applicable for data collected through different venues such as simulations, test tracks, and public roads. It also provides compatibility to integrate with purposely generated testing environments (such as ITE) where data is collected by a different distribution, and only the offline resampling distribution need to be modified accordingly.

Learned safety metric. Most existing safety metrics are primarily based on trajectory prediction of background road users with assumptions on their driving behaviors, which limit their effectiveness and generalization capability. This is particularly true for AV safety training, as the AV policy is changing during the training process. To address this issue, we designed a learned safety metric to identify the safety-critical states with both high precision and recall rates. We utilized the avoidability analysis approach to provide ground-truth labels for the large-scale naturalistic trajectory datasets. Specifically, all snapshots where a crash is unavoidable with evasive driving behaviors are labeled as positive, while others are labeled as negative. Since unavoidable snapshots are rare, existing training approaches suffer from the severely unbalanced data issue48. Instead of rebalancing the data before the training, we proposed to resample positive and negative snapshots from all the data to create balanced training batches in each training step. In this way, the training data in each training step is rebalanced and densified. Results demonstrate that our approach enables a much better precision and recall performance than existing safety metrics (see Supplementary Fig. 1).

Retrospective data densification. Ideally, as the policy \pi is changing during the training process, the episodic data should be re-collected according to the new policy at each training step. However, this is difficult in practice. To address this, we introduce a retrospective evaluation component to re-identify the informative episodes and states for the new AV policy through a counterfactual simulation. This mechanism could update the values of I(\omega(X)=1) and P(\omega(X)=1) in Eq. (11), so the data resampling distribution bq\pi(X i) could be updated for the new AV policy, which reduces the overall policy gap.

Counterfactual simulation. We utilized the counterfactual simulation49 to re-evaluate the safety performance of the AV policy and identify the informative episodes and states in the training data. In the counterfactual simulation, we re-simulate the AV with the new AV policy in the same driving environment as the original simulated AV. SafeDriver handles all the safety-critical states identified by the learned safety metric; otherwise, the base AV model takes control. Calibrated non-linear bicycle models are used to simulate the motion of the AV with control command (i.e., steering angle and acceleration) inputs from the planning tasks. Specifically, we employed the dynamic bicycle model when AV speed is greater than 20 m/s and the kinematic bicycle model when AV speed is less than 20 m/s. The simulation is replayed based on the recorded data until the recorded trajectories reach their end. For crash trajectories, if SafeDriver could avoid the original collision in the counterfactual simulation, we need to re-simulate the driving behaviors of the background vehicles after the original collision moment to interact with SafeDriver, until the collision could be avoided completely or a new collision occurs. Specifically, we simulated the background vehicles by a predefined driving model, such as the IDM and SL2015 provided by SUMO35. The simulation will be terminated either upon collision or after a certain duration has elapsed.

Intelligent testing environment for efficient data collection
As ||\nabla \pi P(\omega(X)=1)||^2 is zero for most episodic data X and non-trivial AV policies \pi, most data collected in NDE has little information for AV training and will be rejected by the offline sampling mechanism as Eq. (13). To improve the data collection efficiency, we utilized the intelligent testing environment (ITE) that was developed in our previous study8, in that AI-based background agents are trained to test AVs in an accelerated mode. Specifically, ITE will provide a dataset with a new distribution q ITE(X), each episode of data with a weight W ITE(X) = P\pi(X) / q ITE(X), which contains much more valuable safety-critical episodes. To leverage this dataset, the safety-critical episodes are resampled by the distribution proportional to their weights W ITE(X) and then resampled according to Eq. (13). We found out that, with the dense learning approach, ITE could accelerate the collection of safety-critical data by multiple orders of magnitude, which dramatically accelerates the training process of AVs’ overall safety performance (see Supplementary Fig. 2). We believe that this approach opens the door for integrating AV testing and training together, accelerating both fields, which deserves further investigation.

Training settings of dense learning
We implemented the dense learning approach in highways, roundabouts, and urban environments, respectively. To keep the fidelity and efficiency of the simulations, we utilized the NDE simulation and ITE as described in Supplementary Sections 2.6 and 2.7 of Supplementary Information. We applied the PPO algorithm developed on the RLLIB 1.11.0 platform50 to parallelly train SafeDriver on 400 CPU cores and 2800 GB memory high-performance computation cluster at the University of Michigan, Ann Arbor. We created a three-layer fully connected neural network, consisting of 256 neurons in each layer, to represent the SafeDriver. The neural network received input data comprising the states of up to 6 background vehicles located within 30m from the AV, where its output is the AV’s acceleration and steering angle. Specifically, for the highway environment, the acceleration spanned from −4 m/s² to 2 m/s², and the steering angle varied between −10 degrees and 10 degrees; for roundabout environments, the acceleration ranged from −8 m/s² to 4.5 m/s², and the steering angle covered a range from −45 degrees to 45 degrees; and for the urban environment, the acceleration exhibited a range between −7 m/s² and 2.5 m/s², while the steering angle was within the −10 degrees to 10 degrees range. We set the learning rate as 10^-5 and the discount factor as 0.99. In each training iteration, a total of 50,000 time steps were used to train the neural network for 30 times. For each driving episode, the reward function is set as −1 for an AV-involved crash and 0 for others.

Field testing settings
Autonomous vehicle with SafeDriver. The vehicle under test is a Lincoln MKZ hybrid from Mcity at the University of Michigan, equipped with various sensors, including OTXS RT3003 RTK GPS, Point Grey camera, Velodyne 32 channel LiDAR, Delphi radars, Xsens MTi GPS/IMU, etc. The AV also had the Nuvo-8208GC computer and the Dataspeed drive-by-wire system installed. We applied the open-source full-stack autonomous driving system, Autoware25, as the base AV model. Specifically, after receiving the ego vehicle’s position and velocity as well as background vehicles’ information, the future path is generated based on Open Planner 1.1351. We applied the pure pursuit algorithm to convert the planned trajectory into the velocity and yaw rate and then used a proportional-integral-derivative controller provided by Data Speed Inc. to further convert them into the vehicle by-wire control commands, that is, steering angle, throttle, and brake percentages. When the learned safety metric alerts in safety-critical situations, SafeDriver receives the normalized observation and outputs the steering angle and acceleration, which are further converted to the throttle and brake percentages.

Mixed reality testing environment. We applied the mixed reality testing environment52 to efficiently evaluate the safety performance of AVs at Mcity, which is one of the leading testing facilities for AV development. We forwarded the states of the background vehicles from the simulation world and the signal information from the physical testing track to the AV through the Internet. Simultaneously, the states of the AV as well as proxy objects and the signal information were synchronized into the simulation world, and the behaviors of virtual background vehicles were determined according to NDE or ITE. Besides, the virtual background vehicles were rendered and blended onto the front camera’s view using pyrender53. To accelerate the testing process for evaluating the crash rates of the AV in NDE, we utilized the ITE that was developed in our previous study8. A framework of the mixed reality testing framework can be found in Supplementary Fig. 7.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Thank you for preparing a revision. I will go through my previous comments step by step. Each comment will have the\nfollowing structure:\na) my initial comment,\nb) response,\nc) new comment,\nd) new response,\ne) latest comment.\n1. Provably safe reinforcement learning\na) Why are the presented results not discussed with respect to approaches that are already provably safe. From a safety\nperspective, the provably safe methods are obviously superior so that the presented approach does not improve the state of\nthe art in that regard.\nb) Such a safety model is not available in the applications of AVs, and further investigations are needed.\nc) These safety models are available for AV, see, e.g., “X. Wang and M. Althoff, “Safe Reinforcement Learning for Automated\nVehicles via Online Reachability Analysis,” in IEEE Transactions on Intelligent Vehicles, doi: 10.1109/TIV.2023.3344226. ”\nand references therein.\nd) We have added experimental comparisons between our approach and the provably safe reinforcement learning model\nproposed.\ne) Unfortunately, the suggested approach has been incorrectly used. The authors write that \"During the evaluation of the\nprovably safe reinforcement learning model, the safety shield occasionally fails to find feasible solutions in safety-critical\nscenarios, with an infeasibility rate as high as 1.75 × 10−2\". This is actually impossible. The provably safe approach is\nprovably safe as it always has a feasible fail-safe solution available. The proof is by induction. Induction start: The vehicle\nstart in a standstill, which is safe. Induction step: The vehicle computes the over-approximative reachable set of other traffic\nparticipants. If the first part of the intended trajectory plus the fail-safe trajectory are safe against the reachable sets of all\nother traffic participants, the new intended plan can be executed. Otherwise, the previous plan can be executed, which\nremains safe as all possible behaviors of other traffic participants have already been considered in the reachable set. The\nonly possibility that a crash can occur is when another traffic participant cuts-in so that a collision is inevitable -- in such a\nsituation the automated vehicle is not at fault. The authors, however, did not distinguish between inevitable collision\nscenarios (which are super rare) and those that are not. If the authors believe that the proof is incorrect, they should mention\nthis in their paper. Otherwise, they presumably implemented the provably safe method incorrectly.\n2. Generalizability\na) The submitted paper does not present a novel machine learning algorithm, but rather an approach for properly selecting\nscenarios in reinforcement learning. Whether this approach is for reinforcement learning is actually not explicitly mentioned.\nIt is also unclear how well the selection of scenarios generalizes. Can this also be used in other application areas, such as\nrobotics, health diagnostics, power systems, etc.? Because the results are only obtained for autonomous driving, the authors\nshould mention that the generalizability is not validated for other application areas.\nb) The presented approach is basically for deep reinforcement learning approaches based on policy gradient and it is\nunclear how well the approach generalizes to other domains.\nc) This is very valuable information, but as far as I can see, this has not been made more explicit in the paper. A reader\nwould not immediately know if they can apply the approach to their own problem.\nd) More research is needed to extend our approach to more generic safety-critical autonomous systems.\ne) Since the authors have not demonstrated the applicability in other domains, they should remove statements that indicate\nthat the approach is applicable to other domains as well.\n3. Novelty\na) Properly selecting scenarios has been addressed in previous works as well. A clear description of the novelty of the paper\nwith respect to the state of the art would certainly be helpful.\nb) The authors agree that several works exist on this topic. They also say that ”We argue that the testing and training for AV\nsafety suffer from a fundamentally different challenge, named the curse of rarity, due to the rarity of safety-critical events\nassociated with high-dimensional variables.”\nc) It is a known fact that only very few scenarios in autonomous driving are of great interest from a safety perspective. We all\nknow this from our own driving experience. A clear justification is still missing in the paper, e.g., one that shows that the\naddressed problem is different from a mathematical point of view.\nd) We thank the valuable suggestion of the reviewer. We have added detailed explanations on the methodological\ncontributions of our approach from a mathematical point of view.\ne) I am sorry, but I still cannot find a crisp mathematical problem statement in this paper. There are many pages in the\nrebuttal the authors refer to. However, I cannot find a connection why the considered problem is different from previous\npapers.\n4. New issues in the rebuttal\n- Def. 1: Why include actions? Trajectories usually are functions from time to the state of the system.\n- Def. 2: How can one \"enter\" a trajectory? A trajectory is not a set.\n- Remark 1: Crash rate seems to be higher than that of humans. Please clarify.\n- Eq. (6) in rebuttal: constraint for t is unnecessary.\n- Below (11) in rebuttal: Isn't the condition trivially greater than 0 other than that it is 0, which one could state directly.\n(Remarks on code availability)\nThe code for the evaluation against provably safe methods is most likely incorrect as detailed in my response above.", "reviewer_3": "Response to the Reviewers’ Comments\nThe authors would like to thank the reviewers for their constructive comments. In response to these\ncomments, we have revised the paper carefully. We believe the revised paper has adequately\naddressed the issues raised in the reviewers’ comments. In the following, we will provide a\nsummary of major revisions made in the revised manuscript (highlighted in red in the manuscript),\nas well as a point-to-point response to all comments.\nResponse to Reviewer #1\nComment 1a. Provably safe reinforcement learning.\nThe provably safe approach is provably safe as it always has a feasible fail-safe solution available.\nThe proof is by induction. Induction start: The vehicle start in a standstill, which is safe. Induction\nstep: The vehicle computes the over-approximative reachable set of other traffic participants. If the\nfirst part of the intended trajectory plus the fail-safe trajectory are safe against the reachable sets\nof all other traffic participants, the new intended plan can be executed. Otherwise, the previous\nplan can be executed, which remains safe as all possible behaviors of other traffic participants\nhave already been considered in the reachable set."}, "author_response": "Response to the Reviewers’ Comments\nThe authors would like to thank the reviewers for their constructive comments. In response to these\ncomments, we have revised the paper carefully. We believe the revised paper has adequately\naddressed the issues raised in the reviewers’ comments. In the following, we will provide a\nsummary of major revisions made in the revised manuscript (highlighted in red in the manuscript),\nas well as a point-to-point response to all comments.\nResponse to Reviewer #1\nComment 1a. Provably safe reinforcement learning.\nThe provably safe approach is provably safe as it always has a feasible fail-safe solution available.\nThe proof is by induction. Induction start: The vehicle start in a standstill, which is safe. Induction\nstep: The vehicle computes the over-approximative reachable set of other traffic participants. If the\nfirst part of the intended trajectory plus the fail-safe trajectory are safe against the reachable sets\nof all other traffic participants, the new intended plan can be executed. Otherwise, the previous\nplan can be executed, which remains safe as all possible behaviors of other traffic participants\nhave already been considered in the reachable set.\nResponse:\nWe respectfully disagree with the statements that \"the provably safe approach is provably safe\nas it always has a feasible fail-safe solution available\", and “all possible behaviors of other traffic\nparticipants have already been considered in the reachable set”. The proof is correct, but only\nunder specific assumptions — particularly regarding the behaviors of other traffic\nparticipants. As pointed out by ref.[1], a well-known review of formal methods, \"All formal\nverification and synthesis, however, is with respect to an environment model. Hence, assumptions\nabout and properties of input data must be explicated into environment models\", indicating that\nprovably safe is only valid under certain assumptions. For AV driving tasks, the assumptions could\ninclude those on vehicle dynamics, traffic rules, traffic behaviors, bounds of disturbances, finite\ntime horizon, weather conditions, etc. For instance, it is assumed that a safe distance to the ego\nvehicle must be kept by other traffic participants when driving behind the ego vehicle or merging\nin front of it (see the constraint of safe distance imposed on other traffic participants in ref.[2],\na frequently-used tool for computing reachable\n𝐶\ns𝑠e𝑎t𝑓).𝑒 When the assumptions are violated, which\nis possible in real applications, the provable safety will not be guaranteed. Examples can be\nfound in our response to Comment 1b. Another example, reachable set calculations are often based\non max vehicle acceleration/deceleration rate, which might be impacted by road surface conditions\nunder different weather conditions that are difficult to predict precisely. Therefore, provable safety\ncan only be guaranteed under certain assumptions with the environment model, which can be\nviolated.\nTo further specify these assumptions both theoretically and empirically, we have also analyzed\nseveral representative research articles. Specifically, we have consulted Prof. Matthias Althoff from\nTechnical University of Munich, as mentioned by Reviewer #1, on the topic of provably safe\nreinforcement learning, and he recommended two references [3] and [4]. These two references\nstudy the action projection for ensuring safety and the fail-safe planning, respectively. We therefore\nbelieve that the references [3] and [4] are representative of the provably safe reinforcement learning\napproach highlighted by Reviewer #1.\nIn ref.[3], the assumptions include: (a) dynamics of the AV and other traffic participants (see\nEquation 1 in ref.[3] and Definition 8 in its reference[2]), (b) the bound of process noises for AV's\ncontrol inputs and measurement errors for states of the AV (see Equation 1 and Definition 5 in\nref.[3]), (c) the traffic behaviors of other traffic participants (see Table 1 and corresponding\nDefinitions 11-15 in ref.[2]), and (d) within a finite time horizon where a feasible safe action can be\nobtained (see Section IV in ref.[3]). The limitation caused by these assumptions is also explicitly\nacknowledged by the authors in ref.[3] that \"Due to over-approximation errors, our safety shield\nmight not be able to always find a feasible solution if one exists\" and \"Our safety shield approach\ncan guarantee safety only for the finite time horizon t_f \". That is, when AV reach out of the time\nhorizon, it is not guaranteed that a feasible safe action can always be found, so a collision with\nother traffic participants could occur.\nIn ref.[4], the first part of assumptions include the longitudinal and lateral dynamics of the AV\n(see Sections II.B.1 and II.B.2 and Equation 1 in ref.[4]). It assumes that the difference between the\nreference path orientation and current orientation of AV is negligibly small, thus the lateral and\nlongitudinal dynamics can be decoupled and the dynamics can be approximately linearized to\nfacilitate solving the fail-safe trajectory (see \"we add a constraint for the ego vehicle to\napproximately follow the pre-defined reference path , i.e., is negligibly small, which enables\nus to utilize the small-angle assumptions sin( ) ≈\n𝛤\nand c\n𝑒\no𝜃s( ) ≈ 1\" in ref.[4]). However, there\nmay be mismatches in AV dynamics, and th\n𝑒\ne𝜃 AV\n𝑒\nm𝜃 ay deviat\n𝑒\ne𝜃 from the reference path for the\nmoment, causing errors in decoupling and linearizing dynamics. The second part of assumptions\ninclude the traffic behaviors of other traffic participants (see Table III in ref.[4] and Table I in the\nreference[2] used by ref.[4]), while there may also be mismatches in dynamics of other traffic\nparticipants and violation of assumed rules. Therefore, all these assumptions could be violated.\nAlthough the authors use uncertainties and disturbance to form a relatively conservative\ndynamics, the safety-critical corner cases caused by violated assumptions are still hard to be\nfully covered. Therefore, the fail-safe planning cannot always guarantee safety in real application.\nIn summary, the provably safe approach can only guarantee safety under specific assumptions\nincluding vehicle dynamics, behaviors of other traffic participants, bounds of disturbances, finite\ntime horizon, etc. Since these assumptions could be violated in practice, the safety performance\ncannot be always guaranteed. In fact, the state-of-the-art AVs are still suffering from the critical\nsafety challenges (please see references [5] and [6]). How to continuously improve the safety\nperformance and further reduce the crash rates of AVs still needs breakthrough, which is the focus\nof this study.\nRevisions.\nWe have added the above explanation in the Supplementary Materials in the revised\nmanuscript. Please see Section 2.5.\nReferences.\n[1] Seshia S A, Sadigh D, Sastry S S. Toward verified artificial intelligence[J]. Communications\nof the ACM, 2022, 65(7): 46-55.\n[2] Koschi M, Althoff M. Set-based prediction of traffic participants considering occlusions and\ntraffic rules[J]. IEEE Transactions on Intelligent Vehicles, 2020, 6(2): 249-265.\n[3] Kochdumper N, Krasowski H, Wang X, et al. Provably safe reinforcement learning via action\nprojection using reachability analysis and polynomial zonotopes[J]. IEEE Open Journal of Control\nSystems, 2023, 2: 79-92.\n[4] Wang X, Althoff M. Safe reinforcement learning for automated vehicles via online reachability\nanalysis[J]. IEEE Transactions on Intelligent Vehicles, 2023.\n[5] California Department of Motor Vehicles. Autonomous Vehicle Collision Reports (2025).\nhttps://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-\nvehicle-collision-reports/.\n[6] Wang H, Shao W, Sun C, et al. A survey on an emerging safety challenge for autonomous\nvehicles: Safety of the intended functionality[J]. Engineering, 2024, 33: 17-34.\nComment 1b. The authors write that \"During the evaluation of the provably safe reinforcement\nlearning model, the safety shield occasionally fails to find feasible solutions in safety-critical\nscenarios, with an infeasibility rate as high as 1.75 × 10−2\". This is actually impossible.\nThe only possibility that a crash can occur is when another traffic participant cuts-in so that a\ncollision is inevitable -- in such a situation the automated vehicle is not at fault. The authors,\nhowever, did not distinguish between inevitable collision scenarios (which are super rare) and\nthose that are not.\nResponse:\nWe have replicated the agents in references [3] and [4] recommended by Prof. Althoff, and\nthe results illustrate that the reasons of infeasible situation and collision include not only other\ntraffic participants' cut-in but also the violation of several assumptions summarized in our response\nto Comment 1a—in which cases the AV could be also at fault. Note that the reference [3] is open\nsource, so we provide the detailed results and source code (more implementation details can refer\nto our response to Comment 1c), while the reference [4] is not open source and the code is shared\nprivately by Prof. Althoff, so we only provide the results. Meanwhile, we have to emphasize the\ndifference between infeasible cases and collision cases. The \"infeasible\" in our context does not\nrefer to the situation of inevitable collision, but failing to find a feasible safe action, which\nonly represents the inevitable intersection between the reachable sets of AV and BVs. It is\nmainly caused by the conservativeness, as mentioned in ref.[3] (see \"Due to over-approximation\nerrors, our safety shield might not be able to always find a feasible solution if one exists\"). In fact,\nmost infeasible cases do not lead to collisions in this case study. The final collision rates of the\nAV in references [3] and [4] are only 4.77e-4 and 1.72e-3 respectively.\nWe would like to use several illustrations to help better understand the \"infeasible\" cases and\ntheir relations to collisions. First, we present three scenarios generated by the source code from\nref.[3]:\n The process from safe to infeasible. In Figure 1, the provably safe AV first plans a safe\naction, which guarantees its safety within a finite time horizon as assumed. However, when\nthe AV reaches the end of the safe action, it is too close to the BV on the left and their reachable\nsets inevitably overlap, thus it fails to find a new safe action. At this time, the AV is out of\nthe assumed time horizon of the last safe action and the assumption for guaranteeing safety\nis violated. Consequently, the AV falls into an infeasible situation without any BV cutting in.\nFigure 1. From safe to infeasible. The blue vehicle is AV and the green vehicles are BVs.\nThe blue and green zones are reachable sets of AV and BVs respectively. The blue line is the\nsafe action planned by AV.\n Infeasible situation without a collision. After encountering the infeasible situation in Figure\n1, the AV automatically cancels the left-turn maneuver to avoid the BV, and eventually\nreaches the destination without any collision, as shown in Figure 2. We emphasize by this case\nthat most infeasible situations do not lead to collisions in real applications. It indicates that\nmany collisions are avoidable after infeasible situations.\n[Figure Redacted]\nFigure 2. Infeasible situation without collision (for ref.[3]).\n AV-responsible collision after an infeasible situation. In Figure 3, the provably safe AV\nfirst plans a safe action to change lane to the left. However, when halfway through the lane\nchange and out of the assumed time horizon of the last safe action, the AV follows the front\nBV too closely and enters an infeasible situation. It executes a default action and fails to avoid\nthe BV, finally colliding with the BV from behind. This is a typical AV-responsible collision\nafter an infeasible situation.\n[Figure Redacted]\nFigure 3. AV-responsible collision after an infeasible situation (for ref.[3]).\n BV-responsible collision after an infeasible situation. In Figure 4, a BV suddenly changes\nlane to the left at a very high speed when the provably safe AV drives normally, violating the\nassumption that BV should keep a safe distance behind AV. The AV fails to plan a feasible\nevasion action and enters an infeasible situation, ending up with a collision. This is a typical\nBV-responsible collision after an infeasible situation.\n[Figure Redacted]\nFigure 4. BV-responsible collision after an infeasible situation (for ref.[3]).\nAs discussed by the 5th \"common sense\" rules in a work of responsibility-sensitive safety[7], if\nthe AV can avoid an accident without causing another one, the AV must do it. In comparison\nto the provably safe AV, our safe driver can perform better in these infeasible situations and reduce\nthe crash rate. In the same scenarios as Figure 3 and 4, our safe driver successfully avoids the\ncollisions. It indicates that our approach can actually do better in these cases. We illustrate the\nresults as follows:\n Avoiding the AV-responsible collision. Comparing Figure 3 and 5, we can see that the AV\nfirst perform the same action of left lane change. However, our approach detects the danger in\nmore advance and takes over to avoid completing the lane change. It decelerates to a greater\nextent and successfully turns right. It finally returns to the middle lane and does not collide\nwith the BV on the left.\nFigure 5. Avoiding the AV-responsible collision by our safe driver.\n Avoiding the BV-responsible collision. Comparing Figure 4 and 6, we note that the BV first\nperform the same dangerous maneuver, i.e., changing to the left lane with a high speed.\nHowever, our approach also identifies the danger early and accelerates decisively, managing\nto avoid the collision with considerable effort.\nFigure 6. Avoiding the BV-responsible collision by our safe driver.\nSecond, we present three scenarios generated by the source code from ref.[4]:\n 1# collision case after an infeasible situation. In Figure 7, the provably safe AV first plans\na safe action to change to the left lane. However, when halfway through the lane change,\nthe\nBV on the left behind does not decelerate to keep the safe distance with AV as assumed.\nFinally, the AV enters an infeasible situation and collides with BV from the side. Since the\ncollision occurred during the AV’s lane change, the AV arguably bears partial\nresponsibility for the incident.\n[Figure Redacted]\nFigure 7. Collision 1 after an infeasible situation (for ref.[4]).\n 2# collision case after an infeasible situation. In Figure 8, the provably safe AV accelerates\nafter the lane change to increase the distance with the BV behind. At this time, the BV on the\nright front suddenly cuts in with a relatively low speed and does not keep the safe distance\nas assumed. The AV could not slow down in time and enters an infeasible situation, finally\ncolliding with the BV from behind.\n[Figure Redacted]\nFigure 8. Collision 2 after an infeasible situation (for ref.[4]).\n 3# collision case after an infeasible situation. In Figure 9, the provably safe AV decelerates\nafter changing to the right lane. The action planned by the AV reduces its speed to nearly 0,\nand the safety layer considers this a safe action because it assumes that the following BV will\ndecelerate to maintain a safe distance. However, the BV cannot keep the assumed safe distance\nwith the AV and finally collides with it from behind. Since highways usually have minimum\nspeed requirements, the AV may also bear some responsibility for this collision.\n[Figure Redacted]\nFigure 9. Collision 3 after an infeasible situation (for ref.[4]).\nRevisions.\nWe have added the above explanation in the Supplementary Materials in the revised\nmanuscript. Please see Section 2.5.\nReferences.\n[7] Shalev-Shwartz S, Shammah S, Shashua A. On a formal model of safe and scalable self-driving\ncars[J]. arXiv preprint arXiv:1708.06374, 2017.\nComment 1c. Unfortunately, the suggested approach has been incorrectly used.\nIf the authors believe that the proof is incorrect, they should mention this in their paper.\nOtherwise, they presumably implemented the provably safe method incorrectly.\nResponse:\nWe do not consider the proof of provably safe method to be incorrect, but we emphasize that\nprovable safety is only valid under certain assumptions, as we demonstrated in our response to\nComment 1a. The code we implemented for provably safe reinforcement learning is from ref.[3].\nWe emailed the authors and they graciously provided the SOTA version. Note that ref.[3] does not\nhave the fail-safe planning module. Moreover, we have newly implemented the AV from reference\n[4], a work of fail-safe planning as emphasized by the Reviewer #1, with the code shared by the\nauthors privately. We believe that these are representative works of provably safe\nreinforcement learning and experimental analysis on them can reflect the SOTA\nperformance of provably safe AV.\nFor the code from ref.[3], we have made necessary changes for a fair and effective comparison.\nSpecifically, since the HighD dataset used in ref.[3] only contains very limited safety-critical\nscenarios, we use a naturalistic and adversarial testing environment (see Sections 6.1 and 6.2 in our\nSupplementary Materials) for safety performance evaluation to improve the effectiveness and\nefficiency of evaluation (see the source code we provided for details). To make a fair comparison,\nwe fine-tune the parameters in ref.[3] to better fit this testing environment, including the maximum\nacceleration and maximum velocity for calculating the reachable set of BVs, the range of\nadmissible acceleration of the AV, the time horizon of safe planning, and assumed disturbances in\nAV's dynamics. To verify that all the changes for fine-tuning are necessary, we have conducted\nsensitivity analysis of these parameters, as shown in Table 1, and selected the best\nexperimental setup (No.2) as the baseline model for a fair comparison. Therefore, we argue\nthat the suggested approaches have been correctly implemented and compared fairly.\nFor the code from ref.[4] that is not open source, we kept its original settings. The overall crash\nrates of reference [3] and [4] are 4.77e-4 and 1.72e-3, respectively. Details of cases for ref.[4] can\nbe found in our response to Comment 1b.\nTable 1. Experimental results of provably safe agent.\nNo. Range of Maximum Maximum Time Consider Infeasible Crash rate\nadmissible acceleration velocity in horizon of disturbances rate\nacceleration in reachable reachable safe in AV's\nfor AV set of BVs set of BVs planning dynamics\n1 [-1, 1] m/s2 11.5 m/s2 50.8 m/s 0.8 s True 0.0903 0.00877\n2 [-4, 2] m/s2 2 m/s2 45 m/s 0.8 s True 0.0174 0.000477\n3 [-4, 2] m/s2 11.5 m/s2 50.8 m/s 0.8 s True 0.0513 0.000814\n4 [-1, 1] m/s2 2 m/s2 45 m/s 0.8 s True 0.0250 0.00627\n5 [-4, 2] m/s2 2 m/s2 45 m/s 0.8 s False 0.0173\n6 [-4, 2] m/s2 2 m/s2 45 m/s 1.2 s True 0.0437\n7 [-4, 2] m/s2 2 m/s2 45 m/s 1.2 s False 0.0260\n*No.1 indicates the original settings in ref.[3].\nRevisions.\nWe have added the above explanation in the Supplementary Materials in the revised\nmanuscript. Please see Section 2.5.\nComment 2. Generalizability.\nSince the authors have not demonstrated the applicability in other domains, they should remove\nstatements that indicate that the approach is applicable to other domains as well.\nResponse. We argue that our approach has the potential to apply in other fields with similar safety\nproblems, as the effectiveness of our approach is guaranteed by the Theorem 1 in our manuscript.\nHowever, we have added \"is theoretically applicable\" to avoid misunderstandings, and we have\nacknowledged that further investigations are needed to validate the applications in other fields.\nRevisions. We have modified our manuscript to \"The dense learning approach proposed in this\nwork is theoretically applicable to the reinforcement learning problem that aims to minimize the\nexpectation of a rare event with an underlying distribution, although further investigations are\nrequired to validate these applications\" (see lines 296-298).\nComment 3. Novelty.\nI am sorry, but I still cannot find a crisp mathematical problem statement in this paper. There are\nmany pages in the rebuttal the authors refer to. However, I cannot find a connection why the\nconsidered problem is different from previous papers.\nResponse. We have demonstrated the mathematical problem statement in our previous rebuttal,\nplease refer to Problem 1 in the Supplementary Materials (pages 3-4). Meanwhile, we have also\nclearly demonstrated the difference between this work and previous approaches, please refer to\nSection 2.1 and 2.2 in the Supplementary Materials (pages 17-19).\nComment 4a. Def. 1: Why include actions? Trajectories usually are functions from time to the state\nof the system.\nResponse. While it is true that in some control-theoretic contexts a trajectory refers only to the\nsequence of states over time, in reinforcement learning and MDP literature, it is standard to define\na trajectory as a sequence of state-action transitions[1]. This definition captures both the evolution\nof the system and the decisions made by the agent, which are essential when analyzing policy-\ninduced behavior or reasoning about distributions over trajectories.\nReferences.\n[1] Sutton R S, Barto A G. Reinforcement learning: An introduction[M]. Cambridge: MIT press,\n1998.\nComment 4b. Def. 2: How can one \"enter\" a trajectory? A trajectory is not a set.\nResponse. We agree that the phrasing \"enters a trajectory\" may be misleading. Our intention was\nto express that an event occurs when the agent executes a trajectory such that , i.e.,\nthe trajectory satisfies the condition defining the event. In other words, the agent's behavior follows\n𝑿 𝜔(𝑿)= 1\na path in the trajectory space for which the event indicator is true. For clarity, we have revised our\nDefinition 2 accordingly to \"An event occurs when an agent executes a trajectory that satisfies\nthe corresponding event specification, i.e., \" (see lines 89-90 in Supplementary\n𝑿\nMaterials).\n𝜔(𝑿)= 1\nComment 4c. Remark 1: Crash rate seems to be higher than that of humans. Please clarify.\nResponse. We mention \" denotes the crash rate such as crash per mile\n−5\nof driving\" only as an ex\n𝐸\nam𝑃𝜋p\n[\nl\n𝑃\ne\n(\n𝜔\nt\n𝜔\no\n(\n𝑿\nh\n𝑿\ne\n)\n=lp\n=un\n1\nd\n)\n]e\nrstand the meaning of the objecti\n1\nv\n0\ne function. It is not\nrelated to the crash rate of human drivers.\nComment 4d. Eq. (6) in rebuttal: constraint for t is unnecessary.\nResponse. We agree that the constraint for t is unnecessary and we would modify Equation 6 in\nSupplementary Materials to\n−1, 𝜔(𝑿)= 1,\n𝑟(𝑺𝑡,𝑿)= {\n0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒.\nComment 4e. Below (11) in rebuttal: Isn't the condition trivially greater than 0 other than that it\nis 0, which one could state directly.\nResponse. Although can only be either greater than 0 or equal to 0,\nthe definition of the se 𝑃 t𝜋 o ( f 𝑿 in ) f ∥ or 𝛻 m𝜋𝑃 at ( iv 𝜔 e ( e 𝑿 v ) en = ts 1 ) ∥2 is\nnot trivial since we exclude the case wher 𝒯 e𝒸 ={𝑋 ∈ Ω∣∣ 𝑃𝜋(𝑿)∥ 𝛻𝜋𝑃(𝜔(𝑿 e ) q = ua 1 ls ) ∥ to2 > 0. 0 T } his\ndefinition is also in line with our subsequent theo\n𝑃\nry𝜋\n(\n(\n𝑿\npl\n)\n=ea\n∥\ns\n𝛻\n𝜋\n𝑃\n(\n𝜔\n(\n𝑿\n)\n=\n1\n)\n∥2\n=\n0\n) and Theorem 1 in\nSupplementary Materials). There are two situations where it equals to 0: (a) ,\nindicating that the event is almost impossible to happen; (b) 𝑃𝜋,\n(\ni\n𝑿\nnd\n)\n=ic\n0\nating that the event provides almost no information. Both of the\n∥\ns\n𝛻\n𝜋\n𝑃\n(\n𝜔\n(\n𝑿\n)\n=\n1\n)\n∥2\n=\n0\nsituations are useless to the\ntraining of AV. We believe that our Definition 3 in Supplementary Materials is accurate, but we\nhave added more explanations to this definition as suggested by the Reviewer.\nRevision. We have added \"Note that there are two situations where\n: the first is , indicating that the event is almost impo\n𝑃\nss𝜋i\n(\n𝑿\nb\n)\n=le\nt\n∥\n𝛻\n𝜋\n𝑃\n(\n𝜔\n(\n𝑿\n)\n=\n1\n)\n∥2\n=\n0\nssible to happen; the second is , indicating that the event provides almost no information. Therefore, we\n𝑃\n𝜋\n(\n𝑿\n)\n=\n0\nexclude the case where as it is useless for learning the key\ninformation\" (see lines 14\n𝑃\n𝜋\n(\n𝑿\n)\n∥\n𝛻\n𝜋\n𝑃\n(\n𝜔\n(\n𝑿\n)\n=\n1\n)\n∥2\n=\n0\n5-1\n4\n8\n) in Supplementary Materials).\nComment 5. The code for the evaluation against provably safe methods is most likely incorrect as\ndetailed in my response above.\nResponse. We argue that our implementation of the provably safe method is correct and represents\nthe SOTA safety performance of it. See our response to Comment 1c for more details."}]

Decision letter