Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles
Authors: Alec Farid
Sushant Veer
Boris Ivanovic
Karen Leung
Marco Pavone
Venue: CoRL 2022 Poster (2022)
Keywords: Run-time Monitoring
Autonomous Vehicles
Trajectory Prediction

Abstract: In modern autonomy stacks, prediction modules are paramount to planning motions in the presence of other mobile agents. However, failures in prediction modules can mislead the downstream planner into making unsafe decisions. Indeed, the high uncertainty inherent to the task of trajectory forecasting ensures that such mispredictions occur frequently. Motivated by the need to improve safety of autonomous vehicles without compromising on their performance, we develop a probabilistic run-time monitor that detects when a "harmful" prediction failure occurs, i.e., a task-relevant failure detector. We achieve this by propagating trajectory prediction errors to the planning cost to reason about their impact on the AV. Furthermore, our detector comes equipped with performance measures on the false-positive and the false-negative rate and allows for data-free calibration. In our experiments we compared our detector with various others and found that our detector has the highest area under the receiver operator characteristic curve.

Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles
Alec Farid",t.. Sushant Veer* 22 Boris Ivanovic", Karen Leung?., Marco Pavone Princeton University, 'NVIDIA Research, Universitty of Washington, *Stanford University afarid@princeton.co edu, {sveer, bivanovic, kaleung mpavone @nvidia. com
Abstract: In modern autonomy stacks, prediction modules are paramount to plan- ning motions in the presence of other mobile agents. However, failures in predic- tion modules can mislead the downstream planner into making unsafe decisions. Indeed, the high uncertainty inherent to the task of trajectory forecasting ensures that such mispredictions occur frequently. Motivated by the need to improve safety of autonomous vehicles without compromising on their performance, we develop probabilistic run-time monitor that detects when "harmful" prediction failure occurs, i.e., a task-relevant failure detector. We achieve this by propagating trajectory prediction errors to the planning cost to reason about their impact the AV. Furthermore, our detector comes equipped with performance measures on the false- silse--osiiive and the false-negatio rate and allows for data-free calibration. In our experiments we compared our detector with various others and found that our detector has the highest area under the receiver operator characteristic curve.
Keywords: Run-time Monitoring, Autonomous Vehicles, Trajectory Prediction
Introduction
In general, autonomous vehicle (AV) stacks forecast the trajectories that non-ego agents- as other vehicles, pedestrians, bicyclists, ate-are likely to take. The accuracy of these predictions crucial for the motion planner to ensure that the AV does not find itself in an unsafe state. Un- fortunately, trajectory forecasting is an endeavor wrought with a high degree of uncertainty, large portion of which, arguably, is alcatoric, arising from the stochasticity of human behavior. The situa- tion is further exacerbated by the epistemic uncertainty that arises from unavoidable learning errors when training
networks from finite driving logs. Therefore, it should come as no surprise that the trajectory-prddiction networks frequently mispredict, leading to misguided motion plans. How then, in the face of such uncertainty, do we ensure safety of the AV without compromising its functionality? We approach this challenge by developing lask-relevant failure detector for the prediction module that only triggers if the estimate of the realized cost at a given time-instare is significantly worse than the cost predictions based on trajectory forecasting.
We require our detector embody the following desiderata: First, the detector should be task rel- evant to reduce the number of uneecsssrr interventions that hurt the AV's functionality, i.c., the detector should only trigger if the failure is harmful to the AV's safety. To better illustrate task- relevance, consider the scenarios in Fig. 1. If the non-ego vehicle is predicted to stay in its lane, but it turns right, into the AV's lane (Fig. 1(a)), the cost for the planner would increase and therefore, the task-relevant detector should trigger. However, if the car turns left, away from the AV's lane (Fig. 1(b),, the cost for the planner would decrease, hence, the detector should not trigger. Note that in both the scenarios in Fig. 1, the predictions are incorrect, but they are task-relevant only in Fig. 1(a). Second, the detector should be an independent module, i.e., it should not modify the prediction module that it monitors. This enables easier integration of the detector with existing AV stacks and also facilitates the prediction module to specialize in its task without having to forego performance in favor of detectability of failures. Finally, the detector should be accompanied with metric to gauge its performance quality to facilitate interpretable and trustworthy decision making. Most prior work satisfies only subset of these desiderata [1, 2, 4]. In this paper we propose an ap-
Equal Contribution
I Work done as an intern with the NVIDIA Research Autonomous Vehicle Research Group
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Actusl
Prediction
Actual
(a) Task-relevant prediction failure
(h) Non-task-relevant prediction failure
Figure I: Illustration task-relevant failure detection. The ego-vehicle and its motion plan green, the non-ego vehicle and the trajectory it takes is in black while the predicted trajectory distribution is in light-grey.
proach that satisfies all these three desiderata by reasoning about the effect of erroneous predictions further downstream at the planner module of the AV stack.
Statement of Contributions. The main contributions of this paper are follows: (i) We formalize notion of anomaly that underpins the design of a task-relevant prediction failure. (ii We develop a fast and efficient task-relevani failure detector for the prediction module used in AV stacks which is simultaneously accompanied by guarantees on the false-positive and false-negative detection rates for improved interpretability. (iii) Owing to the interpretability of our approach, we can perform data-free calibration for our detector. Furthermore, we show in our experiments that our data-free calibrated detector's performance is not very far from the optimally calibrated detector using the efficacy our approach on
Receiver Operator Characteristic (ROC) curve. (iv) We demonstrate nuScenes [5] and nuPlan teaser [6] datasets and provide comparisons with other detection methods in the literature. We find that our detector has the highest Area Under the ROC (AUROC) curve.
Related Work
AV Safety, Since autonomous vehicles require interactions with other road and safety- critical by nature, there has been lot of effort, from regulators, industry, and researchers alike, in ensuring safe AV operations [7. 8, 10, 11, 12, 13, 14, 15, 16] (see [17] for a review). A common approach is to compute "inevitable" collision sets (ICS), c.g., via Hamilton-Jacol reachability com- putation [18], with selected assumptions on other agents' behaviors, and perform shielding where the AV will flag situation as unsafe whenever it is close to entering the ICS and execute appropriate evasive action (e.g.. [1, 19, 20]). A primary challenge selecting reasonable behavior assumptions for ICS computation to balance tractability, interpretability, and compatibility with real-word driv- ing interactions [21, 22]. Most these approaches assume that other agents act in an adversarial manner leading to overly conservative ICS. Instead, in this work, we build run-time monitor that identifies only those prediction failures which are safety-critich detection approach present could also be used as a run-time task-relevant safety monitor.
nature. Furthermore, the general
Anomaly Detection. Anomaly detection or change detection in signal processing literature has been an active area of work for decades (see [23] for review). The goal is to identify events which deviate significantly from a reference input. Such methods have been applied in supervised learning settings to gigher-dimension
inputs such as images [24, 25, 26] (see also [27] for review); how- ever, these methods are not relk-relevant in nature. Other methods perform detection by estimating agent [28, 29] or input [30] atypicality. Recent techniques also allow for detection using the reward reinforcement learning settings using streams sensor inputs [4, 31, 32, 33, 34]. However, these methods do not provide guarantees on the false-positive and false-negative detection rates. Some methods that can provide guarantees come with caveats such as specific architectures [2, 35], only providing guarantees in retrospect (i.e. after multiple anomalies have occurred) [3], on test data that is statistically similar to the training data [36]. We develop an online anomaly detector which provides guarantees for both false positive rate and false negative rate. Unlike many of the works described above, our detector does not require modifying the existing module on which operates.
Planning-aware Prediction. The vast majority of behavior prediction works focus solely on the task of prediction (improving forecasting accuracy, incorporating additional sources of scene con- text, accelerating runtime, etc), in isolation from other components of the autonomy stack [37]. In recent years, however, there has been increasing interest in better integrating prediction with the rest of the robotic autonomy stack. Examples include works that more tightly integrate prediction with

---

object classification [38], tracking [39, 40], and planning [41, 42, 43, 44, 45, 46, 47]. While these approaches focus on better architectural integrations across the stack, a few recent works have also proposed to integrate modules metrically. For instance, [48, 49] present metrics that evaluate per- ception and prediction modules in a planning-aware fashion, e.g., by heavily weighting detection or prediction errors that could cause collision. Our work is similar in that it can identify anomalous predictions which would detrimentally affect ego-vehicle planning, however, does so online with guaranteed false-positive and false-negative detection rates.
3 Problem Definition
We express the dynamics of the AV and the scene around it as sion Process (POMDP) defined as a tuple (X, O,U. ff where X' is the state space which composed of the state of the ego agent x², non-ego agents ze and other variables such as the environment map nhh O is the space observations that the ego agent receives from its sensors, is the space of the control inputs for the ego agent, and [ Me-1) is the stochastic state-transition function.
Markov Deci-
- [0, 20) be cost function at a time instant t. Our framework is agnostic to the choice of the cost function; we only assume that the cost function is formulated such that higher val- ues imply worse / less safe outcomes for the AV. As an example cost function is the closest distance of an AV to other non-ego agents and obstacles in the map. Given the predicted state êttr and the control input u at f time steps in the future, the predicted cost is c := c M(++).
Let C X'
With a slight abuse of notation, let I, E X' be an estimate of the state (instead of the true state) at time l provided by the perception module by processing the sensor observation Ot- The prediction module takes in a history of state estimates Tor and provides a distribution vf +=: ( on the predicted future state trajectories iime "1+1:4T of non-ego agents over the horizon T. The distribution produced by the prediction module induces sequence of distributions (C++-((++ the predicted cost cc+-r for each time-step in the prediction horizon T. The observed cost c := C++ (Wetr, ut++)) at time-step L + T is the cost estimated or "observed" by the AV given state- estimates of the ego and non-ego agents T and the control input u++7. We define the notion of anomaly that will underpin our approach to prediction failure detection:
on
Definition 1 (p-quantile anomaly). For any time-step t + T, let (cc4r//// be the distribution on the predicted cost as described above. If the observed cost C++r lies in the top p-quantile of (e+-(Ce+7 ) then the prediction module suffers from a p-quantile anomaly.
The observed cost cZ is. theoretically speaking, not necessarily drawn from the predicted cost dis- tribution 61t, since the observed cost might depend on uumodelee//noossrrabbl factors. However, under nominal circumstances (i.e., few no task-relevant prediction errors and major deviations by the AV from the open-loop motion plan), it is reasonable to expect the observed cost to be similar to the sampled predicted costs. This motivates the definition of anomaly per Def. and the design of a prediction failure detector based on the concept of p-quantile anomalies (illustrated in Fig. 2). Importantly, the notion of p-quantile anomaly, being a cost-based quantity, intrinsically captures the notion of task-relevance. Note that if the predictor predicts cost-degrading behavior correctly, the observed cost would not lie in top p-quantile as this is in alignment with the predictions.
Area
Φe+1 (Ge+1 )
No anomaly Annomly
Φit (6++4/70a)
No anomaly! Anomaly
Figure 2: Illustration of the concept of y-guantile anomalies. The ego vehicle is red, non-ego vehicles are blue, ground truth trajectories are white, and trajectory predictions are the purple contour plots. each time step, the cost function induces a distribution Φ14. on predicted costs depicted by the blue plots on the right. If the observed cost cf+ is within the top-p quantile of detr. it is p p-quantile anomaly.
3

---

p-Quantile Anomaly Detector
In this section, we present algorithm for detecting p-quantile anomalies along with guaranteed false-positive and false-negative detection rates afforded by our algorithm. Let c (p) be the cost which separates the top p proportion of (1+r(C++71 [20:1) from the bottom 1 - p. If c 2 (p) from then we have p-quantile anomaly. The lack of analytical expression' for prevents exactly computing (p), hence, we use a campling-based approach for detecting if the observed cost lies within the p-quantile of the distribution the predicted cost; Fig. 2 for an illustration of the y-quantile anomaly within the trajectory prediction framework.
Suppose time L + we sample M predicted costs Si+r (c+ ((-)) independent and identically distributed (i.id.) from (i++((i++ (To:e) and arrange them in an increasing order (i.e., CL+T S e7+ S < 2 Our detector D returns positive detection if the observed cost is greater than at least M - n predicted costs in S. where n E {0,- ... .M 1}:
AM
D(n, S) = G+T G+7
(I)
Intuitively, by checking where the observed cost ranks against the sampled predicted costs, detector estimates the quantile of Φ in which the observed cost lies, In the rest this section we present our detection algorithm, provide bounds on it's false-positive and false-negative detection rates, and use these bounds to analytically calibrate the detector by choosing appropriate M and n.
4.1 Detection Algorithm
The detector, Algorithm 1, is executed at each planning cycle. We sample M trajectories for each non-ego agent from the trajectory-prediction network v. Inference from prediction networks is usually bottleneck in real-time control; however, note that for most AV stacks, we do not need to sample these trajectories specially for detection, as they would be drawn regardless for use in planning. Hence, by reusing these samples our detector does not generate significant computational overhead. each planning cycle, our detector receives M predicted costs for each non-ego agent at each prediction time-step (see line in Algorithm 1). For each subsequent time-step in the prediction horizon, we monitor identifier of the non-ego agent responsible for the detection. In Sec. 5.1, we show that this detector runs in less than 0.1 milliseconds (ms) which sufficiently fast for real-time monitoring.
(1). If the detection criteria is met, the detector returns True along with the
Algorithm p-Quantile Anomaly Detector (QAD)
1: Input: For each non-ego agent, sequence of sets Si+r (Cttr.... ..M4 for each (1.... ,T. Input: Cost functions Cr4-r for each T -1, ,T}
3: for wall-clock time w t 1 + do
4: for cach non-ego agent do
5: cpp (Ir. Nw:)
6: if ct 2Md-n then
7:
return (True, non-ego agent Id)
8: end if
end for
10:
end for
4.2 Bounds False-positive and False-negative Rates
Let us first define the false-positive rate (FPR) and the false-negative rate (FNR) for D. Definition Ptalse-positive Rate), FPR the detector is defined as the probability of drawing Stir such that p-quantile anomaly detection occurs when the anomaly does not occur, iee,
FPR
P NoM
(n,n, Setr) A citr cc+((p)
P
1ct
2M- A Cet c(t(())
(2)
S
'vven if the distribution on the predicted trajectories & takes the form of a Gaussian mixture model [50]. the nonlinear cost function may prevent the predicted cost distribution ó from acquiring analytical form.

---

Definition 3 (False Degaseee Rate), FNR of the detector is defined as the probability of drawing Si+7 such that a p-quantile anomaly detection does not occur when the anomaly does occur, i.e.,
FNR == 114
(n, S(4+) A GeT 2 (+r(
P
lott < CctT A C2ts 2 G [p]] (3)
Ideally, for safety-critica systems, such AVs, desire low FNR to ensure that do not miss any unsafe scenarios. This can be trivially achieved by having a detector that triggers all the time (high FPR), but clearly that is not desireable for the AV's performance. Hence, good detector must also exhibit a low FPR. Motivated by this, we provide bounds on the FPR and FNR for our detector.
Theorem (FPR and FNR Bound). Let the detector D for the occurrence of p-quantile anomaly be defined as (1). Then, the FPR (Def. 2) and the FNR (Def. 3) for the detector satisfies:
FPR &
Σ Ý=0
M p0(1 (11-p) -) =: FNR S (
Σ 1=
M) FNR (1
M D (M)
The proof of this theorem is presented in App. A. Thm. ensures that the detector identifies an anomaly (i.e., D holds true), then with probability at least F PR, the observed cost does lie in the top-p quantile of Φe+T. while if our detector does not detect an anomaly (i.e., is not true), then with probability at least FNR, the observed cost does not lie in the top-p quantile of ØL+T-
Our detector satisfies all the three desiderata laid out in Sec. 1. First, detector encodes the notion of task-relevant failure detection since only triggers when the observed cost high compared to the predicted costs, in the sense of the p-quantile anomaly (Def. 1). Second, the detector is independent of the trajectory prediction module that it monitors. Finally, we provide bounds on FPR and FNR of our detector in Thm. 1 that serve a metric on the detector's performance. These bounds facilitate greater interpretability and promote easier detector calibration.
4.3 Data-free Detector Calibration
The analytical upper bounds for the FPR and the FNR in Thm. 1 allows data-free calibration of D by letting us choose M and so as allow a user to specify FPR and FNR values. However, we can only provide a strong bound on either the FPR or the FNR, but not both. This follows from the observation that the upper bounds in Thm. satisfy FPR + FNR It worth clarifying that this is not a fundamental limitation of our detector, but that of our analysis. We show in Sec. 5 that our detector, fact, achieves low FPR and low FNR simultaneously.
5 Experimental Results
In this section, we benchmark the performance of our detector on nuScenes [5] and nuPlan teaser [6] datasets against various other methods. We study the detector in two applications: (i) run-time monitoring of task-relevant prediction network failures and (ii) filtering interesting scenarios from unlabelled expert driving logs. Through these experiments we demonstrate the ability of our detector to provide low FPR and low FNR for detecting task-relevant mispredictions. All experiments were conducted on a desktop computer with an Intel 10-10EE CPU (18 cores) and 64 GB of RAM. Our code is available at: https://githe
com Boom//Wiaab//pred-fee fai-dtttttt.-ddtttttrrr... -ettector.
Trajectory prediction module. While our method is agnostic to the particular prediction method used, in our experiments we employ Trajectron++ [50], state-of-the-ar multi-agent trajectory fore- graph-structured recurrent neural network that predicts an agent's 's future position distribution given its past trajectory history and the past trajectories of its neighboring agents. At its core, Trajectron++ uses a conditional variational autoencoder (CVAE) model the potential for multiple future trajectorics.
casting model, to predict the motion of non-ego agents.
Detection and comparison methods. We compare our peoeetorr--qquatiee anomaly detection BaaD)sagainst four other approaches: (i) likelihood detection baseline which triggers when the likelihood of the achieved states of an agent in the predicted GMM below threshold; (ii) uniform and partial cost degradation tests [4] which uses p-values to detect shifts in the cost distribution; (iii) time-to-collision (TTC) detector which triggers when the TTC drops below a threshold; and (iv) detection based on Hamilton-Jccoii (HJ) reuchability analysis [1] which triggers if collision is possible assuming an adversarial behavior by the non-ego agent. See Sec. B. for more information about the comparison methods.
5

---

5.1 Task-relevant Detection
Overview. We study the use of our detector and other baseline detectors mentioned above as run- time monitors for task-relevant prediction failure. The experiments are conducted in the recently released nuPlan teaser dataset [6] which allows the inclusion of reactive agents in the driving logs.
Planner. We assume a behavior generator provides us with a reference trajectory to follow. Our planner is then tasked with following this trajectory while avoiding collision with other agents on the road, ensuring ego comfort and satisfying the ego's 's dynamical constraints. The planner achieves this by generating motion primitive tree [51] and choosing the trajectory that minimizes the weighted sum of a series of cost functions; further details on the planner are supplied in App. B.2. We use time-to-collisoon cost One (computed assuming that agents continue moving at a constant speed in their current heading), momentum shapeed distance to other agents cost Ca2a. distance to goal cost Cazg, distance the reference trajectory cost caz speed limit cost Cvelocity ego comfort cost Coomton and reversing cost Creverse which penalizes the car for going against the heading direction of the lane:
Ct+- Cie + C22a + W3Cd2g + W4Ca2r WcCvelocity + W6Coomfort + WyCreverse
(4)
The exact expressions for the cost terms and the weights w1, 47 used in the experiments are provided in App. B.3. The motion planner uses trajectory forecasts of other non-ego agents drawn from Trajectron++ to compute the momentum-shaped distance and the time-to-collison cost. The cost functions take about = 0.5 ms to evaluate. Note that these cost evaluations for planning are reused by our anomaly detector in Algorithm 1, thereby minimizing any excessive computational overhead by our detector. The anomaly detection computation takes less than 0.1 ms.
Dataset and Labeling. We test on 260 nuPlan [6] driving scenarios which comprises of total of 2568 prediction horizons. We hand-label each prediction horizon for task-relevant prediction errors by visually inspecting plots of true trajectories of agents along with their predicted trajectories. We emphasize that the labeling done merely by visual inspection and does not use the planning costs or prediction likelihoods. This is to ensure that the labeling procedure is independent of the detection methods promote fair comparisons; see App. B.4 for more details on the labeling procedure.
Remark 1 (Imbalance in Dataset). Despite the use of reactive agents along with custom planner in nuPlan, there is a strong bias in the dataset towards benign scenarios. Only 3.5% of this dataset is labeled as positive. Due the imbalance, there a bias toward detectors exhibit lower FPR and higher FNR, since false-negative errors are amplified while the false-positive errors are mitigated.
Results. We plot the Receiver Operator Characteristic (ROC) curve in Fig. 3 for each detector by varying: the detection rank M - n in (1) for our detector, the p-value for UDT and PDT, the likelihood threshold for the likelihood-basee sub-level set of the value function for the HJ reachability based detector. The ROC curve for our detector the closest to an ideal detector and has the highest Area Under the ROC (AUROC) curve.
detector, the TTC threshold for TTC detector, and the
1.0
Table 1: AUROC and best point from ROC (f AUROC)
Detection Method
AUROC
FPR %
FNR %
0.8
QAD (ours)
0.946110.002
8.4.10.4
7.910077
PDT [4]
0.932-10.005
18.5::0.2
1.8
UDT [4]
0.912-10.002
10.6410.3
11.540.44
0.6
TTC
0,864+0.000
17.3+0.5
13.7#0.4
TPR
HJ Reachabilityy [1]
0,862 000
23.7+0.3
7.210.6
Likelihood
0.693 006 45.9+0.4 24.5+:0.4
QAD (ours)
Table 2: Data-free calibration (1 distance from
0.2
Likelihood
Detection Method
FPR
FNR
Dist.
TTC
HJ Reachability
QAD (FNR) (ours)
4.2-10.33
13.9
0..01
0.0
FNR Ensemble (ours)
15.2.+0.3
8.74..2
0.54
0.0
0.2
0.4
0.6
0.8
1.0
PDT (FPR) [4]
3.7+0.3
26.3+0.66
0.50=
FPR
Figure 3: ROC curve for detectors. We report the mean of 5 trials, the shaded region 2 times the standard deviation.
QAD (FPR) (ours)
3.4:1..3
28.810.9
0.48=00.1
UDT (FPR) [4]
3.5+-0.3
29.910.9
0.47=0.01
HJ Reachability [1]
11.8+0022
48.6+1.7
0.28+
FPR Ensemble (ours)
1.5++0.3
61.6+1.8
0.26+00011

---

We obtain the best calibration for each detector by choosing the point that is farthest from the line in Fig. 3 and report the empirical FPR and FNR for them in Table Our detector has the lowest FPR and the third-lowest FNR. PDT has slightly better FNR than our method, but the FPR is much higher, while for all other methods both FPR and FNR are higher than our method. As one would expect, the naive likelihood baseline does not perform very well while the HJ reachability detector is overly conservative exhibiting a lot of false-positives.
We also compare the performance of our detector with data-free calibration using Thm. I with the data-free calibration of UDT, PDT, and HJ reachability. For our detector, we consider an anomaly to have occurred when the true cost lies in the top-5% of the predicted cost distribution (see Def. 1); we chose the 5% quantile because it is a commonly used threshold for detection in statistics. We include two versions of our detector in this study: (i) calibrated to bound the FPR below 5% and (ii) calibrated to bound the FNR below 5% using Thm. I; as discussed in Sec. 4.3, we cannot simultaneously obtain a strong bound on both FPR and FNR. We use a p-value of 0.05 for UDT and PDT, and a O-sublevel set of the value function for HJ reachability. Additionally, we construct an FPR ensemble detector, which only detects positive if all of QAD (FPR), UDT, PDT, and HJ reachability are positive, and an FNR ensemble detector, which detects positive any of QAD (FNR), UDT, PDT, and HJ reachability are positive. For the ensemble detectors, we use the data- free calibration for each method described above.
These detectors are labeled using markers in the ROC curves in Fig. 3. Their empirical FPR and FNR is reported in Table and we sort the table by distance from the line y = z. We observe that our FPR bounded detector satisfies the 5% bound provided by Thm. I; however, our FNR bounded detector violates the bound. This disagreement between the theory and the empirical performance for FNR occurs because of the mismatch between our anomaly criteria of top-5% quantile and what the human data labeller considers to be an anomaly. A higher empirical FNR suggests that the anomalous quantile should be expanded beyond 5%; case in point, the optimal p-quantile anomaly detector obtained using the ROC curve in Table with 7% FNR corresponds to approximately the top-35% quantile as an anomaly. Furthermore, the imbalance in the dataset also contributes to this problem, as discussed in Rem. 1. More data as well as a more balanced dataset can perhaps remedy this mismatch and lower the FNR; we will explore this in future work. Despite this mismatch, see in Table 2 that our FNR detector performs the best out of all detectors in terms of distance to the y line in the ROC plot. The FNR ensemble and PDT also perform well, favoring false-negatives and positevosiiveeeee respectively. Since false-negative can be safety critical, a designer may choose our FNR ensemble detector at the cost of more calse-positives, Ultimately, our FNR detector is not very far off from the optimal detector obtained from the ROC curve, i.e., with no data at all we are able to calibrate our detector using Thm. to perform very well.
5.2 Filtering expert driving logs
Overview. Given driving data logs, we consider the task of filtering scenarios with anomalies that may be of interest for robsutifying the prediction network or for diagnostic evaluations. We use the nuScenes and nuPlan datasets as the driving logs. For the likelihood detector we choose the threshold 0.05, and for TTC we choose 1 second, while for all other detectors, we use the data-free calibration specified in Sec. 5.1.
Dataset. The dataset consists of 375 nuPlan and 152 nuScenes scenarios with a total of 3890 predic- tion horizons, in all of which the ego-vehicle is driven by a human driver and there are no collisions.
Cost functions. Since the nuScenes and nuPlan driving logs are collected by human driver, we do not have access to the true planning "cost" used by the driver. Instead, we use a weighted sum of cost terms contributed by the momentum-shee shapedd distance and the tine-to-collision for each non-ego agent as a proxy cost: Ct+, = Wj Gne + W2Cala The exact expressions of Oue and caza are provided in App. B.3. Our results suggest that this cost function is well-aligned with human intuition as it allows us to filter out interesting scenarios while ignoring the uninteresting ones.
Results. We compare our detector to all baseline methods described earlier. See Table 3 for a com- parison of the methods and Fig. 4 for representative examples of positive detections. We observe that the likelihood detection and the TTC-based detection both have many more positives than the other methods, followed by HJ reachability. Other methods have a similar number of positives. Ultimately, no driving scenario from this dataset is unsafe, so we inspect the positive detections
7

---

(a)
(c)
Figure 4: Selected scenarios from nuScenes. The ego vehicle is red, non-ego vehicles are blue, pedestrians are green circles, ground truth trajectories are white, and Trajectron++ predictions are the purple contour plots.
from each method to find interesting scenarios. All methods detect a handful of interesting scenar- ios, such as Fig 4(b) where pedestrians are right beside the ego (possibly entering or exiting). We
also see additional interesting scenarios detected by our method but not HJ reachability such in
Table 3: Detections from expert driving logs
Fig. 4(a) where a close-by pedestrian is predicted to stand still, but moves quickly towards the ego at crosswalk. Qualitatively, it seems that our detector, UDT, and PDT all do good job of identifying inter- esting scenarios such as those shown in Fig. 4(a) and (b). We see that HJ reachability, and likelihood de- tection methods more frequently labels driving sce- narios where the predictions accurate and there is nothing inherently unsafe as anomalous such as in Fig. 4(c) and (d).
Detection Method
Percentage Detected
UDT [4]
5.4 + 0.1
QAD (FPR) (ours)
5.4 0.1
PDT [4]
5.8
QAD (FNR) (ours)
6.4
HJ Reachability [1]
8.8
TTC Likelihond
17.4
58.5 0.0
6 Limitations, Future Work, and Conclusion
Limitations. As discussed in Sec. 4.3, we cannot provide strong FPR and FNR guarantee si- multaneously for our detector. This does not seem to be a fundamental limitation of the detector itself because our empirical results in Table 1 suggest that can achieve low FPR and low FNR. As part of our future work, we intend to explore providing stronger guarantees on both FPR and FNR simultaneously. A second limitation of our approach is identifying what quantile of the cost distribution corresponds to an actual anomaly. As we saw in Sec. 5.1, the mismatch in the quantile result in disagreement with the theoretical bounds. From practical perspective, this is not major limitation as we demonstrated that our detector performs well despite making a completely uninformed quantile choice for anomaly detection. Furthermore, we can mitigate this problem by using a small dataset to identify an appropriate p-quantile to be considered an anomaly.
Future Work. As part of future work, hope to use this detector for adapting planning frequencies. If the planner re- re-plans only when necessary. it would free up some of the compute budget for other modules in the AV stack and lower the energy consumption. Since our detector directly monitors the cost of the planner, it would serve as a good indicator of when the motion plan is not evolving expected and therefore, requires re-planning. As proof of concept, present some motivating results in App. B.6. An additional direction for future work is to require multiple p-quantile anomalies before declaring detection instead of single Another exciting direction for this work is providing causal explanations for the detections; e.g., we anticipated the agent continue straight, but switching into our lane. Such explanations can actively be leveraged to plan reactions to prediction failures.
used in this work.
Conclusion. We presented a task-relevant detector for mispredictions by the trajectory forecasting networks used in AVs. Our detector builds a distribution of the predicted future costs and identi- fies anomaly when the true cost of the planner lies in the top-p quantile of that distribution. We provided FPR and FNR guarantees for our detector that facilitate data-free calibration. In experi- ments, we demonstrated that detector out-performs all other baseline approaches we compared against and is compute-efficient. Additionally, we showed that our data-free calibrated detector's performance is close the optimal detector chosen from the ROC curve. We used our detector for run-time misprediction monitoring and for filtering prediction failures from unlabeled driving logs.

---

Acknowledgments
The authors grateful to Anirudha Majumdar for helpful feedback and discussions on this work.
References
[1] K. Leung, E. Schmerling, M. Zhang, M. Chen, J. Talbot, J. C. Gerdes, and M. Pavone. On infusing reachability-basel safety assurance within planning frameworks for human-robot ve- hicle interactions. Int. Journal of Robotics Research, 19:1326-1344 2020.
[2] S. Liu, R. Garrepalli, T. G. Dietterich, A. Fem, and D. Hendrycks. Open category detection with pac guarantees. Proceedings of the International Conference Machine Learning. 2018.
[3] A. Farid, S. Veer, and A. Majumdar. Task-driven out-oo-distribution detection with statistical guarantees for robot learning. Proceedings of the Conference on Robot Learning, 2021.
[4] I. Greenberg and S. Mannor. Detecting rewards deterioration in episodic reinforcement learn- ing. In International Conference on Machine Learning, volume 139, pages 3842-3853, 2021.
[5] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuScenes: multimodal dataset for autonomous driving. In Conference on Computer Vision and Pattern Recognition, 2020.
[6] H. Caesar, Kabzan, K. S. Tan, W. K. Fong, E. Wolff, A. Lang, L.. Fletcher, O. Beijbom, and Omari. nuPlan: closed-loop ml-based planning benchmark for autonomous vehicles. In Conference on Computer Vision and Pattern Recognition ADP3 workshop, 2021.
[7] IEEE Standard for Assumptions in Safety-Relater Models for Automated Driving Systems. IEEE Std 2846-20241 2022.
[8] D. Nistér, H. Lee, J. Ng, and Y. Wang. An introduction to the safety force field. avidia. com///onteett///a//aaae/////ooet wtteiioneeeeeeeieiiiieeeeeeeeeee self-drivvnng-crss https://www. hwtp:://wwwwniiiaaaaaaaa.
pdf, 2019.
[9] S. Shalev-Shwartz, S. Shammah, and A. Shashua. On formal model of safe and scalable self-driving cars. arXiv preprint 1708.06374, 2017.
[10] M. Althoff and M. Dolan. Online verification of automated road vehicles using reachability analysis. 30(4):903-918, 88, 2014.
[11] R. Kianfar, P. Falcone, and Fredriksson. Safety verification of automated driving systems. IEEE Intelligent Transportation Systems Magazine, 5(4):73-86 2013.
[12] General Motors. Self-driving safety report, 2018.
Available https://www gm. com/
hatpattt/ddmmmmmm//////aaa ht////s///////eeee///////eeeeeae ggc.g//////oooommommmmo samcon/gnsaffttreport.. wtw.aattoooooooo..ooo.....
[13] Lyft. Self-driving safety report, 2020. Available at wp-
Rttpaarttte__ot22222222 .df.
[14] Zoox. Safety report volume 2021. Available at attttp:://ox..cco//ffff//
2021.
[15] Waymo. Safety report, 2021. Available at https: http:p//aaa..ccc/aaaaa//t/tttttp Retrieved on July 4, 2021.
[16] NVIDIA. Self-driving safety report, 2021. Available at https://mmages.nn nvidia. com/ wttprriingaarr/aaatteeeeeeeeeeee pdf.
[17] J. Dahl, G. Rodrigues de Campos, C. Olsson, and J. Fredriksson. Collision avoidance: A literature review on threat-assessment techniques. IEEE Transactions on Intelligent Vehicles, 4(1):101-113, 2019. 2019.

---

[18] I. M. Mitchell, A. M. Bayen, and J. Tomlin. A time-dependent hamilton jacobi formulation of reachable sets for continuous dynamic games. In IEEE Transactions on automatic control, volume 50, pages 947-957. IEEE, 2005.
[19] K.-C. Hsu, A. Z. Ren, D. P. Nguyen, A. Majumdar, and F. Fisac. Sim-to-lab-to-real: Safe reinforcement learning with shielding and generalization guarantees. In arXiv:2201.08355 [cs.RO), 2022.
[20] H. Hu, K. Nakamura, and J. F. Fisac. SHARP: Shielding-aware robust planning for safe and efficient human-robot interaction. IEEE Robotics and Automation Letters, 7(2):5591-5598, 2022.
[21] A. Bajesy, K. Leung, E. Schmerling, and M. Pavone. Towards the unification and data-driven synthesis of autonomous vehicle safety concepts. In arXiv preprint arXiv:2107.14412, 2021.
[22] R. Tian, L. Sun, A. Bajesy, M. Tomizuka, and A. D. Dragan. Safety assurances for human- robot interaction via confidence-ann awaree game-theorette human models. In Proc. IEEE Conf. on Robotics and Aulomation, 2022.
[23] M. Basseville. Detecting changes in signals and systems a survey, In Automatica, volume 24, pages 309-326, 1988.
[24] D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the International Conference on Learning Representations, 2017.
[25] K. Lee, K. Lee, H. Lee, and J. Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, 2018.
[26] D. Hendrycks, S. Basart, M. Mazeika, M. Mostajabi, J. Steinhardt, and D. Song. Scaling out-o--distribution detection for real-world settings. arXiv preprint 1r1.v: 1/12,, 2020.
[27] I.. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller. A unifying review of deep and shallow anomaly detection. In Proceedings of the IEEE, volume 109, pages 756-795, 2021.
[28] A. Bera, S. Kim, and D. Manocha. Realtime anomaly detection using trajectory-level crowd behavior learning. In IEEE Conference on Computer Vision and Pattern Recognition Work- shops (CVPRW), pages 1289-1296, 2016.
[29] J. Wiederer, A. Bouazizi, M. Troina, U. Kressel, and V. Belagiannis. Anomaly detection in multi-agent trajectories for automated driving. In Proceedings of the Conference on Robot Learning, volume 164, pages 1223-1233. PMLR, 2022.
[30] A. Sharma, N. Azizan, and M. Pavone. Sketching curvature for efficient out-of-disti detection for deep neural networks. In Proc. Conf. on Uncertainty in Artificial Intelligence, 2021.
[31] A. Sedlmeier, Т. Gabor, T. Phan, L. Belzner, and C. Linnhoff-Popien, Uncertainty-ban based out- of-distribution classification in deep reinforcement learning. In arXiv:2001.00499 [cs.LG], 2019.
[32] A. Sedlmeier, R. Müller, S. Illium, and C. Linnhoff-Popien. Policy entropy for out-of- distribution classification. In Artificial Neural Networks and Machine Learning ICANN, 2020.
[33] F. Cai and X. Koutsoukos. Real-time out-of daitof-distribution detection in leaming-enabled cyber- physical systems. In Proceedings of the International Conference on Cyber- Systems, 2020.
[34] Wu and Ortiz. RLADD: time series anomaly detection through reinforcement learning and active learning. In arX4v:2144.00543 [cs. 2021.
10

---

[35] M. A. Siddiqui, A. Fern, T. G. Dietterich, and S. Das. Finite sample complexity of rare pattern anomaly detection. Proceedings of the Conference on Uncertainty Artificial Intelligence, 2016.
[36] A. Farid, D. Snyder, A. Ren, and A. Majumdar. Failure Prediction with Statistical Guarantees for Vision-Basse Robot Control. In Proceedings of Robotics: Science and Systems, 2022.
[37] A. Rudenko, L. Palmieri, M. Herman, K. M. Kitani, D. M. Gavrila, and K. O. Arras. Human motion trajectory prediction: A survey. Int. Journal of Robotics Research, 39(8):895-923 2020.
[38] B. Ivanovic, K.-H. Lee, P. Tokmakov, B. Wulfe, R. McAllister, A. Gaidon, and M. Pavone. Heterogeneouu--geent trajectory forecasting incorporating class uncertainty. arXiv preprint vrXiv:214.124440 2021. URL htwph::t///et //arxiv org/ 12446.
[39] X. Weng, Y. Yuan, and K. Kitani. PTP: Parallelized tracking and prediction with graph neural networks and diversity sampling. IEEE Robotics and Automation Letters, 6(3):4640-46444 2021.
[40] X. Weng, B. Ivanovic, and M. Pavone. MTP: Multi-hypothesis tracking and prediction for reduced error propagation. In IEEE Intelligent Vehicles Symposium, 2022.
[41] J. Ziegler, P. Bender, T. Dang, and C. Stiller. Trajectory planning for bertha a local, continuous method. In IEEE Intelligent Vehicles Symposium, 2014.
[42] C. Liu, S. Lee, S. Varnhagen, and H. Tseng. Path planning for autonomous vehicles using model predictive control. In IEEE Intelligent Vehicles Symposium, 2017.
[43] H. Fan, F. Zhu, C. Liu, L.. Zhang, L. Zhuang, D. Li, W. Zhu, J. Hu, H. Li, and Q. Kong. Baidu apollo em motion planner, 2018, Available at https : / /arxivv. wrg. aba/abb//// 11wp.//1....00 08048.
[44] W. Zeng. W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In JEEE Conf. Computer Vision and Pattern Recognition, 2019.
[45] B. Ivanovie, A. Elhafsi, G. Rosman, A. Gaidon, and M. Pavone. MATS: An interpretable trajectory forecasting representation for planning and control. In Conf. on Robot Learning, 2020.
[46] Y. Chen, U. Rosolia, C. Fan, D. Ames, and Murray. Reactive motion planning with probabilistic safety guarantees. In Conference on Robot Learning, 2020.
[47] S. Schaefer, K. Leung. B. Ivanovic, and M. Pavone. Leveraging neural network gradients within trajectory optimization for proactive human-robot interactions. In Proc. IEEE Conf. on Robotics and Automation, 2021.
[48] B. Ivanovic and M. Pavone. Injecting planning-awand awareness into prediction and detection evalu- ation. In IEEE Intelligent Vehicles Symposium, 2022.
[49] R. McAllister, B. Wulfe, J. Mercat, I.. Ellis, S. Levine, and A. Gaidon. Control-aware pre- diction objectives for autonomous driving. In Proc. IEEE Conf. on Robotics and Automation, 2022.
[50] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-feasii trajectory forecasting with heterogeneous data. European Conference on Computer Vision, pages 683-700, 2020.
[51] E. Schmerling, K. Leung, W. Vollprecht, and M. Pavone. Multimodal probabilistic model- based planning for human--obee interaction. In Proc. IEEE Conf. on Robotics and Automation, 2018.
11


Reviews:
[{"review_id": "uT6nweoHuJWX", "reviewer": "Reviewer_t48G", "summary": "This paper studies the problem of anomaly detection for trajectory predictors in autonomous vehicles. A probabilistic framework that detects task relevant failures, by propagating trajectory prediction errors to the planning cost is developed. To be able to distinguish between harmful and non-harmful mispredictions, the authors design of a prediction failure detector based on the concept of p-quantile anomalies. The proposed detector comes equipped with performance measures on the false-positive and the false-negative rate and allows for data-free calibration. To benchmark the performance of the anomaly detector, the authors compared their approach against various other methods on nuScenes and nuPlan datasets .The authors demonstrated that the proposed detector outperforms all other baseline approaches and is compute-efficient.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Identified strengths of the paper:\n     - Design of a prediction failure detector based on the concept of p-quantile anomalies.\n     - FPR and FNR to guarantee that the anomaly detector facilitates the data-free calibration. \n     - The obtained results can, in principle, be reproduced. Even though key resources (e.g., proofs, code, data) are unavailable.\n\nWeakness/ Suggested imporvement of the paper:\n      - The authors use Trajectron++, which is based on a conditional variational autoencoder (CVAE) to model the potential for multiple \n         future trajectories. However, the sampling is based solely on likelihood which may only produce samples that correspond to the \n         major modes of the data distribution. How can you garantee the diversity of the set of samples ?\n      - For the results in Tab.3 it would be useful to provide a measure of statistical dispersion (e.g. standard deviation or standard error), to \n        be able to statistical significance when comparing the the previous baselines\n      -  As a qualitative comparison, it could be useful to show some examples, where the proposed anomaly detector outperforms the \n         previous approaches. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "\nThe problem is interesting and useful. The method is reasonable. The experiments conducted on benchmark are convincing. The few issues pointed out above should be addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "Listed above as suggestions for improvement.\n\n", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "tGUtAVVvzp", "reviewer": "Reviewer_dWwY", "summary": "1. The authors develop a probabilistic run-time failure detector for the prediction module of autonomous driving.\n2. The main characteristic of this detector is \"task-relevant\". Namely, by incorporating the planning cost in the detector, the detector is designed to detect false predictions leading to the planning cost deviation. \n3. The authors provide theoretical proof for the FPR, FNR bounds of the proposed detector.\n4. The proposed detector is validated in two datasets and compared to several baseline methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The idea of incorporating planning costs to measure task relevance is good.\n\n2. The problem of detecting prediction failures is essential.\n\n3. The paper is easy to understand and well written.\n\n### Weaknesses\n\n1. Although it claimed that \"our framework is agnostic to the choice of the cost function\", the detector may be **sensitive to the cost function design**. Eq. (1) essentially evaluates how the estimated cost (cost evaluated from state estimated) is ranked among the predicted costs (costs evaluated from the M sampled predicted trajectories). M & n are the hyperparameters for this detector. If the cost function is tuned a little bit, the predicted cost distribution may be completely different.  Take the cut-in case in Fig.1 (a) for example, the planning cost can be either tuned to have a large safety margin or a small one (actually this value may be  adjusted by the user on the fly), to obtain a reasonable balance between FPR and FNR, the M & n may need to be tuned. Should we tune individual detectors for every cost function? ( it may not be realistic). Or one universal setup of M & n will work for all cost function designs?  (the answer may be no since the cost distribution is different and something should be tuned for better FNR and FPR balance) *This may be an issue preventing the detector from working in a real autonomous driving system since the cost function is complicated and tuned very frequently.*\n\n2. For complicated driving scenarios, **how to choose M?** Should we fix the M for all scenarios? For example, for a number of non-ego interactive agents in an intersection (every agent may have multimodal predictions), a very large M may be needed to yield a good estimation of the predicted costs. Will a very large M also work for normal on-lane driving?\n\n3. How the probability of the predicted trajectory will be incorporated?\n\n4. The metrics for evaluation are not strong. FNR/FPR/ROC only represents the precision-recall & accuracy of the detector.  What is the consequence of an FN or FP event? What does it mean by 8% FP or FN rates? Should we abandon the prediction results when a failure is detected (or at what FP level we should abandon the prediction results)?\n\n5.  The failure detection is done by post-validating predicted planning costs against true costs. The detection is always delayed compared to the first frame of abnormal prediction. **What is the average delay for the detector** (for different M & n & FPR & FNR) ? What if there is only one frame of abnormal prediction and then the prediction is automatically recovered?  When we find one positive, should we regard it as a failure? (when the detector finds there is a failure, the prediction may be already normal).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "1. The idea of task-relevant failure detection is good.\n2. The current design of the detector is too simple to work in real autonomous driving vehicles, as listed in the weaknesses.\n3. More thorough evaluation should be presented.\n4. Still a gap to a solid contribution. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "1. Justify whether the detector is sensitive to the cost function design. Try different cost function (with different terms and weights) and show whether a global M & n setup will work.\n\n2. Justify how to choose M? Try complicated scenarios (multiple non-ego agents at intersection) & visualize the predicted cost distribution estimated from M predicted trajectories.\n\n3. How the probability of the predicted trajectory will be incorporated?\n\n4. Evaluate the consequence of an FN or FP event. What does it mean by 8% FP or FN rates? Should we abandon the prediction results when a failure is detected (or at what FP level we should abandon the prediction results)?\n\n5. Justify the delay of the detection?  **What is the average delay for the detector** (for different M & n & FPR & FNR) ? What if there is only one frame of abnormal prediction and then the prediction is automatically recovered?  When we find one positive, should we regard it as a failure? (when the detector finds there is a failure, the prediction may be already normal).\n"}}, {"review_id": "3kYeEvcaBtN", "reviewer": "Reviewer_nVyF", "summary": "In the paper a task-relevant failure detector for monitoring trajectory prediction is proposed. The novelty is, that prediction errors are detected by evaluating the effect of the prediction on the subsequent planning task. A trajectory prediction is ranked abnormal if the planning cost is larger than some threshold. More precise, given the distribution of the prediction module, they propose a sampling based evaluation of the cost distribution to define the p-quantile cost threshold. If the estimated cost given the control inputs of the planning module exceeds the p-quantile threshold, the detector returns a positive detection, i.e. predicts an abnormal event.\n\nThey provide theoretical bounds for the false-positive rate (FPR) and false-negative rates (FNR) of the detector which can be used for data-free calibration of the detector, given either FPR or FNR. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nIn my opinion the paper makes a solid contribution both technically and empirically, while the technical contribution is major. Some comments on the strength:\n\n- Well written and well-defined problem definition.\n- Considering anomaly detection from a task-relevance perspective is a reasonable approach. The influence of errors in perception and prediction on the actual driving task is of major interest. For example, a wrong prediction of a vehicle in far distance of the ego is counted as prediction failure under many trajectory prediction protocols, however might not be a task-relevant failure, if it doesn't hurt the safety of the AV. \n- In AV safety, methods with theoretical proofs are highly valuable. The method allows to compute bounds on the FPR and FNR, although theoretical guarantees cannot be provided simultaneously on both quantities, but either on FPR or FNR.  The analysis of the limitations of the guarantees is noteworthy.\n\n### Weaknesses\nI have some suggestions for improving the paper:\n\n- The paper addresses the detection of trajectory prediction failures which lead to high costs in the planning module. I wonder how failures in prediction are distinguished from real driving anomalies, for example the predictor predicts a sharp manouver because a driver shows rude or inattentive behavior (which is not a prediction error but still results in high planning cost). A discussion would be useful.\n- The approach is compared to multiple baselines for detecting the anomaly: (1) Evaluating the *likelihood* of an achieved agent state under the GMM distribution of Trajectron++, (2) detecting shifts in the cost distribution using *uniform and partial cost degradation*, (3) detecting anomalies with low *time to collision (TTC)* and (4) *Hamilton-Jacobi reachability*. Recently, deep learning methods dominate anomaly detection, in particular trained methods to detect anomalies in multi-agent trajectories [1, 2]. For a complete picture, the comparison to one of the methods would be very useful.\n- A qualitative evaluation of the failure detector on the nuPlan teaser dataset is missing. It would enhance the paper if success and failure cases of the detector would be shown.\n\n[1] Wiederer, Julian, et al. \"Anomaly Detection in Multi-Agent Trajectories for Automated Driving.\" Conference on Robot Learning. PMLR, 2022.\n\n[2] Bera, Aniket, Sujeong Kim, and Dinesh Manocha. \"Realtime anomaly detection using trajectory-level crowd behavior learning.\" Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2016.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written and the proposed approach is both interesting and applicable to any trajectory prediction module. The support of the empirical evaluation could be improved.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "I have only minor issues (beyond my suggestions in \"Strength and Weaknesses\") that I would like the authors to address during revision\n\n- It would be helpful if the notion of the cost terms in eq. (4) is included in the text L.209 - L.212.\n- App. B2 L.504: error in term $15^{(T-1)}$\n- To favor open research, it would be highly appreciated if code and dataset will be made public."}}]

Author rebuttal:

Decision: Accept (poster)