Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies the problem of anomaly detection for trajectory predictors in autonomous vehicles. A probabilistic framework that detects task relevant failures, by propagating trajectory prediction errors to the planning cost is developed. To be able to distinguish between harmful and non-harmful mispredictions, the authors design of a prediction failure detector based on the concept of p-quantile anomalies. The proposed detector comes equipped with performance measures on the false-positive and the false-negative rate and allows for data-free calibration. To benchmark the performance of the anomaly detector, the authors compared their approach against various other methods on nuScenes and nuPlan datasets .The authors demonstrated that the proposed detector outperforms all other baseline approaches and is compute-efficient.
strengths_and_weaknesses: Identified strengths of the paper:
     - Design of a prediction failure detector based on the concept of p-quantile anomalies.
     - FPR and FNR to guarantee that the anomaly detector facilitates the data-free calibration. 
     - The obtained results can, in principle, be reproduced. Even though key resources (e.g., proofs, code, data) are unavailable.

Weakness/ Suggested imporvement of the paper:
      - The authors use Trajectron++, which is based on a conditional variational autoencoder (CVAE) to model the potential for multiple 
         future trajectories. However, the sampling is based solely on likelihood which may only produce samples that correspond to the 
         major modes of the data distribution. How can you garantee the diversity of the set of samples ?
      - For the results in Tab.3 it would be useful to provide a measure of statistical dispersion (e.g. standard deviation or standard error), to 
        be able to statistical significance when comparing the the previous baselines
      -  As a qualitative comparison, it could be useful to show some examples, where the proposed anomaly detector outperforms the 
         previous approaches. 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: 
The problem is interesting and useful. The method is reasonable. The experiments conducted on benchmark are convincing. The few issues pointed out above should be addressed.
issues: Listed above as suggestions for improvement.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: 1. The authors develop a probabilistic run-time failure detector for the prediction module of autonomous driving.
2. The main characteristic of this detector is "task-relevant". Namely, by incorporating the planning cost in the detector, the detector is designed to detect false predictions leading to the planning cost deviation. 
3. The authors provide theoretical proof for the FPR, FNR bounds of the proposed detector.
4. The proposed detector is validated in two datasets and compared to several baseline methods.
strengths_and_weaknesses: ### Strengths

1. The idea of incorporating planning costs to measure task relevance is good.

2. The problem of detecting prediction failures is essential.

3. The paper is easy to understand and well written.

### Weaknesses

1. Although it claimed that "our framework is agnostic to the choice of the cost function", the detector may be **sensitive to the cost function design**. Eq. (1) essentially evaluates how the estimated cost (cost evaluated from state estimated) is ranked among the predicted costs (costs evaluated from the M sampled predicted trajectories). M & n are the hyperparameters for this detector. If the cost function is tuned a little bit, the predicted cost distribution may be completely different.  Take the cut-in case in Fig.1 (a) for example, the planning cost can be either tuned to have a large safety margin or a small one (actually this value may be  adjusted by the user on the fly), to obtain a reasonable balance between FPR and FNR, the M & n may need to be tuned. Should we tune individual detectors for every cost function? ( it may not be realistic). Or one universal setup of M & n will work for all cost function designs?  (the answer may be no since the cost distribution is different and something should be tuned for better FNR and FPR balance) *This may be an issue preventing the detector from working in a real autonomous driving system since the cost function is complicated and tuned very frequently.*

2. For complicated driving scenarios, **how to choose M?** Should we fix the M for all scenarios? For example, for a number of non-ego interactive agents in an intersection (every agent may have multimodal predictions), a very large M may be needed to yield a good estimation of the predicted costs. Will a very large M also work for normal on-lane driving?

3. How the probability of the predicted trajectory will be incorporated?

4. The metrics for evaluation are not strong. FNR/FPR/ROC only represents the precision-recall & accuracy of the detector.  What is the consequence of an FN or FP event? What does it mean by 8% FP or FN rates? Should we abandon the prediction results when a failure is detected (or at what FP level we should abandon the prediction results)?

5.  The failure detection is done by post-validating predicted planning costs against true costs. The detection is always delayed compared to the first frame of abnormal prediction. **What is the average delay for the detector** (for different M & n & FPR & FNR) ? What if there is only one frame of abnormal prediction and then the prediction is automatically recovered?  When we find one positive, should we regard it as a failure? (when the detector finds there is a failure, the prediction may be already normal).

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: 1. The idea of task-relevant failure detection is good.
2. The current design of the detector is too simple to work in real autonomous driving vehicles, as listed in the weaknesses.
3. More thorough evaluation should be presented.
4. Still a gap to a solid contribution. 
issues: 1. Justify whether the detector is sensitive to the cost function design. Try different cost function (with different terms and weights) and show whether a global M & n setup will work.

2. Justify how to choose M? Try complicated scenarios (multiple non-ego agents at intersection) & visualize the predicted cost distribution estimated from M predicted trajectories.

3. How the probability of the predicted trajectory will be incorporated?

4. Evaluate the consequence of an FN or FP event. What does it mean by 8% FP or FN rates? Should we abandon the prediction results when a failure is detected (or at what FP level we should abandon the prediction results)?

5. Justify the delay of the detection?  **What is the average delay for the detector** (for different M & n & FPR & FNR) ? What if there is only one frame of abnormal prediction and then the prediction is automatically recovered?  When we find one positive, should we regard it as a failure? (when the detector finds there is a failure, the prediction may be already normal).
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In the paper a task-relevant failure detector for monitoring trajectory prediction is proposed. The novelty is, that prediction errors are detected by evaluating the effect of the prediction on the subsequent planning task. A trajectory prediction is ranked abnormal if the planning cost is larger than some threshold. More precise, given the distribution of the prediction module, they propose a sampling based evaluation of the cost distribution to define the p-quantile cost threshold. If the estimated cost given the control inputs of the planning module exceeds the p-quantile threshold, the detector returns a positive detection, i.e. predicts an abnormal event.

They provide theoretical bounds for the false-positive rate (FPR) and false-negative rates (FNR) of the detector which can be used for data-free calibration of the detector, given either FPR or FNR. 
strengths_and_weaknesses: ### Strengths
In my opinion the paper makes a solid contribution both technically and empirically, while the technical contribution is major. Some comments on the strength:

- Well written and well-defined problem definition.
- Considering anomaly detection from a task-relevance perspective is a reasonable approach. The influence of errors in perception and prediction on the actual driving task is of major interest. For example, a wrong prediction of a vehicle in far distance of the ego is counted as prediction failure under many trajectory prediction protocols, however might not be a task-relevant failure, if it doesn't hurt the safety of the AV. 
- In AV safety, methods with theoretical proofs are highly valuable. The method allows to compute bounds on the FPR and FNR, although theoretical guarantees cannot be provided simultaneously on both quantities, but either on FPR or FNR.  The analysis of the limitations of the guarantees is noteworthy.

### Weaknesses
I have some suggestions for improving the paper:

- The paper addresses the detection of trajectory prediction failures which lead to high costs in the planning module. I wonder how failures in prediction are distinguished from real driving anomalies, for example the predictor predicts a sharp manouver because a driver shows rude or inattentive behavior (which is not a prediction error but still results in high planning cost). A discussion would be useful.
- The approach is compared to multiple baselines for detecting the anomaly: (1) Evaluating the *likelihood* of an achieved agent state under the GMM distribution of Trajectron++, (2) detecting shifts in the cost distribution using *uniform and partial cost degradation*, (3) detecting anomalies with low *time to collision (TTC)* and (4) *Hamilton-Jacobi reachability*. Recently, deep learning methods dominate anomaly detection, in particular trained methods to detect anomalies in multi-agent trajectories [1, 2]. For a complete picture, the comparison to one of the methods would be very useful.
- A qualitative evaluation of the failure detector on the nuPlan teaser dataset is missing. It would enhance the paper if success and failure cases of the detector would be shown.

[1] Wiederer, Julian, et al. "Anomaly Detection in Multi-Agent Trajectories for Automated Driving." Conference on Robot Learning. PMLR, 2022.

[2] Bera, Aniket, Sujeong Kim, and Dinesh Manocha. "Realtime anomaly detection using trajectory-level crowd behavior learning." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2016.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well written and the proposed approach is both interesting and applicable to any trajectory prediction module. The support of the empirical evaluation could be improved.
issues: I have only minor issues (beyond my suggestions in "Strength and Weaknesses") that I would like the authors to address during revision

- It would be helpful if the notion of the cost terms in eq. (4) is included in the text L.209 - L.212.
- App. B2 L.504: error in term $15^{(T-1)}$
- To favor open research, it would be highly appreciated if code and dataset will be made public.
review_title: null

Meta-review: metareview: The paper addresses an essential problem with a convincing, reasonable approach and analyzes its theoretical guarantees. It is well-written, however, the experimental evaluation needs to be more thorough. 

Main pros:
- The paper addresses an essential problem with a convincing approach and analyzes its theoretical guarantees
- The paper is well-written

Main cons:
- There are several concrete technical concerns that were raised by the reviewers and need to be addressed
- More thorough evaluation is needed

Post-rebuttal update: The revised manuscript successfully addressed the main issues.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)