Title: REDOUBT: Duo Safety Validation for Autonomous Vehicle Motion Planning
Venue: NeurIPS 2025
Accepted: yes
Keywords: autonomous driving

motion planning
Abstract: Safety validation, which assesses the safety of an autonomous system's motion planning decisions, is critical for the safe deployment of autonomous vehicles. Existing input validation techniques from other machine learning domains, such as image classification, face unique challenges in motion planning due to its contextual properties, including complex inputs and one-to-many mapping. Furthermore, current output validation methods in autonomous driving primarily focus on open-loop trajectory prediction, which is ill-suited for the closed-loop nature of motion planning. We introduce REDOUBT, the first systematic safety validation framework for autonomous vehicle motion planning that employs a duo mechanism, simultaneously inspecting input distributions and output uncertainty. REDOUBT identifies previously overlooked unsafe modes arising from the interplay of In-Distribution/Out-of-Distribution (OOD) scenarios and certain/uncertain planning decisions. We develop specialized solutions for both OOD detection via latent flow matching and decision uncertainty estimation via an energy-based approach. Our extensive experiments demonstrate that both modules outperform existing approaches, under both open-loop and closed-loop evaluation settings. Our codes are available at: https://github.com/sgNicola/Redoubt.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces REDOUBT, a duo safety validation framework for motion planning task in autonomous driving. REDOUBT features more robust safety validation, through concurrent scenario OOD detection via latent flow matching, and decision uncertainty estimation quantified by ESDF and modal probability. REDOUBT is validated through multiple planners in both open and closed loop settings, proving effectiveness across other OOD detection methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The motivation for safety validation in autonomous driving is well-articulated and compelling.\n\n2. The paper presents extensive comparisons with OOD detection baselines in both open-loop and closed-loop settings. REDOUBT demonstrates effectiveness across multiple widely adopted planning algorithms.\n\n**Weaknesses**\n\n1. The originality of each verification module is somewhat limited. Prior works such as [1] and [2] have also explored latent flow matching and energy-based methods for domain identification and uncertainty quantification.\n\n2. Despite a thorough introduction and comprehensive experimental comparisons with OOD detection methods, the paper lacks deeper insights into how the proposed identification mechanism concretely improves or mitigates failures in open-/closed-loop planning.\n\n3. Since the train/evaluation split is manually separated based on frequency, it is unclear whether the method can truly identify hard-but-frequent edge cases.\n\n4. The qualiative results are not intuitative enough. More supplementary like closed-loop identification are expected. No code are provided for verification\n\n5. How is each planner trained? Is there poential risk for data leakage?\n\n[1] DECODE: Domain-aware Continual Domain Expansion for Motion Prediction\n\n[2] SOTIF Entropy: Online SOTIF Risk Quantification and Mitigation for Autonomous Driving\n\nQUESTIONS:\n1.  How is the difficulty of a scenario quantitatively defined or evaluated? Can the method distinguish between hard-but-frequent vs. rare-but-trivial cases?\n\n2. How does the proposed identification mechanism concretely benefit downstream planning performance, especially in closed-loop execution?\n\n3. Since the separation of training and evaluation sets is manually based on frequency, is there a risk of implicit leakage or circular reasoning when evaluating OOD detection? How sensitive are the results to the chosen threshold of frequency?\n\n4. To what extent do the detected OOD cases actually correlate with safety-critical planner failures? Could some OOD regions be benign or irrelevant to planning outcomes?\n\n5. Is there an ablation showing the importance of each verification module (e.g., flow-based, energy-based)? Could a simpler or more modular combination of components achieve similar results?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents REDOUBT, a novel approach for self-driving ML planner’s OOD detection. In addition to the trajectory decoding head, it adds two head that predicts 1) uncertainty, supervised by heuristics on collision and offroads and 2) a flow-matching based OOD detector.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n+ The paper is well written and tackles an important problem that is relatively under-explored.\n+ The proposed method is intuitive and easily integratable to arbitrary (ML)-planner stack.\n\n\n### Weaknesses\n- The statement in L57 “We propose REDOUBT, the first systematic safety validation framework for autonomous vehicle motion planning” felt overclaiming. There has been thorough prior work on self-driving safety validation [1,2]. I’d make sure the claim is more specific, e.g. clarify that the approach focuses on OOD detection.\n- My main concern is that there is no closed-loop evaluation on the entire stack – given the OOD detection enabled by REDOUBT, how much does the overall planning stack improve?\n\n[1] Determining Absence of Unreasonable Risk: Approval Guidelines for an Automated Driving System Deployment, Favaro et al.\n[2] Comparison of Waymo rider-only crash data to human benchmarks at 7.1 million miles, Kusano et al.\n\nQUESTIONS:\n- Given the OOD detection enabled by REDOUBT, how much better does the overall planning stack improve? Can the authors evaluate this with standard planning metrics? \n- It’s unclear to me why it has to be flow-matching which requires solving ODE during inference if the sole purpose is to score the trajectory prediction. For example, why not a conditional normalizing flow or VAE?\n- I’d like to see some case or qualitative analysis on the failure modes, e.g. false positives and negatives.\n- What are the additional computes/latency of REDOUBT to a planner stack?\n\nLIMITATIONS:\nThe main limitations at the moment is the lack of closed-loop evaluation with planning metrics on the entire system. Also, no explicit discussion on the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces REDOUBT, a novel duo safety validation framework for autonomous driving motion planning that jointly considers scenario distribution (via OOD detection) and output uncertainty (via planning confidence). The framework is validated across multiple learning-based motion planners on the nuPlan dataset under both open-loop and closed-loop settings and demonstrates consistent improvements over existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The duo validation perspective is insightful. This paper provides a compelling argument for jointly validating both input distribution and output uncertainty, highlighting overlooked unsafe cases that each dimension alone would miss (InD + Uncertain and OOD + Certain). This insight is practically meaningful for AV safety.\n- The use of flow matching for OOD likelihood estimation is interesting and shows consistent improvements over a wide range of post hoc and feature-space baselines.\n- The decision uncertainty estimation module works for both open-loop and closed-loop settings, and is evaluated across multiple motion planners and under three settings (OP/CNR/CR), with consistent performance gains.\n \n**Weaknesses**\n- The paper evaluates OOD detection and uncertainty estimation independently but does not quantify REDOUBT’s overall safety validation performance as a unified system. It would be valuable to define and report metrics that directly reflect end-to-end validation effectiveness.\n- The method assumes access to structured scenario inputs (e.g., object trajectories, maps), which may not be available in end-to-end autonomous systems that go directly from raw sensor input to planned trajectory. Can REDOUBT be adapted to operate in such sensor-to-plan pipelines?\n\nQUESTIONS:\n* The paper evaluates OOD detection and uncertainty estimation separately, but lacks evaluation for overall validation effectiveness. Can the authors discuss how to evaluate the performance of the whole system?\n* How can REDOUBT be applied in end-to-end autonomy systems that operate directly on raw sensor inputs\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents REDOUBT, a safety validation framework for autonomous vehicle motion planning that addresses gaps in current safety assessment approaches. The key innovation lies in its dual validation mechanism that simultaneously evaluates both input distributions (traffic scenarios) and output uncertainty (planning decisions). The framework tackles two challenges: Out-of-Distribution (OOD) detection for identifying unusual driving scenarios, and decision uncertainty estimation for assessing the reliability of planning outputs. REDOUBT employs latent flow matching for OOD detection and an energy-based method for uncertainty quantification based on safety risks from obstacle proximity. The authors demonstrate through comprehensive experiments on the nuPlan dataset that their approach significantly outperforms existing methods across multiple evaluation metrics, providing enhanced safety guarantees for autonomous driving systems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper tries to address a critical and practically important problem in autonomous driving.\n- The proposed approach is technically sound and well-motivated.\n- Comprehensive experimental evaluation on the nuPlan dataset with comparison against existing methods shows clear performance improvements across multiple scenarios.\n\nWeaknesses\n- Limited discussion of computational overhead and real-time performance implications.\n- While the approach shows promise, the practical deployment challenges and integration with existing autonomous driving stacks are not thoroughly discussed.\n- More detailed analysis of failure cases would be valuable.\n- Only InD scenarios with certain decisions can be considered safe; such a condition may be too strict and could cause excessive takeovers in practical driving scenarios.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-22"}