Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper describes the approach taken by the 2023 nuPlan winning team. It tracks two metrics: ego-forecasting and closed-loop performance. 

The work puts forward three general approaches. The first one (PDM-Open) takes as input the centerline predicted by a rule-based planner (IDM), as well as the history of ego, and outputs the predicted ego trajectory. The second one (PDM-Closed) generates a collection of candidate trajectories by using a fixed set of hyperparameters and IDM. Candidate trajectories are then simulated, scored, and the best one is chosen for execution. The final (winning) approach (PDM-Hybrid) uses PDM-Closed to generate a plan and adds it as an input to a model similar to PDM-Open.

PDM-Open excels at ego-forecasting but underperforms at closed-loop evaluation, PDM-Closed is the opposite, and PDM-Hybrid preserves the performance of PDM-Closed while recovering the performance of PDM-Open on ego-forecasting.
review: I think this paper offers interesting insight into the problem of autonomous driving. Mostly it suggests to me that rule-based planners combined with simple learning methods might still have a lot to offer. That said I do have the following concerns about it.

First, a key design decision in PDM-Hybrid is that the planner only modifies trajectories suggested by PDM-Closed beyond the horizon of the simulator’s closed-loop controller (LQR). This is what allowed the team to bring their score on ego-forecasting back up without impacting their closed-loop performance at all. I think this is clever, but I hesitate to cast this as any sort of insight into the problem of autonomous driving beyond pointing to the fact that ego-forecasting is probably a bad metric to use for future competitions. That said I think the authors acknowledge this towards the end of the paper. They might simply want to emphasize it earlier as well instead of introducing both metrics as equals involving a tradeoff.

My second concern with the paper is that it provides limited insight into what leads to good performance in the ego-forecasting task. With PDM-Open, the limited input (ego history and centerline) dramatically reduces the ability of the planner to learn any sort of policy (in contrast to what planners like PlanCNN - a competitive benchmark - are forced to do by removing ego from their inputs). It is therefore not surprising to me that this leads to accumulating error and poor performance in the closed-loop setting. What *is* surprising to me is that this leads to better performance in the ego-forecasting setting, and I think that the paper doesn’t really provide any insight as to why that might be the case.

Overall the paper is well written and the experiment satisfactory. I also think that the methods proposed are simple enough that they are likely to be reproduced by the community.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall I think the experiments are thorough and the results thought provoking. The main strengths of the paper are the simplicity of its approaches and its very strong empirical results. I do not think that the weaknesses in some of the analysis should detract from the paper’s overall strong contributions.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper presents an empirical analysis of the performance of vehicle motion planning approaches in the nuPlan simulator. 
In doing so, the paper focuses on the difference in performance achieved between evaluating in a "closed-loop scheme" and an "open-loop scheme". While "open-loop" evaluation can be done on a static dataset by comparing generated actions of an algorithm with ground-truth labels, closed-loop evaluation requires an actual simulator where progress and collision rates are measured.  

The paper shows that there is a negative correlation between the two evaluation schemes when tested in the nuPlan simulator. Furthermore, the paper states that a simple rule-based planning baseline surpasses all SOTA learning-based methods. 
review: Strengths
========
- The paper is clearly written and reads well. 
- The related work provides a nice overview of rule-based planning and approaches performing ego-forecasting.
- The figures are well made and illustrate the task and its complexity well.
- One of the approaches presented in the paper won the nuPlan 2023 challenge.



Constructive Criticism
=================
- I am unfamiliar with the nuPlan challenge and do not know the level of complexity it poses. From the visualizations in the paper and the supplementary video, the tested environments look very simplistic, it appears that algorithms have perfect knowledge of the environment. Does this challenge contain a perception component? 
- While the study won the inaugural nuPlan challenge, it would be useful to know how the proposed method compares to other methods in more diverse and complex real-world scenarios. A comparison to additional datasets besides nuPlan could enhance the generalizability and robustness of the findings.
- The discussion section reads as if there are only two classes of planning approaches for autonomous vehicles: rule-based methods and approaches based on imitation learning. I'm no expert in the field, but I assume there are approaches that combine interpretable (i.e. close to rule-based) decision-making approaches with data-driven environment models. It would be interesting if the paper could extend on this.
- The paper refers to citation 59 for information about the evaluation metric's composition. Given that the metric is of such central nature to the findings of this paper, I suggest to include a description of it in the main text.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: My recommendation is mainly based on the fact that this paper tackles the task of motion planning in a setting where perception is perfect. Tackling the motion planning task without considering the challenges of perception seems disconnected from the state of the art.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper presents an empirical study on the nuPlan dataset and benchmark. Specifically, the authors compare a series of planners ---including a proposed one that won the nuPlan challenge--- with respect to their open-loop and closed-loop performance. Through extensive ablation studies, the paper discusses key takeaways and directions for future work. One interesting insight is the observation that rule-based baselines exhibit strong performance in closed-loop settings in the nuPlan challenge, suggesting that they should not be prematurely rejected as non-generalizable approaches for some driving tasks.
review: Strengths

-The paper is well-written and easy to follow with nice explanations of the technical components and a helpful discussion of results.

-The empirical study is quite extensive, containing ablation studies and in-depth discussions of the performance of the approaches evaluated.

-The findings are quite intriguing, demonstrating the value of rule-based approaches for some types of driving tasks. Through its rigorous evaluation, the paper follows a few recent works demonstrating the value of well-understood, model-based approaches. See Scholler et al. What the constant velocity model can teach us about pedestrian motion prediction. RA-L 2020; Poddar et al. From Crowd Motion Prediction to Robot Navigation in Crowds. arXiv:2303.01424. 2023. I've felt like there is a point to be made about this, and the authors execute this quite well.

Weaknesses

-Section 3 could be organized a little better. I feel like the introductory headers in bold are helpful but a better organization through grouped subsections could help readability.

-The scenarios highlighted are quite interesting but they feature relatively structured motion. Are there any insights for more interactive scenarios like (unsignalized) intersections? How would the PDM-based approach perform in such settings?

-A discussion of how the insights extracted would likely port over (or not) to the real world would be interesting to add.

***POST REBUTTAL***

The rebuttal addressed my comments and the additional supplementary material strengthened the paper. I remain in favor of acceptance.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this is a well written paper with compelling findings and well presented insights. With the incorporation of a few points of discussion (see above), I am in favor for acceptance.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This research paper investigates the alignment between open-loop trajectory forecasting and closed-loop planning metrics on the nuPlan benchmark in the context of autonomous driving, with learning-based planners excelling at the former but a simple 20-year-old rule-based planner performing best for the latter. Notably, a centerline-only model surpasses methods employing elaborate scene representations, indicating limited long-horizon context value in open-loop forecasting. The paper proposes PDM-Hybrid, which combines rule-based and learned components, yielding state-of-the-art performance on both metrics with enhanced simplicity and efficiency. Extensive experiments demonstrate the effectiveness of this approach.  The paper makes key recommendations like focusing forecasting evaluation on short control-relevant horizons and avoiding reliance on open-loop metrics for planning assessment. Code, data and models are released to facilitate future autonomous driving research.
review: The paper is well written and easy to follow. The paper's core contributions around revealing evaluation issues and limitations of learning methods are quite original. However, some ideas like using rule-based planners as a baseline have been explored before. The proposed PDM-Hybrid model is simple and high-performing but not an entirely new technique. The experimental results are comprehensive but significance is somewhat limited by focusing only on one simulator (nuPlan) and competition-style metrics. The findings may not generalize fully to other environments and tasks. 

The revealing analysis and actionable recommendations do provide valuable new insights that stand to advance autonomous driving research. However, the proposed techniques themselves represent only an incremental advance, as evidenced by the simple model achieving state-of-the-art results. Substantial innovation of new planning and learning techniques is still needed for broader progress.

Strengths:
+ The analysis in the paper reveals fundamental misalignments between open-loop ego-forecasting and closed-loop planning assumptions. It also shows limitations of learning methods compared to a simple rule-based approach.
+ The introduced PDM-Hybrid model achieves state-of-the-art results on nuPlan, winning the 2023 challenge. Its modular design is also flexible.
+ The paper makes specific recommendations for future research, such as focusing ego-forecasting evaluation on short 2 second horizons relevant to control, rather than unrealistic long 8 second predictions.

Weakness:
+ The paper lacks clarity regarding its contributions, making it difficult to discern the originality of the work and distinguish between incremental and novel aspects. The absence of a conclusion section further compounds this issue. It is recommended that the authors include a dedicated paragraph outlining their contributions and incorporate a conclusion section to address these concerns.
+ The modular PDM-Hybrid architecture exhibits limitations, particularly in the fact that its components are not learned end-to-end, potentially hindering the full utilization of crosstalk between different submodules.
+ The absence of information regarding the computational platform used for the simulations is a notable omission. Moreover, the reliance on access to HD maps and perfect perception raises questions about the current real-world applicability of the proposed method. Addressing imperfect state estimation and maps is crucial. Therefore, the authors should include discussions on how their proposed method could be extended to real-world scenarios. Additionally, conducting further evaluations in diverse and interactive simulators would enhance the assessment of generalizability.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, I'm positive about this work, but there are some limitations in terms of novelty, rigor, and technical advancements. The core revelation of misaligned metrics and learning method issues is an excellent contribution. However, the proposed PDM-Hybrid model only offers minor advances compared to established techniques. The experiments are fairly rigorous but lack extensive ablation studies and testing in diverse simulation environments to demonstrate generalizability. While the actionable recommendations for the field are impactful, the paper doesn't propose substantial new techniques to achieve these goals. Additionally, the lack of physical robot validation raises concerns about real-world applicability.
review_title: null

Meta-review: This paper conducts an empirical analysis of vehicle motion planning approaches in the nuPlan simulator. It specifically examines the difference in performance between using a "closed-loop scheme" and an "open-loop scheme". While the reviewers have generally positive about this work, they did identify some writing-related issues that should be addressed in the final version.
Decision: Accept (Poster)