Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper aims to generate trajectories that abide by both kinematic and contextual constraints. The kinematic constraints include motion planning feasibility. The contextual constraints include road compliance, following traffic lights, following the speed limits. The key idea of this paper is to (1) generate a lot of plans and trajectories (2) train an evaluator to give the scores each trajectory.

The plans and trajectories are generated by doing depth first search in HD map for plans, and then use the dynamics to figure out the trajectories.

The evaluator is trained with the objective that regresses the predicted score to the labeled score using a cross entropy loss. The labeled score is computed by how far they are from the ground truth trajectory. The model takes both states, paths, and trajectories from the previous step, and pass them through a bunch of attention layers and output the score.

The authors show low p-ADE and miss rate meaning that it can cover the ground truth trajectory within 2 meters within one of the 6 predicted trajectories with 89% accuracy on Argoverse. The authors show that it is robust when tracking drops. 
strengths_and_weaknesses: The paper is well-written and clear.
The method is quite different from a more mainstream approach of using deep NN to generate trajectories but instead learning an evaluator. This is important and can inspire many more avenues in this direction.
The results demonstrate the improvement over the previous state-of-the-arts. The paper shows the robustness when tracking is dropped.
summary_of_recommendation: The paper proposes an interesting idea of combining planning and learning in motion forecasting. The results surpassed other SOTA. While the paper does a great job considering robustness to tracking, I wonder if there is a reference of the actual problem in tracking. 
issues: What are the failure modes of missing trajectories in the 11%? Can you provide common example scenarios?

How realistic is the tracking dropping? The experimental settings seem a bit constructed. Were there some references to that this is a common issue in practice?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose a prediction model that ranks different plans generated conditioned on reachable paths in the road network. A planner submodule is used to plan along the path, and an evaluator is used to rank the trajectories after encoding them. The authors detail the structure of the evaluator and choice of planner submodule, post-processing selection, as they obtain favorable results on Argoverse.
strengths_and_weaknesses: Strengths: Results  on Argoverse seem good, and some ablations show the exploration of design tradeoffs. Model-based trajectories can improve prediction at long horizons for majority of the driving scenarios, and can leverage the rich literature on planners, including different goals, rewards and constraints. I liked the exploration of tracking impact as it’s often neglected, and the use of local/global coordinates, which make sense and has been also shown in the recent CVPR WAD workshop discussions.

Weaknesses: The paper’s main claim to fame (prediction via planning) and the main structure (i.e use of ranking) has been significantly explored before. While the specific approach is new, the readers would benefit from a more balanced view that targets the (much needed) set of open questions in the field that the paper touches on, such as exploration of the impact of the choices for planner, evaluator, lane handling, planner sample parameters (the path selected, any other parameters that prevent impoverished prediction set?).

summary_of_recommendation: While the approach is not completely novel, and reuses elements from many existing works, it is overall a new combination, is presented clearly, and achieved good result on a valid benchmark. Would be good to see a better presentation with respect to other works that do planning for prediction, or generate-then-evaluate, and to explore the tradeoffs that drove the error down for this system.
issues: * How does the approach depend on the quality and diversity of the proposals made under the assumption of limited number of samples (MoN, N~5..7)? Do more suggested plans hurt performance, does the performance depend on good and “representative” action set? (the main problem is that it’s hard to generalize a models and plans to cover edge cases, which is why people look at model-free trajectory generation)
* Why do we want planning-based predictions? Are we learning feasible or rewarding trajectories, and if so, why the assumption on a reward/planning structure? How does this compare to other approaches such as IRL / reward shaping (“Maximum Entropy Deep Inverse Reinforcement Learning“, “Social behavior for autonomous vehicles”), learned planning for  corrections of predictions (“Differentiable Logic Layer for Rule Guided Trajectory Prediction”), and injection into the discriminator (“Vehicle Trajectory Prediction Using Generative Adversarial Network With Temporal Logic Syntax Tree Features“) beyond citing, as the heavier machinery is in writing planners, seems like testing the other approaches for a comparison would’ve been easy and worthwhile. Would be also good to cite “What the Constant Velocity Model Can Teach Us About Pedestrian Motion Prediction“, as it makes a relevant claim -- learning-based generators are still quite sub-optimal given the motion statistics, to the point that simplified models (such as path following) can outperform them.
* While it’s interesting to ablate the local/global coordinates representation and the F2F attention, the interesting ablations would be to compare the evaluator and planner contributions, since individually they have appeared in several works. Moreover, while adhering to lanes is important, this has been used in predictors before, so would be interesting to compare the specific representation for lane following vs. other approaches.
* Where are the cases where planning is not adhered to and how bad are such mistakes? The main reason for model-free prediction is to handle rare events, but these are not always captured by path following.
* In cases where multiple trajectories are generated for the same path, or nearby examples -- how do they differ? What do they capture? Would be good to hear/see examples, in addition to failure cases shown in the supp. mat.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a technique to generate robust and feasible vehicle trajectory predictions. A model-based motion planner is designed to search reachable reference paths, around which the candidates of trajectory predictions are sampled, guaranteed to be compliant with kinematic constraints and road structures. The trajectories will be scored by a learning-based evaluator, which takes map structure and agent interaction into consideration. 
strengths_and_weaknesses: Strengths:
1) The paper is well-written and easy to follow. 
2) The way to utilize map information is interpretable, and results are guaranteed to be feasible.
3) Most model-based methods fail to generate diverse trajectories in interactive scenes, while learning-based methods suffer from trajectory quality. This paper combines the advantages of both methods and performs well in the argoverse competition.
 
Weaknesses:
1) The novelty of this paper is the combination of a model-based planner and a learning-based evaluator. However, the planner itself is widely used in industry, and the evaluator follows LaneGCN’s structure. I am not sure about the inference time for this proposed method.
summary_of_recommendation: I would recommend this paper because it proposes a new method to make trajectory generators more practical. The experimental results are convincing.
issues: I am curious about the forward inference time.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a novel prediction framework called PRIME that learns to predict vehicle trajectories with a two-part approach with a model-based generator and a learning-based evaluator. This two-part system allows PRIME to generate accurate and feasibility-guaranteed predictions. 
strengths_and_weaknesses: 1) I would suggest that the authors consider adding a discussion of the limitations of the approach to the end of the paper.

Minor comments

1) Figure 4 is small and difficult to read
2)Figure 3 caption is too long and should be reworked
summary_of_recommendation: The paper is well written and tackles a  relevant problem. The authors give a thorough description of their approach as well as experiments validating their claims. The author's two-part approach to trajectory prediction is novel and seems promising.

However, a more robust experiment where more detailed on-road information would give a better picture of how much the PRIME improves feasibility when compared to the other baselines.

Also important is the need of a more
developed discussion around the limits of the approach.
issues: The paper is well written and tackles a well-studied but
still relevant problem. The authors provide a good
literature review and clearly explain their architecture,
presenting some experiments to support their claim.  I believe it is suitable for CoRL.
review_title: null

Meta-review: Summary: This work proposes a prediction mechanism that combines model-based path and trajectory planners with a learned ranking system. The proposed method is supported by results on the Argoverse Benchmark. 

Clarity: All reviewers commented on the good composition of the paper. They seemed to agree that the paper was clearly and concisely written and overall effective in convincing the reader of the promise of the proposed approach. 

Originality: Reviewers noted that while the paper adopts existing techniques and a popular framing of the problem, it does combine specific techniques in a novel way. 

Quality: Reviewers were overall convinced that the results presented on the Argoverse Benchmark demonstrated a believable improvement over the state-of-the-art, however, also made suggestions on how to make the case even stronger. 

Significance: Reviewers had mixed feedback on the impact of the paper. It seems like reviewers were mainly looking to the experimental results to convince them and noted some gaps and lingering questions. 

Pros: 
The paper proposes a compelling way to combine model-based and learning-based tools to solve an important problem in robotics. The paper also addresses salient details such as the effect of tracking error and ablation experiments. 

Main opportunities for improvement: 
The authors are encouraged to focus efforts on addressing the questions raised and do the suggested additional analyses the reviewers requested in the experimental results section. The goal of these additions should be to resolve any lingering questions about the effectiveness of the proposed approach. 

Thank you for considering our feedback, and we look forward to seeing the updated paper.

============== Final Decision

All reviewers recommend acceptance. 

I am not 100% certain if this should be a poster v an oral presentation. The case for oral is that two reviewers gave a strong accept with strong scores in all sub-categories. However, I ultimately recommended poster because sub-scores in impact and originality are mixed (in particular, 2 in impact from reviewer PMNK  and fair in originality from reviewer SJ8P). 
Decision: Accept (Poster)