Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper tackles the problem of agent prediction, an extremely important problem for autonomous driving. The authors leverage the fact that the graph structure of the underlying road scene can be exploited for making trajectory predictions, and they demonstrate how this can be done efficiently by decomposing the trajectory into the lateral and longitudinal components together with clever sampling of candidate trajectories and some other tricks. 
strengths_and_weaknesses: Strengths:

[S1] The results seem solid and the ablations are extensive

[S2] From what I can tell, the approach seems novel and the extensions over prior art are well-explained and make intuitive sense

Weaknesses:

[W1] I find some statements hard to understand, or that they lack precision and clarity. E.g.: "Since the entire scene is aggregated together and used to compute all predictions, it places a strong requirement on the output header" what requirement?

[W2] There seem to be a lot of parameters and hyperparameters in the approach. E.g. 
 - "We consider all lane centerlines within an area of [-50, 50] m laterally and [-20, 80] m longitudinally around the vehicle of interest. We divide the lane centerlines into snippets of length ≤ 20m, and discretize them to a set of N poses with 1m resolution" 
 - "For each node v, we only consider nearby agents within 10m of the lane centerline" 
 - "with direction of motion within a yaw threshold of the pose"
The prevalence of all of these heuristics calls into question the robustness of the proposed approach. For each of these choices it would strengthen the work to have some more context or analysis of the sensitivity of the method to the parameters.  

[W3] The method seems quite complicated and the architecture used is quite complex with many learned compenents. It is fine that it takes a complex solution to solve a complex problem, but I feel the manuscript could do a better job of elevating the discussion so that we may learn some more generally applicable lessons rather than the step by step recipe for solving the exact problem that the authors set out to solve. 


[W4] Minor issues:
probabilitic -> probabilistic
"encodes the scene context a hierarchical representation of" a word is missing somewhere
"fig 3" vs. "figure 3" be consistent. 
"the model tend" -> the model tends

summary_of_recommendation: In summary, I think that the approach seems novel and the results demonstrate its effectiveness. However, I think that the work could be elevated and that the clarity of the exposition could be improved. At present, it comes across as clever set of heuristics and a particularly good job of engineering the components to work together. 
issues:  - Sensitivity analysis with respect to parameter choices
 - Improved clarity of description of method that separates the core generalizable ideas from the heuristics and implementation details.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work focuses on the task of trajectory prediction in a driving context in the presence of multiple road uses. Their approach relies on decomposing the scene into a spatial graph which encodes legal and plausible transitions from one location to another. The key to their approach is that they train a probabilistic prediction policy to generate candidate trajectories in the graph, rather than in coordinate space, and are later transformed to coordinate sequences with another sampling step. By generating probabilistic predictions in graph space and final predictions conditioned on the graph sequences, they more cleanly separate spatial modes.

strengths_and_weaknesses: Strengths:
This paper demonstrates state of the art prediction results on the nuScenes benchmark.
They perform an ablation study to demonstrate the effectiveness of various aspects of their approach
Weaknesses:
The paper is selling the premise that their approach of conditioning the final prediction on an encoded graph traversal is the secret sauce. However, in their ablation study they show that conditioning instead on a goal is only slightly worse (1-2cm avg displacement error). If anything their own results show that conditioning the final prediction on a latent variable contributes just as much to their final performance. In fact, all variants in their ablation study outperform the state of the art listed in their baselines.
Would be nice to see performance on other baselines
summary_of_recommendation: Authors propose novel trajectory prediction approach conditioned on probabilistic graph traversals and achieve state of the art on the nuScenes benchmark. However, from the ablition study it's unclear how much impact the graph traversal truly has. The impact could perhaps be better highlighted by implementing a more diverse set of benchmarks.
issues: I think readers would benefit from a more careful analysis of what makes their approach perform so well. It could be that the success depends less on the novel aspects proposed in the current paper and more on other aspects (i.e. use of transformer, etc...)
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose a method for multimodal trajectory prediction using a graph neural network. They decouple the predicted trajectories components as lateral and longitudinal behaviors and achieve the lateral variability that aggregates part of the scene context by sampling path traversals while ensuring longitudinal diversity via an added latent variable. The experiments are well-presented, and the benchmark evaluations are convincing.
strengths_and_weaknesses: The proposed method utilizes the road content as lane graphs. Therefore, the generated trajectories are refined within the road ranges with lower off-road rates. The methodology is concise and with good performance compared with the state-of-the-art on a public dataset and explains the observations of the results. Overall, the paper is sound and well-structured. However, it is unclear about the lateral and longitudinal behavior division on these two decoders, and the implementations about decoder comparison are confusing. There are also some issues to clarify or revise.
summary_of_recommendation: This paper has done a good job but there are still some weaknesses and issues to solve. But I still recommend this paper to be published.
issues: - Abstract:
The statement "Accurately predicting ... inherent uncertainty in goals and driving behavior. This uncertainty can be loosely decoupled into lateral (e.g., keeping lane, turning) and longitudinal (e.g., accelerating, braking). "It mentions the inherent uncertainty in goals and driving behavior, but the second sentence only explains the uncertainty of driving behaviors.

-Introduction, line 46, "Probabilitic" Is it a typo?

- "Fig. 1.", "Figure 2", "figure 3", "fig. 3" Please be consistent with the notations.

- 4.3 Decoding trajectories conditioned on traversals

The proposed method directly linearly propagates vehicles' motion with latent variable z to achieve longitudinal resilience instead of using the states of the predicted trajectories. But, does it is accurate to qualitatively represent the longitudinal movements?

- Figure 3 is confusing and too small to distinguish the difference. It is recommended to add some notations.

- It is recommended to integrate some details of the supplementary materials into the main content.

- The gif is good to illustrate the information of the proposed method, and it is recommended to add different scenarios with other lateral behaviors.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper focus on the vehicle trajectory prediction, which is one of the necessary parts of autonomous driving technology. In this condition, a future trajectory of a vehicle is predicted based on the environment (e.g., a map of road lane) and past trajectories of other vehicles or pedestrians nearby. While this predicted trajectory is expected to meet the certain constraints such as traffic rules and road topology, it is also involved in uncertainties from other vehicles and pedestrians or drivers’ own behaviors. In this paper, instead of analysing the interaction of these uncertainties with a vehicle, authors decouple them into two parts at the trajectory stage: the lateral changes (e.g., keeping lane, turning) and the longitudinal changes (e.g., accelerating, braking). A model named PGP is therefore raised in this paper, where the lateral variability is captured by a selectively aggregate part with behavior cloning policy based on the features from a graph encoder, and the longitudinal variability is presented by a latent variable model. The prediction on nuScenes dataset shows a SOTA performance.
strengths_and_weaknesses: An encoder-decoder structured model with a policy header included is proposed. The model performs a SOTA results on nuScenes dataset. However, the hypothesis “decoupling uncertainty into lateral and longitudinal” (mentioned in line 3-5) , as the precondition of the method in this paper, has not been discussed by authors. Another main issue is that authors stress two key ideas, predictions conditioned on traversals by behavior cloning and latent variable for longitudinal variability, while these two ideas have not been fully discussed and analysed in the paper, which makes the paper lose focus on its main cocepts.
summary_of_recommendation: The paper fails to fully discuss its key ideas and show the relationship between the SOTA performance and its ideas. It is recommended to reorganize the paper and focus more on these ideas than other details of medel.
issues: 1.	The caption of figure 3 is incomplete. Lacking of explanation of figures.
2.	As a precondition of the method proposed in this paper, decoupling uncertainty into lateral and longitudinal (mentioned in line 3-5) is not fully discussed and analysed, or supported by other literature.
3.	One of the two key ideas, predictions conditioned on traversals, is implemented by sampling path traversals from a learned behavior cloning policy (mentioned in line42-47). However, this key idea is insufficiently discussed, only few details is shown in section 4.2 from line 163 to line 178. It could be more convincing if the structure of SVM used in section 4.2 (mentioned in supplementary material), or the formula of loss function used in line 178, and any other necessary details are included in this paper.
4.	One of the two key ideas, latent variable for longitudinal variability, is seemed to be only discussed in section 4.3 from line 190 to line 196. It could be more convincing if the literature of latent variable, or the reason why vector z is sampled from multivariate standard normal distribution, or the comparison with latent variable part in other SOTA models, and any other necessary details are included in this paper. 
5.	The multi-agent behavior is a main topic in vehicle trajectory prediction. It is implicitly or explicitly considered in SOTA models (e.g., the model Trajectron++ mentioned in comparison). The model PGP proposed by this paper achieved SOTA results. Therefore, it is expected to indicated how these two key ideas or other features of PGP help to describe and analyse the multi-agent system in trajectory prediction.
6.	In figure 3, as it is described from line 258 to line 260, the model performance seems to be different in a straight road and different types of road junctions. However, this difference is not discussed.
review_title: null

Meta-review: This paper proposes a novel method for multimodal trajectory prediction. While the reviewers agree that this paper contains some interesting ideas and the proposed method is effective, they also raise a number of important concerns. The authors should carefully address the reviewers' comments in their rebuttal. 

UPDATE POST DISCUSSION PHASE: I would like to thank the authors for their comments and clarifications during the discussion phase. The reviewers generally agree that this paper provides a valuable contribution, and I concur with this assessment. In the final version of this paper, the authors should carefully address all reviewers' comments and suggestions, in particular regarding hyperparameter sensitivity and exposition clarity. 

Decision: Accept (Poster)