Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Problem: Vehicle Trajectory Prediction as commonly defined and evaluated.

Key insight: agents usually follow lanes, so explicitly use lane graph structure to find likely future agent-agent interactions.

Novel contributions: Future Relationship Module that consumes waypoint occupancy estimates + lane graph and produces a prior distribution of interactions conditioned on the historical trajectory, trained against a posterior distribution conditioned on the full trajectory for training. Ablations show that this module is the key to system performance, as without it performance deteriorates to baseline which only uses historical trajectories.

Evaluation: competitive / SotA performance on standard metrics for popular public Trajectory Prediction benchmarks (NuScenes and Argoverse 1)


strength_and_weaknesses: Strengths: Good performance on popular public Trajectory Prediction benchmarks (NuScenes and Argoverse 1). The novel contribution, the Future Relationship Module, is shown to benefit the performance of the network. Furthermore, Future Relationships Module appears to provide useful features that are orthogonal to the modules of other competitive methods, opening up the possibility of future work that combines these methods together.

Weaknesses: Paper's actualized key insight is "agents usually follow lanes" but they do not do any analysis to show how often this assumption is violated. Due to the fact that the method performs well, presumably it's not very often, but it would be nice to know often it emerges in the train/val dataset and qualitatively what happens to the predictions under these conditions.

NuScenes results not fully SotA due to missing models (https://www.nuscenes.org/prediction?externalData=all&mapData=all&modalities=Any). DGCN_ST_LANE has significantly lower MinADE_5 (1.092) vs the proposed method (1.19 as reported in the paper), a gap larger than reported against the runner up PGP, and DGCN_ST_LANE was submitted on May 25th 2022, which is significantly before the Sept 28 2022 ICLR submission deadline and thus should be mentioned in the results table. However, note that the authors' proposed method *does* outperform DGCN_ST_LANE on MinADE_10 (0.89 in the paper vs 0.974) and so it does maintain some claim to SotA.

Argoverse 1 results were reported on the *validation* set, not the test set on the public leaderboard. This important to note because the proposed method is outperformed on the validation set by HiVT. HiVT performs significantly worse on the test set -- HiVT-128, the method described in Zhou et al., 2022, has significant hits to both mADE_6 (0.66 val vs 0.77 test) and mFDE_6 (0.96 val vs 1.17 test), and is outperformed on this test set by many methods such as DCMS (Ye et al., https://arxiv.org/abs/2204.05859, posted April 12th, 2022) Wayformer (Nayakanti et al., https://arxiv.org/abs/2207.05844, posted July 12, 2022). Certainly the transitive performance degradation from the Argoverse 1 validation set to test set of HiVT may not apply to the authors' method, but this does leave a lingering question of how well the proposed method actually stacks up relative to the state of the art. 
clarity,_quality,_novelty_and_reproducibility: Key insight is interesting, is clearly translated into the design Future Relationship Module, and it performs well on popular public datasets, so I think the content of the paper is good.

The presentation needs work. For example, h^+_x is presented in Figure 4 at the top of page 3 but not defined until the bottom of page 6, in Section 4.4. Furthermore, the English is choppy and sometimes grammatically incorrect -- the content is still intelligible, but it's often mildly distracting.

To the authors, I would recommend:
1) read through where all your variables are defined relative to figures and make sure that there's no more than ~1 page between their use and definitions
2) using an easily accessible English-based LLM e.g. GPT-3 from OpenAI, one paragraph at a time paste in your existing text and ask it to edit the text for grammar and clarity; this should be an easy way to clean up the text with little work on your end.
3) highlight in the conclusion that your Future Relationship Module is compatible with other SotA trajectory prediction methods and future work of combining them together will likely result in an even stronger method!
summary_of_the_review: Tackles the standard problem of trajectory prediction via a key insight that is actualized into a method that's competitive on popular public benchmarks. Likely will have an impact on the problem's community, as the actualized key insight forms a module that should be compatible with other SotA methods and will likely produce better performing future methods.

Conditioned on
1) expansion of the quantitative results section to include the additional methods mentioned (plus other methods deemed relevant from benchmark performance)
2) clean-up of the presentation + English

I would be happy to see this paper presented at ICLR 2023, and I think it would make a nice addition to the trajectory prediction community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, authors utilize the lane information to model the future relationship among agents for trajectory prediction. Specifically, the probability of lane-level waypoint occupancy of vehicles is predicted, and a future relation module is developed for sampling interaction edges. Finally, interaction features are obtained via message passing for decoding. Empirical results on nuScenes and Argoverse are promising.
strength_and_weaknesses: **Strengths**:  
1. Well-organized and clearly written.  
2. Promising results.  


**Weaknesses**:  
1.  Some details are missing. What is the temporal probability of passing the same lane? and how to learn interaction based on that probability? This sentence is confusing and ambiguous.  


2. Novelty is limited. Almost all the techniques are well-studied in the vehicle trajectory prediction task, i.e., the motivation of using HD map (lane information), occupancy via probability, many prior works follow this approach now. The differences between the proposed method and existing baselines are not highlighted, nor the contributions are emphasized.


3. Minor mistakes:  
(1) In Fig. 4, I believe it is Future Trajectory Samples instead of F Trajectory Samples.  
clarity,_quality,_novelty_and_reproducibility: Details are in [Weaknesses]

Clarity:
The clarity is good but with some minor mistakes.

Quality:
Clear-written.

Novelty:
Novelty is not significant.

Reproducibility:
The training details are in the supplementary material, and authors promise to release the code after paper acceptance.
summary_of_the_review: I list my concerns in [Weaknesses].
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed to leverage future relationships for vehicle trajectory prediction. Particularly, the method first predicts future waypoint occupancy on lane segments. Then, a future relationship module defines an interaction edge between every two agents based on the proximity and features of the two agents, and the interaction edges follow Gaussian mixture distributions. Conditioned on the interaction features and intention features, the decoder predicts the trajectory. The results is SoTA on nuScenes dataset and on par to SoTA on Argoverse dataset.
strength_and_weaknesses: Strength:

1. Compared with previous methods which directly use the history to predict the future, this method add an intermediate step, which first predict future waypoint occupancy on lane segments, and then derive explicit future relationships between agents. The paper show that such explicit future relationships are very helpful for predicting the final trajectory of an agent, and the model achieve SoTA results on nuScenes dataset.

2. To derive future relationship, the authors proposed to calculate inter-agent pair-wise feature vectors that follow Gaussian Mixture distributions, and ablation study shows that modeling the stochastic of this feature vector is important.

Therefore, this paper has good methodological contribution. Meanwhile, the experiments are thorough with quantitative, qualitative and ablation results.

Weaknesses:
1. The authors should really work on simplifying the notations. There are too many sup- and subscript and it is hard to follow.

2. Why is the waypoint occupancy conditioned on the sampled goal lane features? Can we integrate out the goal? The Gaussian Mixture  in FRM should be able to capture the stochasticity of goals. This will reduce inference time.

3. In section 4.4, the authors says "the goal feature (\hat{h}^{-}_{\ell})" is sampled F times from the inferred waypoint occupancy, however, in section 4.1 the waypoint occupancy is derived from the goal feature. The two statements seem contradictory.

4. z_{e}^{ij} in equation (7) already contains information about h_x^j, why do we need another element-wise product with h_x^j? Does experiment show that this is necessary?

5. Why is the Argoverse test set results not included in Table 3?

6. The method does introduce quite a lot of computation compared with other baselines such as MultiPath++. It would be better if the authors can report training and inference cost compared with baselines. 
clarity,_quality,_novelty_and_reproducibility: The paper's math notation is not clean and the authors should simplify them.

The proposed future relationship module has clear methodological contribution to the community.

The paper is quite dense and the model contains multiple sub-components (e.g. waypoint occupancy, GMVAE, FRM, etc). Without the code from authors, it is not easy to reproduce the results.
summary_of_the_review: The paper proposed to use vehicles' future relationships to improve trajectory prediction. The paper has clear science contribution and has achieved SoTA results on public dataset.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new approach for vehicle trajectory prediction with multiple vehicles.  Its main insight is that explicitly modeling future interactions between vehicles is a good intermediate step for eventual trajectory prediction, and that this can be done quite effectively by assuming lane-following to identify these possible interactions. This translates into a working algorithm, and results on nuScenes show strong performance improvements over the many prior published works on this benchmark. 

Some weaknesses in this paper are:
- Performance is only convincing on nuScenes, and the author reports on Argoverse test set results are somewhat poor, indicating that the method may rely on assumptions that don't generally hold. It is strongly recommended to the authors that they present these results in future iterations / camera-ready.
- Related to the above point, there isn't a clear analysis of weaknesses and failure cases. For example, in what cases, if any, does the lane-following assumption for interaction prediction fail, and does this then translate to trajectory prediction failures too?
- Computation concerns about the 2-phase trajectory prediction inference, raised during review phase, are yet to be fully addressed.
- Notation clutter in the original paper, and other writing weaknesses, were not addressed in the form of actual paper revisions during the response stage, but authors have promised to fix these.

We have other recommendations to the authors:
- They should more clearly distinguish from prior works like TNT / GoalNet that also use lane information, but in different contexts (not for future interaction prediction).
justification_for_why_not_higher_score: This was broderline, and the paper currently has many weaknesses, as pointed out above.
justification_for_why_not_lower_score: My final decision is influenced mainly by the soundness and simplicity of the idea and the strong results on one very well-known dataset.
Decision: Accept: poster