Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a transformer-based method for vehicle trajectory forecasting. It proposes to combine the tasks of global localization and local refinement for more accurate trajectory forecasting. On the structure side, the authors design a mechanism of motion query pair to model motion prediction as the joint optimization of the two tasks. Moreover, the interaction among agents is considered in the proposed method and collaborates to make the dense future prediction. The proposed method is demonstrated to be efficient on the large-scale Waymo Open Dataset. And an end-to-end variant of the proposed method is also provided for a broader study.
strengths_and_weaknesses: Strengths:
1. The proposed method is well elaborated and the implementation details are necessarily provided to help understand the model design.
2. The performance of the proposed MTR on the Waymo Open Motion Dataset is good, advancing the SoTA under the setting further.
3. The paper is mostly well written that I can understand the motivation, high-level intuition, and model design quickly.

Weaknesses:
1. Some details are not clear, for example, the implementation of MTR-e2e is hard to follow for me given the limited illustration at L273. 
2. Though it may not be necessary, it would be helpful to ablate the choice of query pair number for end-to-end version as well as there is a claim that "since 6 intention points are too sparse to well cover all potential future motions." may need some experiment backup.
3. Given the proposed method stresses on the design of query pair, it would be helpful for us to better understand the efficiency of this by making an ablation study of using only one type of query or both.

questions: I can understand the technical design of the proposed method while I have some concerns as mentioned in the "Weaknesses" part above. It would be helpful if the authors can address them with more clarification, with which I can more confidently assess the significance of this paper and adjust my rating.
limitations: Some technical design of the proposed method is explained at the end of the draft. I don't recognize more potential negative societal impact or limitations of this paper.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an decoder method for motion prediction task, which refines different modes with the static prior and dynamic attention. Its performance on the Waymo Open Motion Dataset is impressive, which demonstrates the effectiveness of iteratively refine the prediction (similar to DETR/DAB-DETR).
strengths_and_weaknesses: Strengths
* The idea of iteratively refining the prediction by Transformer is novel in the motion prediction area.
* The performance on Waymo Motion Dataset is great.
* The ablation experiments is well-organized and convincing

Weakness
* The proposed method is only evaluated on one dataset. It would be more convincing if the experiments could be done on other large scale datasets. However, considering the Waymo Open Motion is fairly large, I think it is okay if there is no enough time/resource to try other datasets.
* Some parts of the proposed methods is not clearly described in the manuscript, which is understandable considering the space limit. 
* The proposed method has similarities with the DAB-DETR in the objection detection area. I think it would better position this work if the authors could discuss about the proposed method's relation with recent DETR related works.
questions: #### Question
* The PE module is first introduced in the NLP field so that tokens' position information could be considered (Transformer is a set operator which ignores the postion information).  However, in the field of motion prediction, the feature itself includes the postion information (coordinate). I am wondering why the proposed method uses the PE + MLP instead of just MLP. Is there any experiments about its effectiveness?
* In equation (6), it uses C^{j-1}. How to initialize it in the first layer is not described.
* I noticed that Waymo Motion Challenge 2022 recently finished. Authors may discuss about those top-ranked methods including golfer, MPA, HDGT, etc, which could mkae the paper more up-to-date.
limitations: The limitation part is ok.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The work addresses the motion forecasting problem for autonomous driving. The authors introduce a transformer-based framework (MTR) that works in the following ways, as highlighted in the contribution section of the paper:

- Separates the modeling of the global intention from local movement refinement in the trajectories in the transformer framework. The predictor is inspired by DETR. 
- Interaction modeling between agents via an auxiliary dense prediction task, essentially letting the model to predict the future directly (I didn't fully understand that part... see below)
- SOTA results on the Waymo dataset. Ranked #1 among results without using ensembles. 

Since I don't understand the 2nd contribution, I cannot recommend accept at this point. Looking forward to understanding it post-rebuttal!

POST REBUTTAL UPDATE: I understand the contribution better now. In line with other reviews' feedback, I'll change the rating from 4 -> 7, and recommend an accept.

strengths_and_weaknesses: Strengths:

Strong results: Significant improvement over the SOTA. I checked the leaderboard on Waymo Open Dataset, and verified the claims. 

Somewhat novelty: Using DETR for motion forecasting is not that new anymore. I reviewed several papers for CVPR'22 and ECCV'22 that contained similar ideas. But those should be considered concurrent work. Additionally, the hierarchical separation of intent vs. fine control adds an additional hint of novelty, though the hierarchical separation by itself is not novel either [9, 51]. 

Good ablation: The paper contains ablation for all the contributions and novelties in their methods. This is great!

Weaknesses:

- Unclear writing: Abused notations, missing descriptions of variables, figures that are not self-contained. See below under "questions".

- Insufficient experiments (minor): The only results are on the Waymo dataset. There are other popular datasets such as Argoverse and nuScenes. Having strong results on a secondary dataset would make the representation a lot stronger. 
questions: line 73: Why not also using ensemble to boost the accuracy a bit to achieve top results? 

Figure 1: The figure is far from self explaining. What do the small road illustrations above the dense future prediction and dynamic map construction say? What are K, T, N and D (I know its in the text, but it doesn't hurt to say it here)? Where do the input "predicted trajectories", "map query" and "query content feature" come from? The figure is way too complicated, and I don't feel rewarded for looking closely....

Eq 2, ln 164: There are equations like   X = f(X), e.g. A = MLP([A, A_t])   what does this even mean? please use subscripts to make it clear that the variable in LHS and RHS are not the same. 

ln 237: where does the auxiliary loss L_a come from? just write equations and not words. And if its just an L1 loss against the GT. I don't understand the intuition why it helps with interaction. It seems like its just like a multi-head network where you supply the GT at several layers of the network. 
limitations: Yes, but it didn't hit the mark. 

The limitation section is about high-level limitation, like "the method is not great for long tail behaviors", not a wish list of future works.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, a Motion TRansformer (MTR) framework is proposed for the motion prediction task, including marginal and joint motion prediction. Specifically, motion query pairs are designed for global intention localization and local movement refinement, which takes advantage of both goal-based methods and the regression methods. Experiments on Waymo Open Dataset indicate the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:  
- Strong motivation and well-organized.
- Promising results.


Weaknesses: (See Questions for details) 
- Some insights/details are not clear.
- Lack of some experiments.

questions: Concerns: 
1. Overall, it is a paper for motion prediction challenge with promising results and clear presentations. I assume it is the initial version of the final 1st place work [1], sharing very similar methods and almost the same framework, thus I would encourage authors to update the paper with [1]. 
2. The most concern is that all the strategies are well-studied in this community, i.e., the transformer idea, scene context and interaction modeling, etc. It would be better if more insights could be included for model design, and highlight the differences/contributions.
3. In Line 134-135, how to pad zeros for trajectories that have less than $t$ frames? In the front/back? Some details are missing here.
4. Some missing references when referring to future interactions, some works [2-4] have been proposed to argue the future interaction point and to consider such future interactions.

Minors:  
1. Line 56, I believe it is 'empirical' instead of 'experimentally'.
2. In References, the names of conferences or journals, i.e. abbreviation or full name, are not consistent.  

I am happy to improve my rating if these concerns could be addressed.


[1] https://storage.googleapis.com/waymo-uploads/files/research/MotionPred/MotionPrediction_MTRA.pdf  
[2] EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning. NeurIPS 2020  
[3] Tra2Tra: Trajectory-to-Trajectory Prediction With a Global Social Spatial-Temporal Attentive Neural Network. ICRA 2021  
[4] StarNet: Pedestrian Trajectory Prediction using Deep Neural Network in Star Topology. IROS 2019  


limitations: There is a limitation section in the main body of the paper.
review_title: null

Meta-review: metareview: This paper proposes to model traffic vehicles using a transformer-based architecture for iteratively refining multimodal trajectory predictions. While the method is related to and builds upon several similar works in the area, it does also introduce some interesting new components such as the iterative refinement and the dynamic attention. Further, the strength of the experimental results from the combined system alone makes this paper important for researchers working in these areas: the method achieves the state of the art for trajectory prediction on two very widely used datasets (Waymo and Argoverse), compared to published leaderboards. All four reviewers unanimously agree that this paper is above the bar for acceptance, and I concur. 
confidence: Certain
Decision: Accept