Motion Transformer with Global Intention Localization and Local Movement Refinement
Authors: Shaoshuai Shi
Li Jiang
Dengxin Dai
Bernt Schiele
Venue: NeurIPS 2022 Accept (2022)
Keywords: Motion Prediction
Autonomous Driving
Transformer

Abstract: Predicting multimodal future behavior of traffic participants is essential for robotic vehicles to make safe decisions. Existing works explore to directly predict future trajectories based on latent features or utilize dense goal candidates to identify agent's destinations, where the former strategy converges slowly since all motion modes are derived from the same feature while the latter strategy has efficiency issue since its performance highly relies on the density of goal candidates. In this paper, we propose the Motion TRansformer (MTR) framework that models motion prediction as the joint optimization of global intention localization and local movement refinement. Instead of using goal candidates, MTR incorporates spatial intention priors by adopting a small set of learnable motion query pairs. Each motion query pair takes charge of trajectory prediction and refinement for a specific motion mode, which stabilizes the training process and facilitates better multimodal predictions. Experiments show that MTR achieves state-of-the-art performance on both the marginal and joint motion prediction challenges, ranking 1st on the leaderbaords of Waymo Open Motion Dataset. Code will be available at https://github.com/sshaoshuai/MTR.

Motion Transformer with Global Intention Localization and Local Movement Refinement
Shaoshuai Shi, Li Jiang, Dengxin Dai, Bernt Schiele Max Planck Institute for Informaties, Saarland Informatics Campus {sshi, lijiang, ddai, schiele) @mpi tapp--iff-a. -inf mmg.
Abstract
Predicting multimodal future behavior of traffic participants is essential for robotic vehicles to make safe decisions. Existing works explore to directly predict future trajectories based on latent features utilize dense goal candidates identify agent's ' destinations, where the former strategy converges slowly since all motion modes are derived from the same feature while the latter strategy has efficiency issue since its performance highly relies on the density of goal candidates. In this paper, we propose Motion TRansformer (MTR) framework that models motion prediction as the joint optimization of global intention localization and local movement refinement. Instead of using goal candidates, MTR incorporates spatial intention priors by adopting small set of learnable motion query pairs. Each motion query pair takes charge of trajectory prediction and refinement for a specific motion mode, which stabilizes the training process and facilitates better multimodal predictions. Experiments show that MTR achieves state-of-the-arr performance both the marginal and joint motion prediction challenges, ranking 1s on the leaderboards of Waymo Open Motion Dataset. Code will be available at https://gii thub. com/ waam//shhsshuaiMMR.
Introduction
Motion forecasting is a fundamental task of modern autonomous driving systems. It has been 57 351 as it crucial for robotic vehicles to understand driving scenes and make safe decisions. Motion forecasting requires to predict future behaviors of traffic participants by jointly considering the observed agent states and road maps, which is challenging due to inherently multimodal behaviors of the agent and complex scene environments.
receiving increasing attention recent years [19 46 29
To cover all potential future behaviors of the agent, existing approaches mainly fall into different lines: the goal-based methods and the direct-regression methods. The goal-based methods 19 [1] adopt dense goal candidates cover all possible destinations of the agent, predicting the probability of each candidate being a real destination and then completing the full trajectory for each selected candidate. Although these goal candidates alleviate the burden of model optimization by reducing trajectory uncertainty, their density largely affects the performance of these methods: fewer candidates will decrease the performance while more candidates will greatly increase computation and memory directly predict a set of trajectories based on the encoded agent feature, covering the agent's 's future behavior adaptively. Despite the flexibility in predicting broad range of agent behaviors, they generally converge slowly as various motion modes are required to be regressed from the same agent feature without utilizing any spatial priors. They also tend to predict the most frequent modes of training data since these frequent modes dominate the optimization of the agent feature. this paper, we present a unified framework, namely Motion TRansformer (MTR), which takes the best of both types of methods.
cost. Instead of using goal candidates, the direct-regression methods B5] 47
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In our proposed MTR, we adopt small set of novel motion query pairs to model motion prediction as the joint optimization of two tasks: The first global intention localization task aims to roughly identify agent's intention for achieving higher efficiency, while the second local movement refinement task aims to adaptively refine each intention's 's predicted trajectory for achieving better accuracy. Our approach not only stabilizes the training process without depending on dense goal candidates but also enables flexible and adaptive prediction by enabling local refinement for each motion mode.
Specifically, each motion query pair consists of two components, i.e., a static intention query and a dynamic searching query. The static intention queries are introduced for global intention localization, where we formulate them based on small set of spatially distributed intention points. Each static intention query is the learnable positional embedding of an intention point for generating trajectory of a specific motion mode, which not only stabilizes the training process by explicitly utilizing different queries for different modes, but also eliminates the dependency on dense goal candidates by requiring each query to take charge of large region. The dynamic searching queries are utilized for local movement refinement, where they are also initialized as the learnable embeddings of the intention points but are responsible for retrieving fine-graines local features around each intention point. For this purpose, the dynamic searching queries are dynamically updated according to the predicted trajectories, which can adaptively gather latest trajectory features from a deformable local region for iterative motion refinement. These two queries complement each other and have been empirically demonstrated their great effectiveness in predicting multimodal future motion. Besides that, also propose a dense future prediction module. Existing works generally focus on modeling the agent interaction over past trajectories while ignoring the future trajectories' interaction. To compensate for such information, we adopt a simple auxiliary regression head to densely predict future trajectory and velocity for each agent, which are encoded as additional future context features to benefit future motion prediction of our interested agent. The experiments show that this simple auxiliary task works well and remarkably improves the performance of multimodal motion prediction.
Our contributions are three-fold: (1) We propose a novel motion decoder network with a new concept of motion query pair, which adopts two types of queries to model motion prediction as joint optimization of global intention localization and local movement refinement. It not only stabilizes the training with mode-specific motion query pairs, but also enables adaptive motion refinement by iteratively gathering fine-grained trajectory features. (2) We present an auxiliary dense future prediction task to enable the future interactions between our interested agent and other agents. It facilitates our framework to predict more scene-compliant trajectories for the interacting agents. (3) By adopting these techniques, we propose MTR framework that explores transformer encoder-decoder structure for multimodal motion prediction. Our approach achieves state-of-the-art performance on both the marginal and joint motion prediction benchmarks of Waymo Open Motion Dataset (WOMD) [14), outperforming previous best ensemble-free approaches with +8.48% mAP gains for marginal motion prediction and - .98% mAP gains for joint motion prediction. As of 19 May 2022, our approach ranks 1st on both the marginal and joint motion prediction leaderboards of WOMD.
2 Related Work
Motion Prediction for Autonomous Driving, Recently, motion prediction has been extensively studied due to the growing interest in autonomous driving, and it typically takes road map and
agent history states as input. To encode such scene context, early works 36 31 12
60 B
typically rasterize them into image so as to be processed with convolutional neural networks (CNNs). LaneGCN 27 builds a lane graph toscalability capture map topology. VectorNet [16] is widely adopted by recent works [19 43 35 47 due to its efficiency and scalability, where both road maps and agent trajectories are represented as polylines. We also adopt this vector representation. but instead of building global graph of polylines, we propose to adopt transformer encoder on local connected graph, which not only better maintains input locality structure but also is more memory-efficient to enable larger map encoding for long-term motion prediction.
Given the encoded scene context features, existing works explore various strategies to model multi- 44 40) propose to generate set of trajectory samples
modal future motion. Early works [ 20 39
to approximate the output distribution. Some other works p 21 33 37 41 parameterize multi-
modal predictions with Gaussian Mixture Models (GMMs) to generate compact distribution. HOME series [18 generate trajectories with sampling on a predicted heatmap. IntentNet 7 considers intention prediction as a classification with 8 high level actions, while (2) proposes region-based

---

(b)
g-
XxTx2
Xx +
XxD
Prudicted Trajectory
Quiry Ojulating
GMM Prediction
Pelyline Fronder
Key
Content Feature
D ense Puture
Prediction
Dynamic Map
Lo Collection
Value
Transformer Deroder Layer
Query Fisitio
Quary Consent
Transformer Encoder
Xx1x2
Xx 2)
XDD
Graph
Predicted Trajettoyy Initialization
Morton Queryy Pair
Query Content Initidiinaion ion
-
Agent
Prodicted trajectury
Trojectary-speiilic map
Interssted
feature
agest
Figure I: The architecture of MTR framework. (a) indicates the dense future prediction module, which predicts a single trajectory for each agent (e.g., drawn yellow dashed curves in the above of (a)). (b) indicates the dynamic map collection module, which collects map elements along each predicted trajectory (e.g., drawn the shadow region along each trajectory in the above part of (b)) to provide trajectory-specifee feature for motion decoder network. (c) indicates the motion decoder network, where K is the number of motion query pairs, T is the number of future frames, D is hidden feature dimension and N is the number of transformer decoder layers. The predicted trajectories, motion query pairs, and query content features are the outputs from last decoder layer and will be taken as input to next decoder layer. For the first decoder layer, both two components of motion query pair are initialized as predefined intention points, the predicted trajectories are replaced with the intention points for initial map collection, and query content features are initialized as zeros.
training strategy. Goal-based methods 61 estimate several goal points of the agents and then complete full trajectory for each goal.
40
15
30 are another kinds of models where they first
Recently, the large-scale Waymo Open Motion Dataset (WOMD) [4] is proposed for long-term motion prediction. To address this challenge, DenseTNT [19] adopts a goal-based strategy to classify endpoint of trajectory from dense goal points. Other works directly predict the future trajectories based on the encoded agent features [35] or latent anchor embedding 47). However, the goal-based strategy has the efficiency concern due to a large number of goal candidates, while the direct- regression strategy converges slowly as the predictions of various motion modes are regressed from the same agent feature. In contrast, approach adopts a small set of learnable motion query pairs, which not only eliminate the large number goal candidates burden by utilizing mode-specific motion query pairs for predicting different motion modes.
also alleviate the optimization
Some very recent works [45] 23 22 also achieve top performance on WOMD by exploring Mix-and- Match block [45), variant of MultiPath++ [23] or heterogencous graph (22). However, they generally focus on exploring various structures for encoding scene context, while how design a better motion decoder for multimodal motion prediction is still underexplored. In contrast, our approach focuses on addressing this challenge with a novel transformer-basse motion decoder network.
has been widely applied in natural language processing 58). Our approach is inspired by DETR [4] and its follow-up especially DAB-DETR (28). where the object query is considered as the positional embedding of spatial anchor box. Motivated by their great success in object detection, introduce novel concept of motion query pair to model multimodal motion prediction with prior intention points, where each motion query pair takes charge of predicting a specific motion mode and also enables iterative motion refinement by combining with transformer decoders.
Transformer.
Transformer
48
and computer vision 13
50
works (63) 32 56
25
28
10
59
3 Motion TRansformer (MTR)
We propose Motion l'Ranffrmeer (MTR), which adopts novel transformer reccoder-decode structure with iterative motion refinement for predicting multimodal future motion. The overall structure illustrated in Figure [ In Sec. 3.1, we introduce our encoder network for scene context modeling. In we present motion decoder network with a novel concept of motion query pair for predicting multimodal trajectories. Finally, in Sec. 3.3 we introduce the optimization process of our framework.
Sec. 3.2
3

---

3.1 Transformer Encoder for Scene Context Modeling
The future behaviors of the agents highly depend on the agents' interaction and road map. To encode such scene context, existing approaches have explored various strategies by building global interacting graph [16] [9] or summarizing map features to agent aggent-wise features [35 471. We argue that the locality structure is important for encoding scene context, especially for the road map. Hence, propose transformer encoder network with local self self-attention to better maintain such structure information.
Input representation. We follow the vectorized representation [6] organize both input trajectories and road map as polylines. For the motion prediction of a interested agent, we adopt the agent-centric strategy 61 19 47 that normalizes all inputs the coordinate system centered at this agent. Then, a simple polyline encoder is adopted to encode each polyline as an input token feature for the transformer encoder. Specifically, we denote the history state of Na agents as Ais NN. Xt. where is the number of history frames, C. is the number of state information (e.g., location, heading angle and velocity). and we pad zeros at the positions of missing frames for trajectories that have less than / frames. The road map is denoted as Min E N. RR XnxCm, polylines, is the number of points in each polyline and Cm is the number of attributes of each point (e.g., location and road type). Both of them are encoded by PointNet-like
where Nm, is the number of map
(38 polyline encoder as:
Ap á (MLP(Am)), Mp e (MLP(M))),
(1)
where (L is multilayer perceptron network, and is max-pooling summarize each polyline features as agent features Ap E R Na x and map features Mp E IR Nm x with feature dimension D.
Scene context encoding with local transformer encoder. important for motion prediction. For example, the relation of two parallel lanes is important for modelling the motion of changing lanes, but adopting attention on global connected graph equally considers relation of all lanes. In contrast, we introduce such prior knowledge to context encoder by adopting local attention, which better maintains the locality structure and are more memory memory-efficient. Specifically, the attention module of j-th transformer encoder layer can be formulated as:
The local structure of scene context is
GO MultiHaaAAtt (query=G*~14 + PBgs-1, key=x(Gr-1) + PEx valuc=x valu)=x(G'-1)), (2)
where MultiHeadAttm(;--)) the multi-head attention layer 488. GO [Ap. Mpp E N.N.+NN...XD concatenating the features of agents and map, and k(-) denotes k-nearest neighbor algorithm to find k closest polylines for each query polyline. PE denotes sinusoidal position encoding of input tokens, where we utilize the latest position for each agent and utilize polyline center for each map polyline. Thanks to such local self- our framework can encode a much larger area of scene context.
The encoder network finally generates both agent features Apust E N. D and map features M E xD. which are considered as the scene context inputs of the following decoder network.
Dense future prediction for future interactions. Interactions with other agents heavily affect behav- iors of our interested agent, and previous works propose to model the multi-agent interactions with hub-host based network [64], dynamic relational reasoning [26). social spatial-tempont network (55). etc. However, most existing works generally focus on learning such interactions over past trajectories while ignoring the interactions of future trajectories. Therefore, considering that the encoded features A have already learned rich context information of all agents, we propose to densely predict both future trajectories and velocities of all agents by adopting a simple regression head on A:
S = MLP(Agus),
(3)
where S, N. includes future position and velocity of each agent at time step i, and T is the number of future frames to be predicted. The predicted trajectories S1.T are encoded by adopting the same polyline encoder as Eq. ( to encode the agents' future states as features Atsure E g.N.. which are then utilized to enhance the above features A by using a feature concatenation and three MLP layers as A = MLP((Apesn (LLP(AAss:AAmuurr)) Afuure)). This auxiliary task provides additional future context information to the decoder network, facilitating the model to predict more scene-compliant future trajectories for the interested agent. The experiments in Table demonstratess that this simple and light-weight auxiliary task can effectively improve the performance of multimodal motion prediction.
3.2 Transformer Decoder with Motion Query Pair
Given the scene context features, transformer-based motion decoder network adopted for multi- modal motion prediction, where propose motion query pair model motion prediction as the

---

joint optimization of global intention localization and local movement refinement. Each motion query pair contains two types of queries, i.e., static intention query and dynamic searching query, for conducting global intention localization and local movement refinement respectively. As shown in Figure 2 our motion decoder network contains stacked transformer decoder layers for iteratively refining the predicted trajectories with motion query pairs. Next, we illustrate the detailed structure.
Global intention localization aims to localize agent's potential motion intentions in an efficient and effective manner. We propose static intention query to narrow down the uncertainty of future trajectory by utiliizig different intention queries for different motion modes. Specifically, we generate K. representative intention points 7 XRx2x2 by adopting k-means clustering algorithm on the endpoints of ground-truth (GT) trajectories, where each intention point represents an implicit motion mode that considers both motion direc-
ct (Kx0)
(3Cx2)
OK x2)
Upubrinng
(XxXx66
Connat Futturs
tion and velocity. We model each static intention query as the learnable positional embedding of the intention point as:
Add Norm
GMM Prediction
Multi-Hten Attention
Q MLP (PE(I)),
(4)
XN
Postion Embeeding
Add Norm
where PE(.) is the sinusoidal position encoding, and Q, E I.DD Notably, each intention query takes charge of predicting trajectories for a spe- cific motion mode, which stabilizes the training process and facilitates predicting multimodal tra- jectories since each motion mode has their own learnable embedding. Thanks to their learnable and adaptive properties, we only need small number of queries (e.g., 64 queries in our set- ting) for efficient intention localization, instead of using densely-place pllaced goal candidates to cover the destinations of the agents.
Multi-llead Self-Attention
Cli (XxD)
J: OX (2)
x/ (X)
Quary Contentt Fosture
Nation Qurry Pair
Stattc Intention Query
Dynamic Searching Query
Figure The network structure of our motion decoder network with motion query pair.
Local movement refinement aims to complement with global intention localization by iteratively gathering fine one-gnaieed trajectory features for refining the trajectories. We propose dynamic searching query to adaptively probe trajectory features for each motion mode. Each dynamic searching query is also the position embedding of spatial point, which is initialized with its corresponding intention point but will be dynamically updated according to the predicted trajectory in each decoder layer. Specifically, given the predicted future trajectories Yi,T {Y?' E RRxx2 i== = 1.... T3 in j-th decoder layer, the dynamic searching query of (j 1)-th decoder layer is updated as follows:
OS ++ = MLP (PE()))).
(5)
As shown in Figure for each motion query pair, we propose dynamic map collection module to extract fine-grained trajectory features by querying map features from a trajectory-aligne local region, which is implemented by collecting L polylines whose centers are closest to the predicted trajectory. As the agent's 's behavior largely depends on road maps, this local movement refinement strategy enables to continually focus on latest local context information for iterative motion refinement.
Attention module with motion query pair. In each decoder layer, static intention query is utilized to propagate information among different motion intentions, while dynamic searching query is utilized to aggregate trajectory-specific features from scene context features. Specifically, we utilize static intention query as the position embedding of self-attention module follows:
C2
MultiHeadAAtmnuuerryyyr
+ +Qr, key= + +Q,, value=Q1),
(6)
where CV-1 IR KD is query content features from (j 1)-th decoder layer, Co is initialized to zeros, and Cli E Cx is the updated query content. Next, we utilize dynamic searching query as query position embedding of cross attention to probe enjjectory-speciff features from the outputs of encoder. Inspired by [32 28), , we concatenate content features and position embedding for both query and key to decouple their contributions to the attention weights. Two arossaatenniinn modules
5

---

A
Static Intention Query
* Dynamic Searching Query
Predicted trajectory
Trajectory-specilic map features
0.3
A
Prediction from jayer -1
Motion Query
Motion
Motion Query
Prediction of layer
Figure 3: The illustration of dynamic map collection module for iterative motion refinement.
adopted separately for aggregating features from both agent features A and map features M as:
C/ MultiHeaadAtm((quey=CCC Q3), key=|A, PEa], value=A),
C MultiHeaaAAuun(queey=CCC 2/ key=|a(M), PEa(()) value=a(M)),
C MLP((CA CMr)
where [ indicates feature concatenation, (MM) is the aforementioned dynamic map collection module to collect T. trajectory-aligned map features for motion refinement. Note that for simplicity, in Eq. (6) and 77. we omit the residual connection and feed-forward network in transformer layer [48]
Finally, C E Cx is the updated query content features for each motion query pair in j-th layer.
Multimodal motion prediction with Gaussian Mixture Model. For each decoder layer, we append prediction head to C' for generating future trajectories. As the behaviors of the agents are highly multimodal, we follow g [7] to represent the distribution of predicted trajectories with Gaussian Mixture Model (GMM) at each time step. Specifically, for each future time step i E (1,.... 7), we predict the probability p and parameters (Jar, Hly ox. yy: p) of each Gaussian component as follows
Z = MLP(C").
(8)
where Z E Rxx6 includes K. Gaussian components. (1 Ozi My Tyi p) with probability distri- bution PI:K. The predicted distribution agent's position at time step i be formulated as:
K
(o(
Σ pk No Ma, 02:0y My: uy p)-
(9)
where (o) is the occurrence probability the agent at spatial position o E R The predicted trajectories y
i.T can be generated by simply extracting the predicted centers Gaussian components.
3.3 Training Losses
Our model is trained end-to-end with two training losses. The first auxiliary loss is L1 regression loss to optimize the outputs of Eq. (3) For the second Gaussian regression loss, we adopt negative log- likelihood loss according to Eq. ( maximum the likelihood of ground-truth trajectory. Inspired by (7) we adopt hard-assignment strategy that selects one closest motion query pair as positive optimization, where the selection is implemented by calculating the distance between each intention point and the endpoint of GT trajectory. The Gaussian regression loss is adopted in each decoder layer, and the final loss is the sum of the auxiliary regression loss and all the Gaussian regression loss with equal loss weights. Please refer to appendix for more loss details.
Gaussian component
4 Experiments
4.1 Experimental Setup
Dataset and metrics. evaluate our approach on the large-scale Waymo Open Motion Dataset (WOMD) [14,, which mines interesting interactions from real-world traffic scenes and is currently

---

Table 1: Performance comparison of marginal motion prediction the validation and test set of Waymo Open Motion Dataset. t: The results are shown italic for reference since their performance achieved with model ensemble techniques. We only evaluate our default setting MTR on the test set by submitting to official test server due to the limitation of submission times of WOMD.
Method
Reference
minADE +
minFDE
Miss Rate +
mAP
MotionCNN 24
CVPRR 2021
0.7400
1.4966
0,2091
0.2136
ReCoAl 62
CVPRw 2021
0.7703
1.6668
0.2437
0.2711
DenseTNT
ICCV 2021
1.0387
1.5514
0.1573
0.3281
Test
SceneTrunsformer
ICLR 2022
0.6117
121116
0.1564
0.2788
MTR (Ours)
0.6050
1.2207
0.1351
0,4129
MultiPath++
ICRA 2022
0.5557
1,1577
0.1340
0,4092
MMTR-Advanced-ems (Ours)
0,5640
1.1344
0.1160
0,4492
MIR (Ours)
0.6046
12251
0.1366
0,4164
Val
MTR-c2e (Ours)
0,5160
1,0404
0.1234
0,3245
MTR-ens (Ours)
0.5686
1.1534
0.1240
0.4323
MTR-Advanced-ems (Ours)
0.5597
1.1299
0.1167
0.4551
Table 2: Performance comparison of joint motion prediction on the interactive validation and test set of Waymo Open Motion Dataset.
Method
Reference
minADE
minFDE 1
Rate
mAP
Waymo LSTM baseline [14
ICCV 2021
1.9056
5.0278
0.7750
0.0524
HcatlRm4
CVPRw 2021
1.4197
3,2595
0.7224
0.0844
AIR 54
CVPRw 2021
1.3165
2.7138
0.6230
0.0963
Test
Scene Transfrrrfrrem
35
ICLR 2022
0,9774
2.1892
0,4942
0,1192
M21 [43
CVPR 2022
1.3506
2.8325
0,5538
0.1239
MTR (Ours)
0,9181
2.0633
0,4411
0.2037
Val
MTR (Ours)
0,9132
2.0536
0,4372
0,1992
the most diverse interactive motion dataset. There are two tasks in WOMD with separate evaluation metrics: (1) The marginal motion prediction challenge that independently evaluates the predicted motion of each agent (up to 8 agents per scene). (2) The joint motion prediction challenge that needs to predict the joint future positions of interacting agents for evaluation. Both of them provide second of history data and aim to predict 6 marginal or joint trajectories of the agents for seconds into the future. There are totally 487k: training scenes, and about 44k validation scenes and 44k testing scenes for each challenge. We utilize the official evaluation tool to calculate the evaluation metries, where the mAP and miss rate are the most important ones as in the official leaderboard
Implementation details. For the context encoding, we stack 6 transformer encoder layers. The road map is represented as multiple polylines, where each polyline contains up to 20 points (about 10m in WOMD). We select Nm = 768 nearest map polylines around the interested agent. The number of neighbors in encoder's local self-attention set to 16. The encoder hidden feature dimension set D 256. For the decoder modules, we stack 6 decoder layers. L is set to 128 to collect the closest map polylines from context encoder for motion refinement. By default, we utilize 64 motion query pairs where their intention points are generated by conducting k-means clustering algorithm on the training set. To generate future trajectories for evaluation, we use non-maximun suppression (NMS) select top 6 predictions from 64 predicted trajectories by calculating the distances between their endpoints, and the distance threshold set as 2.5m. Please refer to Appendix for more details.
Training details. Our model trained in end-to-end manner by AdamW optimizer with learning rate of 0.0001 and batch size of 80 scenes. We train the model for 30 epochs with 8 GPUs (NVDIA RTX 8000), and the learning rate decayed by factor of 0.5 every 2 epochs from epoch 20. The weight decay is set as 0.01 and we do not use any data augmentation.
MTR--22e for end-to-end motion prediction. We also propose an end-to-end variant of MTR, called MTR-e2e, where only 6 motion query pairs are adopted so as to remove NMS post processing. In the training process, instead of using static intention points for target assignment as in MTR, MTR-e2e selects positive mixture component by calculating the distances between its predicted trajectories and the GT trajectory, since 6 intention points are too sparse to well cover all potential future motions.
4.2 Main Results
Performance comparison for marginal motion prediction. Table shows our main results for marginal motion prediction, our MTR outperforms previous ensemble-fres approaches [19] 35) with
7

---

Table 3: Effects of different components in MTR framework. All models share the same encoder network. "latent learnable embedding" indicates using 6 latent learnable embeddings as queries of decoder network, and "iterative refinement" indicates using 6 stacked decoders for motion refinement.
Global Intention
htraaiiv
Local Movement Dense Future
minADE
minFDE
Miss Rate
mAP
Localization
Refinement
Refinement
Prediction
Latent learnable embedding
X
X
0.6829
1.4811
0.2128
0.2633
Static intention query
0.7036
1.4651
0.1845
0.3059
Static intention query
X 0.6919
1.4217
0.1776
0.3171
Statie intention query
0.6833
1.4099
0.1756
0.3234
Static intention query
V 0.6735
1.3847
0.1706
0.3284
Static intention query
V
0.6697
1.3712
0.1668
0.3437
remarkable margins, increasing the mAP by +8.48% and decreasing the miss rate from 15.64% to 13.51%. In particular, our single-model results of MTR also achieve better mAP than the latest work MultiPath+ [47, where it uses a novel model ensemble strategy that boosts its performance.
Table D also shows the comparison of MTR variants. MTR-e2e achieves better minADE and minFDE by removing NMS post-proe post-procsssing, while MTR achieves better mAP since it learns explicit meaning of each motion query pair that produces more confident intention predictions. We also propose simple model ensemble strategy to merge the predictions of MTR and MTR-e2e and utilize NMS to remove redundant predictions (denoted as MTR-ens), and takes the best of both models and achieves much better mAP. By adopting such ensemble strategy to 7 variants of our framework (e.g., more decoder layers, different number of queries, larger hidden dimension). our advanced ensemble results (denoted as ATTR-Addanced-ens achieve best performance the test set leaderboard.
Performance comparison for joint motion prediction. To evaluate our approach for joint motion prediction, combine the marginal predictions of two interacting agents into joint prediction as in 66 14 43
where we take the top joint predictions from 36 combinations of these two agents. The confidence of each combination is the product of marginal probabilities. Table 2 shows that our approach outperforms state-of-the-arrs [35 43 with large margins on all metrics. Particularly, MTR boosts the mAP from 12.39% to 20.37% and decreases the miss rate from 49.42% to 44.11% The remarkable performance gains demonstrate the effectiveness of MTR for predicting scene-consistent future trajectories. Besides that, we also provide some qualitative results in Figure to show our predictions in complicated interacting scenarios.
As of May 19, 2022, our MTR ranks 1st the motion prediction leaderboard of WOMD for both two challenges [1]. Our approach with more ensembled variants of MTR (i.e.. Advanedvd ens) also won the champion of Motion Prediction Challenge in Waymo Open Dataset Challenge 2022 53
The significant improvements manifest the effectiveness of MTR framework.
4.3 Ablation Study
We study the effectiveness of each component in MTR. For efficiently conducting ablation experi- ments, we uniformly sampled 20% frames (about 97k scenes) from the WOMD training set according to their default order, and empirically find that it has similar distribution with the full training set. All models are evaluated with marginal motion prediction metric on the validation set of WOMD.
Effects of the motion decoder network. We study the effectiveness of each component in decoder network, including global intention localization, iterative refinement and local movement refinement. Table 3 shows that all components contributes remarkably the final performance in terms of the official ranking metric mAP. Especially, our proposed static intention queries with intention points achieves much better mAp (i.e., +4.26%) than the latent learnable embeddings thanks to its mode-specific querying strategy, and both the iterative refinement and local movement refinement strategy continually improve the mAP from 30.59% 32.34% by aggregating more fine-grained trajectory features for motion refinement.
Effects of dense future prediction. Table shows that our proposed dense future prediction module significantly improves the quality of predicted trajectories (e.g. +1.7%% mAP), which verifies that future interactions of the agents' trajectories are important for motion prediction and our proposed strategy learn such interactions to predict more reliable trajectories.
8

---

0.38
Static intertoo Points Predictions
Static Intention
Table 4: Effects of local self-attention in transformer encoder. "#polyline" is the number of input map polylines used for context encoding, and large number of polylines indicate that there is larger map context around the interested agent.
0.36
0.34
0.26
0.30
0.4
0.28
Mics
0.2
0.26
0.20
0.24
"OOM" indicates running out of memory.
0.22
Attcation
#Polyline
minADE
minFDE
MR 1 MAP T
6 16 32 64
100
6 16 32
64 100
Global
256
0.683
1,4011
0.1217 0.3295
Number of Motion Query Pairs
Number Motion Query Pairs
Global
512
0.6783
1,4018
0.1716 0.3280
Figure 4: MTR framework with different number of mo- tion query pairs, and two different colored lines demon- strate different strategies for selecting the positive mix- ture component during training process.
Global
768
00M
00M
00M 00M
1,3835
Local
512
0.6707
1.3749
0.1670 0.3392
Local
768
0.6697
1.3712
0.1668 0.3437
Local
1024
0.6757
1.3782
0.1663 0.3452
1+8k high
t-0s low
(a) V2 is passing the intersection turn left with high speed. Our model predicts multimodal behaviors for any case, prodicted to yield
(b) P2 is passing the road through the crosswalk while V1 the right-tum lane right. yleld for
(6) Our model predicts multimodal behaviors for V1:go striight and turn right, since it still has a distancee the intersection. predicted to yield for Vi when turning
predicted
V1 moving towards intersection.
Figure 5: Qualitative results of MTR framework on WOMD. There are two interested agents in each scene (green rectangle), where our model predicts 6 multimodal future trajectories for each of them. For other agents (blue rectangle), a single trajectory is predicted by dense future prediction module. We use gradient color to visualize the trajectory waypoints at different future time step, and trajectory confidence is visualized by setting different transparent. Abbreviation: Vehicle (V), Pedestrian (P).
Effects of local attention for context encoding, Table 4 shows that by taking the same number of map polylines as input, local self-attentin attentien in transformer encoder achieves better performance than global attention (i.e., +0.77% mAP for 256 polylines and +1.12% mAP for 512 polylines), which verifies that the imput local structure important for motion prediction and introducing such prior knowledge with local attention can benefit the performance. More importantly, local attention is more memory-efficient and the performance keeps growing when improving the number of map polylines from 256 to 1,024, while global attention will run out of memory due to its quadratic complexity.
Effects of the number of motion query pairs with different training strategies. As mentioned before, during training process, MTR and MTR-e2c adopt two different strategies for assigning positive mixture component, where MTR depends on static intention points (denoted as a) while MTR-e2e utilizes predicted trajectories (denoted as 8). Tigure/fivestiieeee of motion query pairs under these two strategies, where have the following observations: (1) When increasing the number of motion query pairs, strategy a achieves much better mAP and miss rate than strategy 3. Because intention query points can ensure more stable training process since each intention query points is responsible to specific motion mode. In contrast, strategy 3 depends on unstable predictions and the positive component may randomly switch among all components, so a large number of motion query pairs are hard to be optimized with strategy 3. (2) The explicit meaning each intention query point also illustrates the reason that strategy a consistently achieves much better mAP than strategy B, since it can predict trajectories with more confident scores to benefit mAP metric. (3) From another side, when decreasing the number of motion query pairs, the miss rate of strategy a greatly increases, since limit number of intention query points can not well cover all potential motions of agents. Conversely, strategy 3 works well for a small number of motion query pairs since its queries not in charge of specific region and can globally adapt to any region.
the effects of the number

---

5 Conclusion
In this paper, we present MTR, a novel framework for multimodal motion prediction. The motion query pair is defined model motion prediction the joint optimization of global intention localiza- tion and local movement refinement. The global intention localization adopts a small set of learnable static intention queries efficiently capture agent's motion intentions, while the local movement re- finement conducts iterative motion refinement by continually probing fine-grained trajectory features. The experiments on both marginal and joint motion prediction challenges of large-scale WOMD dataset show that our approach achieves state-of of ohiee-of-theaarr performance.
Limitations. The proposed framework adopts an agent-centric strategy to predict multimodal future trajectories for one interested agent, leading redundant context encoding for multiple interested agents in the same scene. Hence, how to develop a framework that can simultancously predict multimodal motion for multiple agents is one important future work. Besides, the rule-based post-processing can result in suboptimal predictions for minADE and minFDE metrics, and how to design a better strategy produce required number of future trajectories (e.g.. 6 trajectories) from full multimodal predictions (e.g., 64 predictions) is also worth exploring for a more robust framework.
References
[1] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In CVPR, 2016.
[2] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2022.
[3] Yuriy Biktairov, Maxim Stebelev, Trina Rudenko, Oleh Shliazhko, and Boris Yangel. Prank: motion prediction based on ranking. In NeurlPS, 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[5] Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urtasun. Spagnn: Spatially-aware graph neural networks for relational behavior forecasting from sensor data. In ICRA, 2020.
[6] Sergio Casas, Cole Gulino, Simon Suo, Katic Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forecasting. ECCV, 2020.
[7] Sergio Casas, Wenjie Luo, and Raquel Urtasun. Intentnet: Learning to predict intention from raw sensor data CoRL, 2018.
[8] Sergio Casas, Abbas Sadat, and Raquel Urtasun. Mp3: A unified model map, perceive, predict and plan. In CVPR, 2021.
[9] Yuning Chai, Benjamin Sapp. Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. CoRL, 2019.
[10] Xuesong Chen, Shaoshuai Shi, Benjin Zhu, Ka Chun Cheung, Hang Xu, and Hongsheng Li. Mppnet: Multi-frame feature intertwining with proxy points for 3d temporal object detection. In ECCV, 2022.
[11] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In arXiv preprint arXiv: 04,000 2018
[12] Nemanja Djuric, Vladan Radosavljevic, Henggang Cui, Thi Nguyen, Fang-Chieh Chou, Tsung- Han Lin, Nitin Singh, and Jeff Schneider. Uncertainty- short-term motion prediction of traffic actors for autonomous driving. In WACV, 2020.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[14] Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In ICCV, 2021.
[15] Liangji Fang. Qinhong Jiang, Jianping Shi, and Bolei Zhou. Tpnet: Trajectory proposal network for motion prediction. In CVPR, 2020.
10

---

[16] Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov. Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In CVPR, 2020.
[17] Thomas Gilles, Stefano Sabatini, Dzmitry Tsiskkou. Bogdan Stanciulescu, and Fabien Moutarde. Gohome: Graph-oriented heatmap output for future motion estimation. In arXiv preprint arXiv: arXiv:2109.0188: 109.0127, 2021.
[18] Thomas Gilles, Stefano Sabatini, Damitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Home: Heatmap output for future motion estimation. In /TSC, 2021.
[19] Junru Gu. Chen Sun, and Hang Zhao. Densetat: End-to-end trajectory prediction from dense goal sets. In ICCV, 2021.
[20] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan; Socially acceptable trajectories with generative adversarial networks. In CVPR, 2018.
[21] Joey Hong, Benjamin Sapp, and James Philbin. Rules of the road: Predicting driving behavior with a convolutional model of semantic interactions. In CVPR, 2019.
[22] Xiaosong Jia, Penghao Wu, Li Chen, Hongyang Li, Yu Liu, and Junchi Yan. Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. In arXiv preprint arXiv:2205.0975. 2022.
[23] Stepan Konev. Mpa: Multipath++ based architecture for motion prediction. In arXiv preprint arXiv:2206. 10041, 2022.
[24] Stepan Konev, Kirill Brodt, and Artsiom Sanakoyeu. Motionenn: A strong baseline motion prediction in autonomous driving. In Workshop on Autonomous Driving, CVPR, 2021.
[25] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang. Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. CVPR, 2022.
[26] Jiachen Li. Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. In NeurIPS, 2020.
[27] Ming Liang, Bin Yang. Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In ECCV, 2020.
[28] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi. Hang Su, Jun Zhu, and Lei Zhang, DAB-DETR: Dynamic anchor boxes are better queries for DETR. In ICLR, 2022.
[29] Yieheng Liu, Jinghuai Zhang, Liangii Fang, Qinhong Jiang, and Bolei Zhou, Multimodal motion prediction with stacked transformers. In CVPR, 2021.
[30] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In ECCV, 2020.
[31] Francesco Marchetti, Federico Becattini, Lorenzo Scidenari, and Alberto Del Bimbo. Mantra: Memory augmented networks for multiple trajectory prediction. In CVPR, 2020.
[32] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In ICCV, 2021.
[33] Jean Mercat, Thomas Gilles, Nicole EI Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Pita Gil. Multi-head attention for multi-modal joint vehicle motion forecasting. In ICRA, 2020.
[34] Xiaoyu Mo, Zhiyu Huang, and Chen Ly. Multi-modal interactive agent trajectory prediction using heterogeneous edge-enhanced graph attention network. In Workshop on Autonomous Driving, CVPR, 2021.
[35] Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roclofs, Alex Bewley, Chenxi Liu, Ashish Venugopal. et al. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In ICLR, 2022.
[36] Scong Hyeon Park, Gyubok Lee, Jimin Seo, Manoj Bhat, Minseok Kang, Jonathan Francis, Ashwin Jadhav, Paul Pu Liang, and Louis-Philippe Morency. Diverse and admissible trajectory forecasting through multimodal context understanding. In ECCV, 2020.
[37] Tung Phan-Minn Minh, Elena Corina Grigore, Freddy A Boulton, Oscar Beijbom, and Eric M Wolff. Covernet: Multimodal behavior prediction using trajectory sets. In CVPR, 2020.
11

---

[38] Charles R Qi. Hao Su. Kaichun Mo. and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In CVPR, 2017.
[39] Nicholas Rhinehart, Kris M Kitani, and Paul Vernza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting, In ECCV, 2018.
[40] Nicholas Rhinchart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction conditioned on goals in visual multi-agent settings. In ICCV, 2019.
[41] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically- feasible trajectory forecasting with heterogeneous data. In ECCV, 2020.
[42] Shaoshuai Shi, Li Jiang. Dengxin Dai, and Bernt Schiele. Mtr-a: 1st place solution for 2022 waymo open dataset challenge motion prediction. arXiv preprint 100xv:22209.00332 2022.
[43] Qiao Sun, Xin Huang, Junru Gu, Brian C Williams, and Hang Zhao. M2i: From factored marginal trajectory prediction to interactive prediction. In CVPR, 2022.
[44] Charlic Tang and Russ R Salakhutdinov. Multiple futures prediction. In NeurlPS, 2019.
[45] Xiaocheng Tang, Soheil Sadeghi Eshkevari, Haoyu Chen, Weidan Wu, Wei Qian, and Xiaoming Wang. Golfer: Trajectory prediction with masked goal conditioning mnm network. In arxiv preprint arXiv:2007.00738, 2022.
[46] Ekaterina Tolstaya, Reza Mahjourian, Carlton Downey, Balakrishnan Vadarajan, Benjamin Sapp, and Dragomir Anguelov. Identifying driver interactions via conditional behavior prediction. In ICRA, 2021.
[47] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Comman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In ICRA, 2022.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, 2017.
[49] Jiashun Wang, Huazhe Xu, Medhini Narasimhan, and Xiaolong Wang. Multi-person 3d motion prediction with multi-range transformers. In NeurlPS, 2021.
[50] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks, In CVPR, 2018.
[51] Waymo. Waymo open dataset interaction prediction challenge 2021.
httpa://vaamo..om open/ 2021. Accessed: 2022-05-19.
[52] Waymo. Waymo open dataset motion prediction challenge 2021.
https:// http:///aaaooo.o//ppeee com/oper.
hhallengess202//mooot http:///toootoo-e-eiiiiioooooooo 2021. Accessed: 2022-05-19.
[53] Waymo. Waymo open dataset motion prediction challenge 2022.
https://waymo cww.:pon.....o/.//e Challenges//e 2022://////2mmmmmooooomo 2ttp://mtiit--peeiiiiiii.. prediction/ 2022. Accessed: 2022-09-21.
[54] David Wu and Yunnan Wu. Air2 for interaction prediction. In Workshop on Autonomous Driving, CVPR, 2021.
[55] Yi Xu, Dongchun Ren, Mingxia Li, Yuehai Chen, Mingyu Fan, and Huaxia Xia. Tra2tra: Trajectory-to- trajectory prediction with global social spatial-temporal attentive neural network. In RA-L, 2021.
[56] Zetong Yang. Li Jiang. Yanan Sun, Bernt Schiele, and Jiaya Jia. A unified query-based paradigm for point cloud understanding. In CVPR, 2022.
[57] Maosheng Ye, Tongyi Cao, and Qifeng Chen. Tpen: Temporal point cloud networks for motion forecasting. In CVPR, 2021.
[58] Fangao Zeng, Bin Dong, Tiancai Wang, Xiangyu Zhang, and Yiehen Wei. Motr: End-to-end multiple-object tracking with transformer. In arXiv preprint arXin:2105.0322 2021.
[59] Hao Zhang, Feng Li. Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In arXiv preprint arXiv:2203.0300 036005 2022.
[60] Yifan Zhang, Jinghuai Zhang, Jindi Zhang, Jianping Wang, Kejie Lu, and Jeff Hong. A novel learning framework for sampling- aamling-basedd based motion planning in autonomous driving. In AAAI, 2020.
12

---

[61] Hang Zhao, Jiyang Gao. Tian Lan, Chen Sun. Benjamin Sapp. Balakrishnan Varadarajan, Yue Shen,
Yi Shen, Yuning Chai, Cordelia Schmid, al. Tat: Target-driven trajectory prediction. In CoRL, 2020.
[62] Chen Ly Zhiyu Huang, Xiaoyu Mo. Recoat: A deep learning framework with attention mechanism for multi modal motion prediction. In Workshop on Autonomous Driving. CVPR, 2021.
[63] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
[64] Yanliang Zhu, Deheng Qian, Dongehun Ren, and Huaxia Xia. Starnet: Pedestrian trajectory prediction using deep neural network in star topology. In IROS, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work only for academic research purpose.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material tation details are posted on Section 4.1 and the supplemental matcrials. The data is public dataset and our code will be available.
URL)? [Yes] Our implemen-
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1 for training details, and the data splits of ablation experiments are mentioned in Section 4.3
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We use fixed random seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (c.g., code, data, models) or curating/relcusing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See our reference.
(b) Did you mention the license of the assets? [No] They are publicly available for academic use.
(c) Did you include any new assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] They are publicly available for academic use.
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] We only use the public dataset.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "nI_-TLdMHRL", "reviewer": "Reviewer_4kMs", "summary": "The work addresses the motion forecasting problem for autonomous driving. The authors introduce a transformer-based framework (MTR) that works in the following ways, as highlighted in the contribution section of the paper:\n\n- Separates the modeling of the global intention from local movement refinement in the trajectories in the transformer framework. The predictor is inspired by DETR. \n- Interaction modeling between agents via an auxiliary dense prediction task, essentially letting the model to predict the future directly (I didn't fully understand that part... see below)\n- SOTA results on the Waymo dataset. Ranked #1 among results without using ensembles. \n\nSince I don't understand the 2nd contribution, I cannot recommend accept at this point. Looking forward to understanding it post-rebuttal!\n\nPOST REBUTTAL UPDATE: I understand the contribution better now. In line with other reviews' feedback, I'll change the rating from 4 -> 7, and recommend an accept.\n", "questions": "line 73: Why not also using ensemble to boost the accuracy a bit to achieve top results? \n\nFigure 1: The figure is far from self explaining. What do the small road illustrations above the dense future prediction and dynamic map construction say? What are K, T, N and D (I know its in the text, but it doesn't hurt to say it here)? Where do the input \"predicted trajectories\", \"map query\" and \"query content feature\" come from? The figure is way too complicated, and I don't feel rewarded for looking closely....\n\nEq 2, ln 164: There are equations like   X = f(X), e.g. A = MLP([A, A_t])   what does this even mean? please use subscripts to make it clear that the variable in LHS and RHS are not the same. \n\nln 237: where does the auxiliary loss L_a come from? just write equations and not words. And if its just an L1 loss against the GT. I don't understand the intuition why it helps with interaction. It seems like its just like a multi-head network where you supply the GT at several layers of the network. ", "limitations": "Yes, but it didn't hit the mark. \n\nThe limitation section is about high-level limitation, like \"the method is not great for long tail behaviors\", not a wish list of future works. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nStrong results: Significant improvement over the SOTA. I checked the leaderboard on Waymo Open Dataset, and verified the claims. \n\nSomewhat novelty: Using DETR for motion forecasting is not that new anymore. I reviewed several papers for CVPR'22 and ECCV'22 that contained similar ideas. But those should be considered concurrent work. Additionally, the hierarchical separation of intent vs. fine control adds an additional hint of novelty, though the hierarchical separation by itself is not novel either [9, 51]. \n\nGood ablation: The paper contains ablation for all the contributions and novelties in their methods. This is great!\n\nWeaknesses:\n\n- Unclear writing: Abused notations, missing descriptions of variables, figures that are not self-contained. See below under \"questions\".\n\n- Insufficient experiments (minor): The only results are on the Waymo dataset. There are other popular datasets such as Argoverse and nuScenes. Having strong results on a secondary dataset would make the representation a lot stronger. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "miAW_MP8fEc", "reviewer": "Reviewer_b34B", "summary": "This paper proposes a transformer-based method for vehicle trajectory forecasting. It proposes to combine the tasks of global localization and local refinement for more accurate trajectory forecasting. On the structure side, the authors design a mechanism of motion query pair to model motion prediction as the joint optimization of the two tasks. Moreover, the interaction among agents is considered in the proposed method and collaborates to make the dense future prediction. The proposed method is demonstrated to be efficient on the large-scale Waymo Open Dataset. And an end-to-end variant of the proposed method is also provided for a broader study.", "questions": "I can understand the technical design of the proposed method while I have some concerns as mentioned in the \"Weaknesses\" part above. It would be helpful if the authors can address them with more clarification, with which I can more confidently assess the significance of this paper and adjust my rating.", "limitations": "Some technical design of the proposed method is explained at the end of the draft. I don't recognize more potential negative societal impact or limitations of this paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is well elaborated and the implementation details are necessarily provided to help understand the model design.\n2. The performance of the proposed MTR on the Waymo Open Motion Dataset is good, advancing the SoTA under the setting further.\n3. The paper is mostly well written that I can understand the motivation, high-level intuition, and model design quickly.\n\nWeaknesses:\n1. Some details are not clear, for example, the implementation of MTR-e2e is hard to follow for me given the limited illustration at L273. \n2. Though it may not be necessary, it would be helpful to ablate the choice of query pair number for end-to-end version as well as there is a claim that \"since 6 intention points are too sparse to well cover all potential future motions.\" may need some experiment backup.\n3. Given the proposed method stresses on the design of query pair, it would be helpful for us to better understand the efficiency of this by making an ablation study of using only one type of query or both.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MIuRPHkh6rJ", "reviewer": "Reviewer_Xhm6", "summary": "In this paper, a Motion TRansformer (MTR) framework is proposed for the motion prediction task, including marginal and joint motion prediction. Specifically, motion query pairs are designed for global intention localization and local movement refinement, which takes advantage of both goal-based methods and the regression methods. Experiments on Waymo Open Dataset indicate the effectiveness of the proposed method.", "questions": "Concerns: \n1. Overall, it is a paper for motion prediction challenge with promising results and clear presentations. I assume it is the initial version of the final 1st place work [1], sharing very similar methods and almost the same framework, thus I would encourage authors to update the paper with [1]. \n2. The most concern is that all the strategies are well-studied in this community, i.e., the transformer idea, scene context and interaction modeling, etc. It would be better if more insights could be included for model design, and highlight the differences/contributions.\n3. In Line 134-135, how to pad zeros for trajectories that have less than $t$ frames? In the front/back? Some details are missing here.\n4. Some missing references when referring to future interactions, some works [2-4] have been proposed to argue the future interaction point and to consider such future interactions.\n\nMinors:  \n1. Line 56, I believe it is 'empirical' instead of 'experimentally'.\n2. In References, the names of conferences or journals, i.e. abbreviation or full name, are not consistent.  \n\nI am happy to improve my rating if these concerns could be addressed.\n\n\n[1] https://storage.googleapis.com/waymo-uploads/files/research/MotionPred/MotionPrediction_MTRA.pdf  \n[2] EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning. NeurIPS 2020  \n[3] Tra2Tra: Trajectory-to-Trajectory Prediction With a Global Social Spatial-Temporal Attentive Neural Network. ICRA 2021  \n[4] StarNet: Pedestrian Trajectory Prediction using Deep Neural Network in Star Topology. IROS 2019  \n\n", "limitations": "There is a limitation section in the main body of the paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n- Strong motivation and well-organized.\n- Promising results.\n\n\nWeaknesses: (See Questions for details) \n- Some insights/details are not clear.\n- Lack of some experiments.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EsAEXZmp7gn", "reviewer": "Reviewer_3fr6", "summary": "This paper proposes an decoder method for motion prediction task, which refines different modes with the static prior and dynamic attention. Its performance on the Waymo Open Motion Dataset is impressive, which demonstrates the effectiveness of iteratively refine the prediction (similar to DETR/DAB-DETR).", "questions": "#### Question\n* The PE module is first introduced in the NLP field so that tokens' position information could be considered (Transformer is a set operator which ignores the postion information).  However, in the field of motion prediction, the feature itself includes the postion information (coordinate). I am wondering why the proposed method uses the PE + MLP instead of just MLP. Is there any experiments about its effectiveness?\n* In equation (6), it uses C^{j-1}. How to initialize it in the first layer is not described.\n* I noticed that Waymo Motion Challenge 2022 recently finished. Authors may discuss about those top-ranked methods including golfer, MPA, HDGT, etc, which could mkae the paper more up-to-date.", "limitations": "The limitation part is ok.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The idea of iteratively refining the prediction by Transformer is novel in the motion prediction area.\n* The performance on Waymo Motion Dataset is great.\n* The ablation experiments is well-organized and convincing\n\nWeakness\n* The proposed method is only evaluated on one dataset. It would be more convincing if the experiments could be done on other large scale datasets. However, considering the Waymo Open Motion is fairly large, I think it is okay if there is no enough time/resource to try other datasets.\n* Some parts of the proposed methods is not clearly described in the manuscript, which is understandable considering the space limit. \n* The proposed method has similarities with the DAB-DETR in the objection detection area. I think it would better position this work if the authors could discuss about the proposed method's relation with recent DETR related works.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept