SEPT: Towards Efficient Scene Representation Learning for Motion Prediction
Authors: Zhiqian Lan
Yuxuan Jiang
Yao Mu
Chen Chen
Shengbo Eben Li
Venue: ICLR 2024 poster (2024)
Keywords: motion prediction
autonomous driving
self-supervised learning
Primary area: applications to robotics, autonomy, planning

Abstract: Motion prediction is crucial for autonomous vehicles to operate safely in complex traffic environments. Extracting effective spatiotemporal relationships among traffic elements is key to accurate forecasting. Inspired by the successful practice of pretrained large language models, this paper presents SEPT, a modeling framework that leverages self-supervised learning to develop powerful spatiotemporal understanding for complex traffic scenes. Specifically, our approach involves three masking-reconstruction modeling tasks on scene inputs including agents' trajectories and road network, pretraining the scene encoder to capture kinematics within trajectory, spatial structure of road network, and interactions among roads and agents. The pretrained encoder is then finetuned on the downstream forecasting task. Extensive experiments demonstrate that SEPT, without elaborate architectural design or manual feature engineering, achieves state-of-the-art performance on the Argoverse 1 and Argoverse 2 motion forecasting benchmarks, outperforming previous methods on all main metrics by a large margin.

Published as a conference paper at ICLR 2024
SEPT: TOWARDS EFFICIENT SCENE REPRESENTATION LEARNING FOR MOTION PREDICTION
Zhiqian Lan", Yuxuan Jiang", Yao Mu, Chen Chen, Shengbo Eben Li School of Vehicle and Mobility, Tsinghua University
(1 [1azq21,, jyx21 @mmai 1s. t.s t.sissnnnnnn nighuaa. edu. cn 11 iishbotts i niguua. edu.cm cn
ABSTRACT
Motion prediction is crucial for autonomous vehicles to operate safely in complex traffic environments. Extracting effective spatiotemporal relationships among traf- fie elements is key accurate forecasting. Inspired by the successful practice of pretrained large language models, this paper presents SEPT, a modeling frame- work that leverages self-supervised learning to develop powerful spatiotemporal understanding for complex traffic scenes. Specifically, our approach involves three masking-reee reconstruction modeling tasks on scene inputs including agents' trajecto- ries and road network, pretraining the scene encoder to capture kinematics within trajectory, spatial structure of road network, and interactions among roads and agents. The pretrained encoder is then finetuned on the downstream forecasting task. Extensive experiments demonstrate that SEPT, without claborate architec- tural design manual feature engineering, achieves state-of-the-an art performance on the Argoverse and Argoverse motion forecasting benchmarks, outperform- ing previous methods on all main metrics by large margin.
INTRODUCTION
Accurately predicting the future trajectories of surrounding road users is crucial to a safe and effi- cient autonomous driving system. In addition kinematic constraints, the future motions of traffic agents may be shaped by many factors in the traffic scene, including shape and topology of roads, and surrounding agents' behaviors. Modeling and understanding these intricate relationships within the scene has long been a core challenge for motion prediction.
Researches in the early stage predominantly use rasterized semantic images to represent the whole scene from a top-down view, and fuse the information with convolutional neural networks (Lee et al., 2017 Chai et al. 2020; Phan--Mnh et al., 2020). Due to the information loss during rasterization, recent researches have shifted to a vectorized paradigm in which the agents and roads are represented as set of vectors. This representation has served the foundation for numerous advanced methods that employ graph neural networks and transformers for scene encoding (Liang et al. 2020 Zhou 2002). However, these prevailing SOTA methods typically embrace sophisticated architectural designs, which often rely on anchor-based modeling (Wang al. 2022) or proposal-refinement scheme (Shi et al. [202 Wang et al.) 2023b) to enhance the prediction performance. These empirical techniques consequently lead to intricate information processing pipeline.
While previous attempts on scene encoding have focused on innovations in feature engincering and architectural design, we believe that the encoders built universal architectures can develop strong comprehension on traffic scenes through a properly designed training scheme. A promising direction is explore self-supervised learning (SSL). Large language models like GPT-3 (Brown al. 2020) have leveraged SSL on large text corpora to learn broadly applicable linguistic knowledge, achieving significant advancement on diverse set of NLP tasks. This inspires that motion prediction models can implicitly be endowed with useful knowledge about traffic scenes such as environment dynamics and social interactions by effective self-supervised objectives.
Equal contribution.
Cooresponndence Shengho Eben Li, email: lishho@ saabootssinguu..cdu.cm

---

Published as a conference paper at ICLR 2024
(a) MTM
(b) MRM
(c) TP
Figure 1: Illustration for proposed scene understanding training tasks. For each picture, the
masked or truncated scene inputs are painted in black while the corresponding reconstructed results painted in blue.
Driven by this inspiration, we propose Scene Encoding Predictive Transformer (SEPT), a neat and powerful motion prediction framework that leverages SSL to progressively develop the spatiotempo- ral understanding for traffic scenes. We construct the self-supervised pretraining scheme for SEPT's scene encoder capture three key aspects of scene context: temporal dependency within historical trajectory, spatial structure of road network, and interactions among roads and agents, yielding three corresponding pretraining tasks: Masked Trajectory Modeling (MTM), Masked Road Modeling (MRM), and Tail Prediction (TP). MTM, as shown in Figure Ia randomly masks and reconstructs some waypoints in the agents' trajectories, matic constraints. Similarly. MRM (Figure Ib) randomly masks some portion of the road vectors and then predicts the masked part, allowing the encoder to effectively capture the topology and con- nectivity of road network. While the previous two tasks handle single input modality, the third task TP (Figure Tep focuses on interactions between modalities by conducting a short-term motion pre- diction task. In this task, divide the agents' trajectories into two sections, named as head and tail, and the objective is to predict the tail section based on the preceding head section and the road con- text. The pretrained encoder is then finetuned to the downstream motion prediction task. Compared to several recent works exploring SSL in motion prediction (Chen et al. 2023 Cheng et al. 2023), method adopts unique pretraining task design and achieves better prediction performance.
encode the temporal dependency arising from kine-
The effectiveness of the proposed SEPT are demonstrated by extensive experiments from two as- pects. First, the model with pretraining achieves significant improvement over the model learned from scratch on all motion forecasting metrics consistently, showing that the model gains beneficial knowledge through pretraining objectives. Second, the three self-supervised pretraining tasks effec- tively collaborate with each other and yield positive effects on the final performance in an additive manner. Meanwhile, our experimental results on Argoverse and Argoverse 2 datasets establish SEPT as a top- nop-performing model, ranking 1s across all primary metrics on both large-scale motion forecasting benchmarks. Moreover, compared to the strongest baseline (Zhou et al., (023) to our knowledge, our model achieves twice faster inference speed with only 40% network parameters.
2 RELATED WORKS
Scene encoding with transformers. Inspired by the remarkable success of transformer architec- ture, attention mechanism has been extensively used in motion prediction to model the long-range and complicated interactions within the traffic scene. Early attempts have employed attention in
specific sub-modules to encode spatial relationships such as social interactions (Mercat et al. 2020 Salzmann et al., 2020), or agent-map relationships (Huang et al.) 2022). few works model traffic elements as graph and adopt transformer structure to exploit relations graph- based inputs (Wang jet al.) 2022; Jia et al.) 2023). Recent SOTA approaches, on the other hand, tend to process all input modalities, including trajectories and road network, jointly with hierarchically stacked transformer blocks (Ngiam et al., 2022] Nayakanti et al.) 2023 Zhou et al. 2023). These approaches represent the whole traffic scene 3-D tensors (space- time-ttie--eeeee (ume-ffattre) and perform factorized attention alter- nately along the temporal and spatial axes for several rounds to exploit spatiotemporal dependencies. Though sharing the same architectural design concept, the proposed SEPT adopts compact
2

---

Published as a conference paper at ICLR 2024
information processing pipeline, where temporal and spatial information encoded sequentially. This leads to neat model architecture with fewer functional blocks.
Self-supervised learning in motion prediction. Some previous studies have explored self- supervised learning in motion prediction. To the best of our knowledge, the works most closely related our research are Traj-MAE (Chen et al., 2023) and Forecast-MAE (Cheng et al. 2023). Following the idea of masked autoencoder (He et al., 2022), Traj-MAE designs two independent mask-reconstruutton tasks on trajectories and road map input, to train its trajectory and map encoder separately. Such separation leads to a notable limitation that the spatial relationship between agents and roads are not stressed during pretraining. In contrast, SEPT introduces the Tail prediction (TP) pretraining task, which considers both trajectories and roads inputs to capture the spatiotemporal relationships in the traffic scene. Our ablation study has shown the significant impact of TP on the final performance. The other work, Forecast-MA MAA,, adopts a distinct masking strategy for agent tra- jectory. This approach incorporates the ground truth future trajectories of agents into its pretraining stage and predict the future given its history vice versa. Its main difference with our SEPT is the granularity of the tokenization for the scene inputs: it takes the whole historical trajectory or a poly- line of lane segments a single token. SEPT, on the other hand, treats waypoint in trajectory or short road vector as a token, which can better capture the motion patterns, as well as the dependency between motions and road structure. In addition, not all self-supery tasks in Forecast- contribute to the downstream prediction task positively. As is reported in its ablation study, several combinations of tasks may degrade the final performance compared to models learned from scratch. Instead, SEPT's ablation studies show that all task combinations could improve performance in a consistent and additive manner.
APPROACH
3.1 INPUT REPRESENTATION
SEPT represents the traffic scene input with the following modalities:
Agent trajectories are represented as tensor [A, Dzl, which captures the recent trajectories of A nearest traffic agents relative to the target agent, including itself. Each agent's trajectory is represented as time series of T state vectors, containing coordinates, timestamp, agent type, and any other attributes provided in the dataset. All position coordinates are transformed into the local coordinate system of the target agent at its T-th frame. In addition, not all agents have full T frames of history as the target agent. Shorter agent history is padded to T frames and properly masked for further attention computation.
Road network is represented as a set of R road vectors [R, D.], based on the vectorized map representation from VectorNet (Gao et al. 2020). Each road vector is directed lane centerline segment with features including start and end positions, length, turn direction, and other semantics from the dataset. To obtain fne-grained inputs for the road network, lane centerlines are segmented into vectors spanning no longer than meters in length. Representing the entire road network in this way results in an input vector set of massive scale, bringing excessive computational and memory burden in self-attention modules. SEPT introduces a road pruning module based on the idea of static path planning (Guan al. 2022 Diang et 2023). to identify potential routes feasible for traversal by the target agent within the predictive horizon and construct a more compact subset [S, Dr].
3.2 MODEL ARCHITECTURE
Figure 2 illustrates the simple but expressive model architecture of the proposed SEPT. The model comprises an encoder for scene encoding and decoder that predicts weighted set of trajectories based on scene memory embeddings.
Projection layer projects different input modalities into shared high dimensional vector space R. in order to perform follow-up attention operations. A projection layer in SEPT is single linear layer with ReLU activation, i.e., Project(z)) = max(Wx + b, 0).
TempoNet consists of Ky stacked Transformer encoder blocks, and is used for agent history en- coding. TempoNet takes input agent history embeddings [A, T, D) and conducts self-attention
3

---

Published as a conference paper at ICLR 2024
Input
Encoder
Decoder
Output
TMD.... ...
Projection
reduction
learnable queries [N, DJ
TempoNet
T
agent histories
trajectories
[A, T, Dx]
agent embs [A.
Cross Attender
MLPS
[N, H,
SpaNet
Pruning
Projection
0.7
0.2
0.1
Concat
scene memory [A+S, P)
road not [R. Dv]
key road vectors [S, D-]
road embs [S, D]
probabilities [N.
Figure 2: The overall architecture of SEPT
operation along temporal dimension T. After that, the embeddings across time steps are aggregated through max pooling produce agent embeddings [A, D). Additionally, TempoNet applies the sim- plified relative position embedding proposed in T5 (Raffel et al. 2020). to encode relative order of time steps.
SpaNet concatenates Ks Transformer encoder blocks as TempoNet does, and its primary objective to capture spatial relationships within the traffic scene. Its scene embedding input is formed by concatenating agent and road embeddings. Notably, compared to many previous SOTA methods, SEPT avoids the use of dedicated modules that separately encode agent interaction, road network, and agent-road relations. Instead, SpaNet leverage self attention across spatial dimension A + S to fulfill these encoding objectives a unified way.
Cross Attender comprises Ko stacked cross attention layers. This module cross attends set of N learnable queries [N, D] with the scene memory embeddings [A + S, D] from the encoder. This generates N embeddings [N, D] corresponding to the N output modalities. Each embedding vector then mapped from the hidden space to the physical world through two MLPs, producing the predicted 2D trajectory and its associated confidence score.
3.3 SCENE UNDERSTANDING TRAINING
In the scene understanding training stage, we train the encoder of SEPT understand the tem- poral and spatial relationships in the traffic scene, and produce high quality latent representations for the downstream trajectory prediction task. This is achieved with three self-supervised mask- reconstruction learning tasks on TempoNet, SpaNet, and their alignment, as is shown in Figure 3 In this stage, the model optimized with the summation of the three tasks' objectives.
3.3. TASK 1: MASKED TRAJECTORY MODELING (MTM)
MTM targets TempoNet for understanding the temporal dependencies in agents' trajectories. As is shown in Figure 3al randomly substitute certain ratio of frames with learnable mask token for all eligible input trajectories. By eligible here, we refer to trajectories longer than a certain threshold, because short trajectories model to learn meaningful representations. For MTM and two more scene understanding tasks below, the hidden vectors are fed into a shallow MLP decoder to produce reconstruction vectors, and the objective function is defined as the masked Le loss between reconstruction and original vectors.
often of low quality, and may be informative enough for the
3.3.2 TASK 2: MASKED ROAD MODELING (MRM)
As is shown Figure 3b MRM shares a similar idea with MTM but it targets SpaNet to learn the spatial structure of road vector input. Different from the trajectories in MTM, road segments form a graph rather than a sequence, where simple positional encoding becomes unsuitable. Without bias about relative position information, learning with the token-level mask would be difficult, since
4

---

Published as a conference paper at ICLR 2024
TempoNet
MLP
SpaNet
MLP
history embs
MTM: reconstruct
road embs w/ attribute mask
MRM: reconstruct masked vectors
w/ token mask
masked trajectories
(a) Masked Trajectory Modeling
(b) Masked Road Modeling
T
reduction
take A tokens
MLP
A
TempoNet
SpaNet
TP: prediet tail sections
history embs w/ truncation
road embs
Concat
(c) Tail Prediction
Figure 3: SEPT tasks to learn the temporal and spatial relationship in the traffic scene
time consuming linear assignment problem needs to be solved to compute the reconstruction loss. We instead employ an attribute-level mask technique by setting all attributes, except for the coor- dinates of the starting point, zero for selected road feature vectors. The starting point serves as a hint for SpaNet to reconstruct the original vectors. In doing so, SpaNet has to aware of the connectivity and continuity among unordered road vectors in order recover the masked attributes like coordinates of endpoint.
3.3.3 TASK TAIL PREDICTION (TP)
In TP, the agents' historical trajectories are divided into two sections along the trajectory axis: the head and the tail. The model trained to predict the tail section given the head section by employing TempoNet and SpaNet jointly, which can be seen as a simplified version of motion prediction, as is shown Fig Specifically, the agents' historical trajectories are truncated to first T7, tokens before fed into TempoNet. Trajectories shorter than threshold kTT are excluded from prediction. Then, the embedding vectors of the truncated trajectories and the road embeddings are concatenated and fed into SpaNet. Finally. the hidden vectors of corresponding agents fed into shallow MI.P decoder to predict the tail sections. This task designed to align the trajectory representation learned in MTM with the road context representation learned in MRM and model their relationships, since accurate prediction requires effective use of both spatial and temporal information.
3.4 MOTION PREDICTION TRAINING
In the motion prediction training stage, we concatenate the pretrained encoder with the decoder, and train the whole model on the labeled dataset in an end-to-end fashion. For the motion prediction task, the loss function L consists of two terms: the trajectory regression loss Creg and the classification loss Ldts, which are defined in (T]
L= == + Celss Creg L, Tyt);
Lats - log Pj.
where Ti, P; for .....,N are predicted trajectories with their normalized probabilities, and refers to the index of the trajectory closest to ground truth in terms of average displacement error (ADE):
j = arg min ADE Tg)).
5

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Dataset. The effectiveness of our approach is verified on Argoverse and Argoverse 2, two widely- used large-scale motion forecasting datasets collected from real world. The Argoverse dataset consists of 324 557 driving scenarios, with each scenario contains 2-second historical context and 3-second future to predict. In contrast, the Argoverse dataset comprises 250 000 scenarios, char- acterized by an extended 5-second historical context and longer prediction horizon of 6 seconds. The trajectories both datasets are sampled 10 Hz.
Metries. Following the evaluation protocol used by Argoverse competition, we calculate the fol- lowing metrics:
minFDEg: the l2 distance between the endpoint of the best of k forecasted trajectories and the ground truth.
minADEg: the average l2 distance between the best of k forecasted trajectories and the ground truth.
miss rate (MRg): the ratio of scenarios where minFDE, exceeds a threshold of m.
D-minFDE,t the minFDEg added by (1.0 - p)² where the probability of the best forecasted trajectory.
The best of k trajectories refers the trajectory with its endpoint closest to ground truth endpoint. These metrics are calculated for k = 6 and k = 1, except for b-minFDE,, which is only calculated for k = 6.
Training. In the scene understanding training stage, we concatenate train, validation and test dataset as the pretrain dataset with labels dropped. The model trained for 150 epochs with constant learning rate of 2 x 10 In the downstream motion prediction training stage, we train and validate following the split of the Argoverse dataset. The model trained for 50 epochs with the learning rate decayed linearly from 2 10-4 to O. Both stages are trained with a batch size of 96 on single NVIDIA GeForce RTX 3090 Ti GPU.
4.2
EXPERIMENTAL RESULTS
Comparison with baselines. We compare SEPT performance with SOTA published methods on the Argoverse 1 and Argoverse leaderboards, including DenseTNT Gu al. (2021). HOME Gilles
et al. 2021), MultiPath+++ Varadarajan et al.] (2022). GANet Wang et al. (2023a), MacFormer Feng
al. (202)) DCMS Ye et al. (2023), Gnet Gae al. (003). Wayformer Nayakanti al. (2023)
ProphNet Wang et al. (2023b) and OCNet[Zhon et al.
2223 as is shown in Table [
Method
b-minFDEs
minADEs
minFDEs
MR.
minADE;
minFDE,
MR1
DenseTNT
1.976
0.882
1.281
0.126
1.679
3.632
0.584
HOME
1.860
0.890
1.292
0.085
1.699
3.681
0.572
MultiPath++
1.793
0.790
1.214
0.132
1.624
3.614
0.565
GANet
1.790
0.806
1.161
0.118
1.592
3.455
0.550
MacFormer
1.767
0.812
1.214
0.127
1.656
3.608
0.560
DCMS
1.756
0.766
1.135
0.109
1.477
3.251
0.532
Gnet
1.751
0.789
1.160
0.117
1.569
3,407
0.545
Wayformer
1.741
0.768
1.162
0.119
1.636
3.656
0.572
ProphNet
1.694
0.762
1.134
0.110
1,491
3,263
0.526
QCNet
1.693
0.734
1.067
0.106
1.523
3.342
0.526
SEPT (Ours)
1.682
0.728
1.057
0.103
1.441
3.178
0.515
Table Argoverse leaderboard results sorted by b-minFDEs. bold, and the second best is underlined.
The best entry a metric is marked

---

Published as a conference paper at ICLR 2024
Method
b-minFDEg
minADEg
minFDEs
MRa
minADE;
minPDE
MR
MacFormer
1.905
0.701
1.377
0.186
1,838
4.686
0.612
Gnet
1.896
0.694
1.337
0.180
1.724
4.398
0.588
ProphNet
1.882
0.657
1.317
0.179
1.764
4.768
0.610
QCNet
1.779
0.619
1.191
0.144
1.563
3.962
0.548
SEPT (Ours)
1.736
0.605
1.151
0.137
1,485
3.700
0.545
Table 2: Argoverse 2 leaderboard results.
On the Argoverse I dataset, our result outperforms all previous entries and ranks 1s on all metrics except for MRg. For MR2, HOME employs "sparse sampling" technique to specialize miss rate performance while sacrifices other metrics. In this case, SEPT, being a general purpose method, ranks 2d Furthermore, for the Argoverse 2 dataset, SEPT consistently maintains its leading position on the leaderboard, ranking 1 on all metrics, Moreover, compared to QCNet, the strongest baseline in both benchmarks, SEPT has only 40% of its network parameters (around 9.6 million) while exhibits inference speed which is twice faster.
Comparison with related works. As discussed in related works, Traj-MAE and Forecast-MA. share a similar idea of mask-reconstruution for trajectory prediction, but struggle to reach SOTA performance to justify pretraining. Traj-MAE reports minADEg, minFDE, and MRg on Argoverse test set, and Forecast-MAE reports these metrics on Argoverse 2 test set. As is shown in Ta- bleß] SEPT outperforms these two methods by a large margin, further demonstrating our method's effectiveness.
Method
minADEs
minFDEs
MRg
Method
minADEg
minFDEg
MRg
Traj-MAE
0.81
1.25
0.137
Forecast-MAE
0.690
1.338
0.173
SEPT
0.728
1.057
0.103
SEPT
0.605
1.151
0.137
(a) Comparison with Mraj-MAE on Argoverse
(b) Comparison with Forecast- MAA on Argoverse
Table Performance comparison with related works
Trajectory prediction visualization. We compare the prediction results between the model with pretraining and the model trained from scratch in Figure 4 The visualization demonstrates that the model with pretraining can better capture the multimodality of driving purposes (shown in case 1) and generate the trajectories with improved conformity to road shapes (shown in case 2 & 3).
4.3 ABLATION STUDIES
The ablation studies focus on the scene understanding stage, specifically the effectiveness of various combinations of scene understanding tasks and the impact of their hyperparameters. We conduct ablation experiments following the same routine as the main experiment, and report the evaluation performance on the Argoverse validation dataset. To minimize the impact of randomness, conduct multiple experiments with different seeds for each configuration.
Effectiveness of scene understanding tasks. MTM, MRM and TP are three tasks introduced to leamn good prior for the downstream trajectory prediction task. To investigate the effectiveness, we conduct experiments for all combinations, and the results are shown Figure and Table [
In Figure 5 we use solid lines for combinations involving task TP, and dashed lines for combinations without this task. It is clear that combinations with TP consistently outperform the opposites by learning faster at the beginning and performing better after convergence. This indicates that TP effective to align TempoNet and SpaNet, therefore improve the overall representation quality of scene encoding. The other two tasks, MTM and MRM, can also improve performance in consistent and additive manner. Furthermore, it should be noted that the configuration with all tasks active not only achieves the best performance, but also has the lowest variance over 5 runs among combinations, as is shown the shaded area in Figure [ This suggests that our proposed training approach help the model converge good level consistently. Table 4 includes the

---

Published as a conference paper at ICLR 2024
w/ pretraining
w/ pretraining
(a) case 1
(c) case
Figure 4: Visualization results on three selected scenarios from Argoverse 1 validation set. The target agent's history and ground truth future are shown in red and green, respectively, while model's predictions are shown in blue. In each case, the upper image corresponds to the model that learns from scratch while the lower image corresponds the model with scene understanding training.
Serncch
Task configuration
Final evaluation metrics minADEg
MRM
MTM MRM TP
b-minFDE,
minFDE,
MR.
MTM--MRM
2.0
1.657
0.687
1.016
0.100
MTM-IP
1.620
0.670
0.980
0.095
1.9
MRM-TF
MTM-MRM-TP
1.628
0.674
0.988
0.096
1.603
0.663
0.964
0.091
1.605
0.661
0,964
0.091
1,586
0.653
0.947
0.089
1.588
0.653
0,950
0.090
10
20
30
40
50
1.563
0.643
0.927
0.086
Epoch
Figure 5 & Table 4: The training curves report b-minFDEg performance for all task combinations. The solid or dashed lines correspond to the mean and the shaded regions correspond to 1-a confi- dence interval over 5 runs. The table reports the final performance of all k 6 metrics. The best entry for metric is marked bold.
final performance for all & 6 metrics. It can be seen that other metrics exhibit similar patterns of improvement to b-minFDEg discussed above. Training curves for those metrics are included in the Appendix.
Effects of mask hyperparameters. In this ablation experiment, we show the impact of mask hyper- parameters on final performance. For MTM and MRM, the main mask hyperparameter is the mask ratio implemented as sampling a mask from Bernoull(p). For TP, the mask hyperparameter is the length of visible history T₁.- For each of these three hyperparameters, we pretrain with the single corresponding task, then finetune with identical setting get the final performance. As is shown in Figure outperform the model that trains from scratch. This suggests the effectiveness of our task design and
6 mask hyperparameters do not have a significant impact in suitable ranges and always
8

---

Published as a conference paper at ICLR 2024
its robustness to choices of hyperparameters, In main experiment, simply PMIM 0.5, PMRM = 0.5 and Tы = 8/20 (for Argoverse 1/2) without tuning.
1.64
1.64
1.64
1.63-
1.63
1.63-
bminnfD.
baiiFDDa
beninFDa
1.62
1.62
1.62-
1.61
0.2
0.4
3.6
0.8
0.3
0.4
00.5
0.5
0.7
0.8
2
5
10
Tomporal Mook Ralio
Spatial Mask Ratio
Visible History
(a) MTM
(b) MRM
(c)
Figure 6: Final b-minFDEs performance of varying mask settings for each task, averaged over 5 runs.
5 CONCLUSION
In this paper, we present Scene Encoding Predictive Transformer (SEPT), powerful modeling framework for accurate and efficient motion prediction. Leveraging three self-supervised mask- reconstruction learning tasks for scene understanding, we achieve SOTA performance on two large- scale motion forecasting datasets with a neat and unified model architecture. The ablation studies further demonstrate that introduced pretraining tasks can significantly improve prediction perfor- mance in consistent and additive manner. Meanwhile, we acknowledge the limitation that, due to its agent-centric scene representation, SEPT is single-agent prediction framework, and is not trivially extendable to multi-agent joint prediction. the future, we will investigate further and generalize it to more powerful multi-agent prediction framework.
ACKNOWLEDGEMMET
This work is supported by NSF China under 52221005. It is also partially supported by Tsinghua University- Didi Joint Research Center for Future Mobility.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.. 2020.
Yuning Chai, Benjamin Sapp. Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple prob- abilistic anchor trajectory hypotheses for behavior prediction. Leslie Pack Kaelbling. Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 86-99. PMIR, 30 Oct-01 Nov 2020. URL htttps::/pppceee
mlr www...ppesss.. 00/ httpcchaaa2.. html
Hao Chen, Jiaze Wang, Kun Shao, Furui Liu, Jianye Hao, Chenyong Guan, Guangyong Chen, and Pheng-Ann Heng. Traj-mae: Masked autoencoders for trajectory prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 8351-8366 October 2023.
Jie Cheng. Xiaodong Mei, and Ming Liu. Forecast-mae: Self-supervised pre-training for motion forecasting with masked autoencoders. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages (679-8689 October 2023.
Chen Feng, Hangning Zhou, Huadong Lin, Zhigang Zhang, Ziyao Xu, Chi Zhang, Boyu Zhou, and Shaojie Shen. Macformer: Map-agent coupled transformer for real-time and robust trajectory prediction, 2023.

---

Published as a conference paper at ICLR 2024
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11525-11532 11533, 2020.
Schmid. Vectornet:
Xing Gao, Xiaogang Jia, Yikang Li, and Hongkai Xiong. Dynamic scenario representation learn- ing for motion forecasting with heterogeneous graph convolutional recurrent networks. 1EEE Robotics Autom. Lett., 8(5):2946-2953, 2023. doi: N0.11099/RAA22233330000
URL https:
//doi. 10. 1109 LRA. 2023. 3262150
Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. HOME: heatmap output for future motion estimation. In 24th IEEE International Intelligent Transportation Systems Conference, ITSC 2021, Indianapolis, IN, USA, September 19-22, 2021, pages 500-507, IEEE, 2021. doi: 10.1109/TSC49978... 2022448878821111.. 1564444. URL https: //doi. 2021. 9564944
/tg//0.11099/10
Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 15283-15292. IEEE, 2021. doi: 10. 109/CC/49992 2021.01502. URL http::://doo..oroo/.. 111..//00/...1110.77/17133222220 2021. 01502
Yang Guan, Yangang Ren, Qi Sun, Shengbo Eben Li, Haitong Ma, Jingliang Duan, Yifan Dai, and Bo Cheng. Integrated decision and control: Toward interpretable and computationally efficient driving intelligence. IEEE transactions on cybernetics, 53(2):859-87) 873, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
Zhiyu Huang, Xiaoyu Mo, and Chen Lv. Multi-modal motion prediction with transformer-batt neural network for autonomous driving. In 2022 International Conference Robotics and Au- tomation (ICRA), pages 2605-261 IEEE, 2022.
Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heteroge- neous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE transactions on pattern analysis and machine intelligence, 2023.
Yuxuan Jiang, Guojian Zhan, Zhiqian Lan, Chang Liu, Bo Cheng, and Shengbo Eben Li. A rein- forcement learning benchmark for autonomous driving in general urban scenarios. JEEE Trans- actions Intelligent Transportation Systems, 2023.
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 336-345, 2017.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In Computer Vision-ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part II 16, pages 541-556. Springer, 2020.
Jean Mercat, Thomas Gilles, Nicole El Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Pita Gil. Multi-head attention for multi-modes joint vehicle motion forecasting. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9638 9644. IEEE, 2020.
Nigamaa Nayakanti, Rami A1-Rfou, Aurick Zhou, Kratarth Goel, Khaled S. Refaat, and Ben- jamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In IEEE International Conference Robotics and Automationn ICRA 2023, London, UK. May 29 June 2, 2023, pages 2980-2987, IEEE, 2023, doi: 10..1499/ A.2.4488899.2..... 1.222.101666099, 10160609. URL 09/ CRA 8891 2023. 2111131.222221110001000 90160699
https: /tops: http///oo.org//////
10

---

Published as a conference paper at ICLR 2024
Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jef- frey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David J Weiss, Ben- jamin Sapp. Zhifeng Chen, and Jonathon Shlens. Scene transformer: A unified architecture for predicting future trajectories multiple agents. In International Conference on Learning Repre- sentations, 2022. URL htttpss://ppeeeeeiiieeeee
net. forum?? d==m3EE550 HsG
Tung Phan-Minh, Elena Corina Grigore, Freddy A Boulton, Oscar Beijbom, and Eric M Wolff. Covernet: Multimodal behavior prediction using trajectory sets. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14074-14083, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2 21 (1(1):4855555) 2020.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasbble trajectory forecasting with heterogeneous data. In Computer -isi 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16. pages 683-700. Springer. 2020.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems, 35:6531-6543, 2022.
Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S. Refaat, Nigamaa Nayakanti, Andre Cormman, Kan Chen, Bertrand Douillard, Chi-Pang Lam, Dragomir Anguelov, and Benjamin Sapp. Multipath++:: Efficient information fusion and trajectory aggregation 2022 International Conference on Robotics and Automation,
for behavior prediction. ICRA 2022, Philadelphia. PA, USA, May 23-27, 2022, pages 7814-7821. IEEE, 2022. doi: 4LLL00//RRA46669.2200..08888 2022. 9812107
URL https: /lttps://oo..o org /ttt:s///6o..c2g/1011109009 11t.:11119///1143110
Jingke Wang, Tengju Ye, Ziqing Gu, and Junbo Chen. Lip: Lane-based trajectory prediction for au- tonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17134-17142, 2022.
Mingkun Wang, Xinge Zhu, Changqian Yu, Wei Li, Yuexin Ma, Ruochun Jin, Xiaoguang Ren, Dongehun Ren, Mingxu Wang, and Wenjing Yang. Ganet: Goal area network for motion fore- casting. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK. May 29 June 2, 2023, pages 1609-1615, IEEE, 2023a. doi: 10.1 10911194489999.2000 10160468. URLhttss:: /oi. owo.oorgg/ooo 11111111//////448111111 2023 101 60468
ICRA48891
Xishun Wang, Tong Su, Fang Da, and Xiaodong Yang. Prophnet: Efficient agent-centric motion forecasting with anchor-informed proposals, 2023b.
Maosheng Ye, Jiamiao Xu, Xunnong Xu, Tengfei Wang, Tongyi Cao, and Qifeng Chen. Bootstrap motion forecasting with self-consistent constraints, 2023.
Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector trans- former for multi- multi-agent motion prediction. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 8823-68 8833,883,30 2022.
Zikang Zhou, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Query-centric trajectory predic- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11863-178777 June 2023.
11

---

Published as a conference paper at ICLR 2024
7 APPENDIX
7.1 RESULTS OF ABLATION STUDIES
Results of ablation studies for more evaluation metries are shown in the following figures, which further corroborated the conclusions drawn in the main text.
Scroch
Scrtch
0.20
Scrrch
1.5
MTM MRM
0190
MTM
MRM
MRM
MIM
MTM--MRM
MTM-MIRM
MTM-MICM
minFDE
1.3
MRM+TP
MTS-TI
TP
minADEe
0.80
MRM-1P
MTM-IP
TP
MRa
0.16
TP
MRM-TP
MIM-TP
MTNS-VIMM-TP
MTTMMMIMM-1P
00.1
MTMM-MIM-TP
1.1
0.30
0.65
09
0.08
Epoch
Epech
Epach
(a) minFDEs
(b) minADEs
(c) MRg
Figure 7: The training curves report minFDEs, minADEs and MR, performance of all task com- binations. The solid or dashed lines correspond to the mean and the shaded regions correspond to I-sigma confidence interval over 5 runs.
1,000
1,050
1,050
1.025
1,025
1,025
minFDEs
minF7F2
0,950
0,950
0.950
0.2
044
0.6
0.8
0.3
0.4
0.5
0.66
0.7
0.8
2
4
10
Tempural Mask Ratio
Spotial Mask Ratio
Visille History
(a) MTM
(b) MRM
(c)
Figure 8: Final minFDE, performance of varying mask settings for each task, averaged over runs.
0.69
0.69
0.69-
0.68
minADEs
0.68
0.68
0.67
0.67
0.67
0.66
0.66
0.66
0.65
0.2
0.4
0.6
08
0.4
0.7
2
Temporal Mack Rato
Spatial Misk Ratio
Visible History
(a) MTM
(b) MRM
(c) TP
Figure 9: Final minADEs performance of varying mask settings for each task, averaged over runs.
12

---

Published as a conference paper at ICLR 2024
0.0055
0100755
0.0975-
0.0000
MR:
0.0050
MRz
0 0.099))
0.0925-
0.0900
0.0800
0.0900-
23
5.0
10.0
Tauporal
Special
Visible History
(a) MTM
(h) MRM
(c) TP
Figure 10: Final MRs performance of varying mask settings for each task, averaged over runs.
7.2 IMPACT OF PRETRAINING DATASET
To assess the impact of pretraining data on the downstream task, we compare the models pretrained on varying datasets: (a) train set, (b) train and validation set, and (c) train, validation and test set (consistent with the main experiment's setting). The performances are evaluated on the validation set, as reported in Figure and Table Our preliminary observations offer two insights. Firstly, there is only minor performance differences between setting (a) and (b). This suggests that the improvement brought about by pretraining cannot be simply attributed to exposure to the dataset for evaluation. Secondly, augmenting pretraining with additional data from the test set also improves performance validation set. This hints the notion that pretraining might derive benefits from more diverse dataset, aligning with the discoveries in LLM pretraining studies.
2.1
1.4
train
train val
2.0
train
val tost
b-minRDe
1.9
Dis
1.2
1.8
1.1
0.9
20
30
50
10
20
40
50
Epach
Epoch
(a) D-minFDE
(b) minFDEg
train val
train val
val
0.80
minADEs
0.14
MRa
0.12
0.10
0.65
40
Epach
Egoch
(c) minADEs
(d) MRs
Figure 11: The training curves report b-minFDEg, minFDEg. minADE, and MR, performance of models pretrained with different pretraining dataset settings. The solid lines correspond to the mean and the shaded regions correspond to 1-sigma confidence interval over 5 runs.
13

---

Published as a conference paper at ICLR 2024
Data split
Final evaluation metrics minADE,
Train
Validation
Test
b-minFDEB,
minFDE;
MR;
1.577
0.650
0.937
0.088
1,569
0,647
0.931
0.087
1.563
0.643
0.927
0.086
Table 5: Final performance of different pretraining dataset settings.
7.3
HYPERPARAMETERS
Table 6 reports the hyperparameters for the SEPT network architecture. In all transformer layers, layer norm is done prior attention and feedforward operations, and bias is cancelled in feedforward networks.
Arch
Parameters
Values
Projection
output size
256
depth
TempoNet
num head
8
64
depth
2
SpaNet
num head
8
dim_head
64
depth
3
Cross attender
num-head
dim
64
query size
256
num hidden
MLP traj/ / MLP score
hidden size
512
Table 6: Network hyperparameters
14


Reviews:
[{"review_id": "fvLVjY2pW5", "reviewer": "Reviewer_JZgu", "summary": "This paper presents a modeling framework (SEPT) that leverages self-supervised learning to extract spatiotemporal relationships for traffic motion prediction. It involves three masking-reconstruction modeling tasks on scene inputs including agents’ trajectories and road network, pretraining the scene encoder to capture kinematics within a trajectory, spatial structure of road network, and interactions among roads and agents. The proposed approach is evaluated on two public datasets, i.e., Argoverse 1 and Argoverse 2, and achieves state-of-the-art performance.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This work is well motivated in that the encoders built on universal architectures can develop a strong comprehension of traffic scenes through a properly designed training scheme. It leverages self-supervised learning to progressively develop the spatiotemporal understanding of traffic scenes.\n- The proposed method is novel and technically sound.\n- The effectiveness of the proposed SEPT is demonstrated by solid experiments on two of the largest datasets. It achieves significant improvement over the model learned from scratch on all motion forecasting metrics consistently.\n- Extensive ablation study shows that the three tasks collaborate and yield positive effects on the final performance.\n- This paper provides a good literature review, problem formulation, and discussion.\n- This paper is well organized and presented, and thus easy to follow.", "weaknesses": "- When it comes to model efficiency, the paper only compares with QCNet which is one of the largest models among all comparison methods. Since time-consuming is one of the main drawbacks of previous methods as is claimed in Section 1, it would be good to know the exact inference speeds of SEPT and the other comparison methods.\n- It would be good if the authors could talk more about their research findings in addition to presenting their approach, which I believe will inspire other researchers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TROf6PnDOT", "reviewer": "Reviewer_fR6X", "summary": "The paper presents SEPT (Scene Encoding Predictive Transformer), a framework for motion prediction crucial for autonomous vehicles operating in complex traffic environments. SEPT leverages self-supervised learning to develop a powerful spatiotemporal understanding of complex traffic scenes. The approach involves three masking-reconstruction modeling tasks on scene inputs, including agents’ trajectories and road networks, pretraining the scene encoder to capture kinematics within trajectories, spatial structures of road networks, and interactions among roads and agents. SEPT, without elaborate architectural design or manual feature engineering, achieves state-of-the-art performance on the Argoverse 1 and Argoverse 2 motion forecasting benchmarks, outperforming previous methods on all main metrics by a significant margin. The pretrained encoder is then fine-tuned on the downstream forecasting task, demonstrating that the model gains beneficial knowledge through pretraining objectives.", "questions": "1. Regarding the related work, you’ve cited that the lane mask SSL task of Forecast-MAE didn’t exhibit considerable advancement in the ablation study. However, in Traj-MAE, there was a significant enhancement noted in the HDMap mask. Isn't the consistent improvement across all SSL tasks already an established finding prior to this research?\n2. In table 4, there is a noted reduction of 6.4% in minADE on the validation set relative to the baseline without SSL pretraining. Meanwhile, SEPT shows a 10.1% and 12.3% decrease in minADE compared to Traj-MAE and Forecast-MAE, respectively. Could you elucidate whether the baseline model, even without the SSL pretraining, has already exceeded the performance of previous state-of-the-art methods? And what are the primary contributors to this enhancement in performance?\n3. On the other hand, it seems that in Traj-MAE, its pretraining method improved the baseline's 0.732 minADE to a final 0.604 minADE, marking a relative decrease of 17.5%. Would this imply that Traj-MAE's pretraining technique holds a comparative advantage or superiority over SEPT’s methodology?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. SEPT achieves state-of-the-art performance on two large-scale motion forecasting benchmarks (Argoverse 1 and Argoverse 2), outperforming previous methods by a significant margin in terms of various metrics.\n2. The ablation studies in the paper show that all combinations of the introduced tasks consistently improve the model’s performance. Each task contributes additively to the enhancement of the model's predictive accuracy.\n3. The manuscript is well-written overall and presents its content in a clear manner.", "weaknesses": "1. The Introduction effectively illustrates the potential of SSL in motion prediction tasks. However, it could benefit from acknowledging previous works such as Traj-MAE and Forecast-MAE that have already explored SSL in motion prediction. The three self-supervised tasks presented appear to be inspired or derived from these preceding studies.\n2. The framework presented in the paper is well-organized and operational, but it seems to lack novelty and groundbreaking contributions in the area of study.\n3. It's observed that the predominant improvement in SEPT's performance relative to the previous state-of-the-art isn’t chiefly attributed to SSL pretraining. This seems to be somewhat divergent from the central thesis and focus of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eeGECGqmsG", "reviewer": "Reviewer_siKy", "summary": "The paper presents a modeling approaching for the motion prediction problem for AVs that involves pre-training on self-supervised training objectives (masking and prediction of trajectory and road-graph portions) followed by fine-tuning for the downstream motion prediction task. \n\nPre-training is multi-task, with the masking of intermediate trajectory steps towards capturing motion kinematics, masking of road network vector attributes for better road topology and connectivity understanding, and sequential step masking of trajectory steps to create a motion prediction problem similar to downstream and capture spatio-temporal relationships between agent motion and road network.\n\nThis setup and the pre-training tasks with an ablation study of each of their contributions towards the final metrics are the main contributions of the paper. The results are demonstrated on Argoverse 1 and 2 challenges, where the paper achieves SOTA results with a convincing margin.", "questions": "Mainly the one mentioned in the \"Weaknesses\" section:\n- What is the impact on the results when not using the test set for pre-training?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Pre-training and self-supervised learning for the motion prediction task are under-explored areas, and this paper sets a strong benchmark with strong results on a widely used public leaderboard.\n\n- The authors have opted for a simple stacked-transformer architecture without any bag-of-tricks, that helps keep the focus on the contributions from the pre-training tasks and setup.\n\n- The ablation studies are extensive and well set up, providing insights into contributions from the each of the proposed components.\n\n- Overall presentation of the paper is good and it is easy to follow. Along with the simplistic architecture, it would allow others to reproduce and build upon this work.", "weaknesses": "- The paper mentions using the test set (unlabeled wrt the actual task) for self-supervised learning. While the test set from Argoverse *is* available to users, the intention is to provide it so that users can run their own inference and submit the predictions. Using the unlabelled part of this dataset (road network, agent motion histories) for pre-training could be giving the model an unfair advantage. I would encourage the authors to report the results without using the test set anywhere in their training / validation, or at the very least report an ablation result without using the test set for pre-training.\n\n- While results are reported on both Argoverse 1 and 2, it would be good to see generalization of the proposed methods to at-least one other dataset such as waymo open dataset or nuScenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "riwEh7rzbk", "reviewer": "Reviewer_Ukgp", "summary": "This paper combines several self-supervised reconstruction objectives as pretraining for motion prediction. The proposed method obtains state-of-the-art performance on two public leaderboards.\n\n\nUpdate:\nThis work demonstrates great performance and their findings might benefit the community, though the structure and idea is not entirely new. I change my score to 6.", "questions": "1. Could the authors explain more about the difference between your method and Traj-MAE/ForecastMAE and discuss why your method better performance than them? It might be the case that \"the devil is in the details\" and I think it is important to find out these details as your contributions. Otherwise, the proposed methodology is basically the same as Traj-MAE/ForecastMAE, which has nothing new.\n\n2. Do the authors plan to open source the code so that the community could easily reproduce the results and find out the working part? \n\n3. Missed reference: it seems that some classic transformer based works are not mentioned, which might cause wrong understanding of the contributions of this work:\n\n[D] Motion Transformer with Global Intention Localization and Local Movement Refinement. NeurIPS 2022 Oral\n\n[E] HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding. IEEE TPAMI\n\n[F] Multi-modal motion prediction with transformer-based neural network for autonomous driving. ICRA 2022\n\n\nIn summary, the proposed method has strong emperical performance on the public leaderboards but the method described in the paper basically has nothing new compared to existing methods, which makes their contributions unclear. Since the authors neither provide any detailed comparisions and analysis with existing methods nor mention their plans for open source, I do not think the paper could provide much value to the community, thus I give a borderline reject.  If the authors could solve the aforementioned issues, I may change my scores accordingly.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Strong emperical performance. The proposed method ranks top on two widely used leaderboards.", "weaknesses": "- Limited novelty: The model structure is not new, which is basically SceneTransformer [A], which should be considered as contributions. For the three pretraining objectives, it is very similar to Traj-MAE[B] and ForecastMAE[C], as stated by the authors as well. Compared to [B] and [C], they achieve better performance, but the paper does not give a thorough analysis regarding the difference. As a result, it is hard to determine the real working part of the method.\n\n[A]Scene Transformer: A unified architecture for predicting multiple agent trajectories. ICLR 2022\n\n[B]Traj-MAE: Masked Autoencoders for Trajectory Prediction. ICCV 2023\n\n[C]Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoders. ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)