JointMotion: Joint Self-Supervision for Joint Motion Prediction
Authors: Royden Wagner
Omer Sahin Tas
Marvin Klemp
Carlos Fernandez
Venue: CoRL 2024 (2024)
Keywords: Self-supervised learning
representation learning
multimodal pre-training
motion prediction
data-efficient learning
TL;DR: Self-supervised pre-training method for joint motion prediction in self-driving vehicles.

Abstract: We present JointMotion, a self-supervised pre-training method for joint motion prediction in self-driving vehicles. Our method jointly optimizes a scene-level objective connecting motion and environments, and an instance-level objective to refine learned representations. Scene-level representations are learned via non-contrastive similarity learning of past motion sequences and environment context. At the instance level, we use masked autoencoding to refine multimodal polyline representations. We complement this with an adaptive pre-training decoder that enables JointMotion to generalize across different environment representations, fusion mechanisms, and dataset characteristics. Notably, our method reduces the joint final displacement error of Wayformer, HPTR, and Scene Transformer models by 3%, 8%, and 12%, respectively; and enables transfer learning between the Waymo Open Motion and the Argoverse 2 Motion Forecasting datasets.

JointMotion: Joint Self-Supervision for Joint Motion Prediction
Royden Wagner!:*
Ömer Şahin Tas**
Marvin Klemp'
Carlos Fernandez'
Karlsruhe Institute of Technology FZI Research Center for Information Technology
Abstract: We present JointMotion, self-supervised pre-training method for joint motion prediction in self-driving vehicles. Our method jointly optimizes scene- level objective connecting motion and environments, and an instance-level ob- jective to refine learned representations. Scene-level representations are learned via non-contrastive similarity learning of past motion sequences and environment context. At the instance level, we use masked autoencoding to refine multimodal polyline representations. We complement this with an adaptive pre-traime pre-training de- coder that enables JointMotion generalize across different environment repre- sentations, fusion mechanisms, and dataset characteristies. Notably, method reduces the joint final displacement error of Wayformer, HPTR, and Scene Trans- former models by 3%, 8%, and 12%, respectively; and enables transfer learn- ing between the Waymo Open Motion and the Argoverse Motion Forecasting datasets. Code: https://githiii con.//iitaatt/tuuuee-mooott
Keywords: Self-supervised learning, representation learning, multimodal pre- training, motion prediction, data-efficient learning
(a)
Motion sequences
Joint scene representation
Environment context
MM
Encoder
MM
Proj
2
2F
Proj
H
Encoder
X
(b)
Ix, I. w. a, v. Oyaw mu cal [mask]...
b,yy
Ix. y. ny cTul
[mask]-
....---ma [mask]
Figure 1: JoinnMotionn. (a) Connecting motion and environments: Our scene-level objective learns joint scene representations via non-contrastive similarity learning of motion sequences M and en- vironment context E. (b) Masked polyline modeling: Our instance-level objective refines learned representations via masked autoencoding of multimodal polyline embeddings (i.c., motion, lane, and traffic light data).
"Joint first authors. Corresponding author: royden. wagner edu
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
Self-supervised learning (SSL) [1] by design excels in applications with large amounts of unlabeled data and limited labeled data (e.g.. [2, 3, 4)). However, recent SSL methods combined with su- pervised fine-tuning outperform plain supervised learning with the same amount of data [5, 6] and shorter training times [7]. This makes SSL a versatile choice improve existing methods in wide range of applications.
In this work, we focus on improving joint motion prediction for self-driving vehicles. Motion pre- diction aims to predict the future motion of traffic agents given their past motion and environment context, such as lane data and traffic light states. Therefore, motion prediction is essential for es- timating future interaction among traffic agents [8] and subsequently assessing the risk of planned trajectories [9, 10]. The majority of recent motion prediction methods perform marginal prediction, where future motion is predicted for each agent individually (e.g.. [11, 12]). In contrast, joint motion prediction predicts scene-wide motion modes (i.e., capturing joint set of future motion sequences for multiple agents), which enables interaction modeling.
Since joint motion prediction requires scene-wide representations, we propose scene-level SSL ob- jective connecting motion and environment. We combine this with an instance-level SSL objective that refines learned representations, enhancing overall prediction accuracy. As scene-level objective, match past motion sequences in scene the corresponding environment context, such map data and traffic light states. Thereby, model learns which sets of motion sequences are likely in given enviromment including interaction among agents. As instance-level objective, we reconstruct masked polylines representing past motion sequences, lanes, and past traffic light states. We com- plement this with an adaptive pre-training decoder, which leads to the high generalizability of our method.
Overall, our main contributions are the design choices that enable our method generalize across
(a) environment representations (i.e., scene-centric, pairwise relative, and agent-centric) en- abled by our complementary scene- and instance-level pre-training objectives,
(b) information fusion mechanisms (i.e., early and late fusion) enabled by the combination of our adaptive pre-training decoder and our instance-level objective, and
(c) dataset characteristics (i.e., varying sequence lengths and feature sets) enabled by the trans- ferability of learned representations.
2 Related work
SSL for motion prediction in self-driving vehicles. Inspired by the success of SSL in computer vision (e.g., [3, 4, 6]) and natural language processing (e.g., [2, 13, 14]), related methods apply SSL to pre-train motion prediction models for self-driving. PreTrra [15] learns joint embed- ding space for trajectory and map data via contrastive learning. During pre-training, the similarity of trajectory and map embeddings from the same traffic scene maximized, while the similarity to embeddings from other scenes minimized (i.e., negative examples). This objective is coun- terproductive when similar scenes are sampled in a mini-batch, e.g., the map embeddings of two four-way stops should be similar, but are erroneously used as negative examples against each other. [17], and RMP [18] are masked autoencoding
The concurrent works Traj-MAE [16], Forecast- methods for pre-training motion prediction models. They mask trajectory and/or lane tokens and learn to reconstruct them. Traj-MAE and Forecast-MAE represent motion sequences as trajectory polylines, where each polyline point represents only the agents' position omitting available features such as velocity and acceleration profiles, agent dimensions, yaw angles, and temporal order (see Fig. (b)). Furthermore, they do not include traffic light states as environment context. SSL.-Lanes [19] extends lane masking with the objectives of distance intersection prediction, maneuver and success/failure classification. Although these objectives are well adapted to motion prediction, they require non-trivial heuristics (c.g., for clustering maneuvers).

---

Joint motion prediction for self-driving vehicles. Joint motion prediction aims to predict the joint distribution of future motion sequences over multiple agents in a traffic scene. Thus, a predicted motion mode represents a scene-wide set of motion sequences with one sequence per agent. Scene Transformer [20] is an encoder-decoder transformer model for joint motion prediction. The en- coder learns global scene-centric representations (i.c., agent, lane, and traffic light features), the decoder transforms these embeddings into joint motion modes. Global position embeddings used for the learned scene-centric representations, which are more difficult to learn than pairwise relative position embeddings (cf. [21, 22]). MotionI. [23] reformulates motion prediction as lan- guage prediction task and learns a vocabulary of discrete motion vectors. Joint prediction modes are generated by autoregressively decoding sequences of motion vectors for multiple agents. Mo- tionDiffuser [24] performs joint motion prediction as conditional denoising diffusion process. A set of noisy trajectories (i.e., positions disturbed by Gaussian noise) transformed by a denoiser into of trajectories that approximates the ground truth joint prediction mode. The learned denoiser conditioned environment context such lane data and traffic light states. Both MotionLM and MiffonDiffuser use an agent-centric encoder [12], which leads to repeated computation since a traffic scene is encoded for each agent individually. Furthermore, their best performing variants exhibit high inference latency (MotionLM: 250 ms?, MotionDiffuser: 409 ms).
Marginal motion prediction with auxiliary interaction prediction objectives. marginal motion mode represents single motion sequence for agent. Many recent methods extend marginal prediction models with modules for interaction prediction objectives (i.e., dense [25, 26, 27] or conditional prediction [28, 29]). In such methods, the main prediction modules are still trained in a marginal manner, where training targets are marginal motion modes per agent. The additional modules are trained to combine marginal predictions or the corresponding latent representations by minimizing overlap between them. Thus, these methods do not learn to model scene-wide joint motion modes from ground up. but how combine marginal modes with less overlap. This limits interaction modeling the learned re-combination of marginal motion predictions.
3 Method
We present our method for self- self-supervised pre-training of motion prediction models in two steps. First, we describe our scene-level SSL objective (connecting motion and environments), then, our instance-level SSL objective (masked polyline modeling) and the adaptive pre-training decoder.
3.1 Connecting motion and environments
Joint motion prediction requires scene-wide representations decode joint modes, which include predictions of multiple agents within scene. Therefore, we propose pre-training motion prediction models by connecting motion and environments (CME). This scene-level objective aims to learn joint embedding space for motion sequences and environment context (see Figure I (a)). Thus, model implicitly learns which motion is likely in a given environment, including traffic rules and interaction among agents.
In detail, use the past motion of all agents within scene generate scene-level motion embed- ding M in Figure 1) and combine lane data and traffic light states to corresponding environment embedding (Z& in Figure 1). These embeddings are generated by modality-specine encoders (i.e., for motion, lanes, and traffic lights) followed by global average pooling and an MLP-based pro- jector (Proj in Figure 1). We perform average pooling on the intermediate embeddings (H M and E in Figure 1) to be invariant to variations in the number of agents and the complexity envi- ronments. Following [30], we use two separate MLPs with LayerNorm and ReLU activations as projectors, each having hidden dimension of 2048 and output dimension of 256. We the -ppelity-specific encoders of recent motion prediction models (e.g., [20, 21)) without modifica- tions and remove the additional projector after pre-training. The joint embedding space is learned
"This latency is linearly extrapolated, as inference latency is only reported up the second best/largest model (see Table 8 in the Appendix of [23])
3

---

by similarity learning via redundancy reduction. Following [31, 32], we reduce the redundancy of vector elements per embedding (i.e., for M and 2 individually) and maximize their similarity by approximating the cross-correlation matrix C of ZM and 2 to the corresponding identity matrix:
LCMB Apd ΣΣOH s7i
E(I
where i,j index the vector dimension of the embeddings Z. The redundancy reduction term is scaled by Ared and ensures that individual embedding elements capture different features. Therefore, it pre- vents representation collapse with trivial solutions for all embeddings across all scenes (c.g.. zero vectors). Non-trivial yet identical solutions are not explicitly prevented, but are unlikely, as empiri- cally shown [31]. In contrast to [15], our scene-level objective does not require negative examples, which are difficult define in this context. Unlike [32], this objective maximizes the similarity of embeddings from different modalities (i.c., motion and environment) rather than augmented views of the same modality, removing the requirement to develop suitable augmentations.
3.2 Masked polyline modeling with adaptive decoding
Scene-level representations are well suited to provide an overview of traffic scenes, but lack instance- level details. For example, the exact position of traffic agents in the past or lane curvatures. There- fore, we combine our scene-level objective with the instance-level objective of masked polyline modeling (MPM) to refine learned representations.
Inspired by masked sequence modeling [2, 6], we mask elements of polylines representing past mo- tion sequences, lanes, and past traffic light states and learn reconstruct them from non-masked elements and environment context. As shown in Figure I, we represent traffic agents with 10 fea- tures rather than just past positions (cf. [16, 17J). In detail, we reconstruct agent positions (z, v), dimensions (I, w, h), acceleration a, velocity v, yaw angle fyaw. temporal order n, and classes cs (i.e., vehicles, cyclists, and pedestrians). For lanes, we reconstruct positions (x, y) and lane classes a. For traffic light state sequences, we reconstruct positions (z, y). temporal order nh, and state classes CTL (i.e., green, yellow, and red). Following [21], positions, dimensions, accelerations, ve- locities, and yaw angles represented float values and temporal order and agent classes as boolean one-hot encodings.
For models with late or hierarchical fusion mechanisms (e.g., [20, 21)). we use the modality-specific encoders to generate embeddings per modality (see Figure 2 (a)). Afterwards, we concatenate these embeddings and use a shared local decoder reconstruct masked sequence elements from non- masked elements and context from other modalities. As local decoder, we use transformer blocks with PreNorm [33], local attention [34] with attention window of 32 tokens, 8 attention heads, rotary positional embeddings [35], and FeedForward layers with an input and hidden dimension of 256 and 1024.
For models that employ early fusion mechanisms (e.g., [12]), we use learned queries and shared decoder reconstruct the input sequences (see Figure 2 (b)). Such models learn a compressed latent representation for multi-modal input. Therefore, we learned queries in the same number input tokens to decompress these representations and reconstruct the input sequences. We use regular cross-attention mechanism between the learned queries and compressed latent representations and a local self-attention mechanism within the set of learned queries. The resulting transformer blocks have the same hyperparameters as in the late fusion setup.
For both variants, we use random attention masks for masking and masking ratio 60%. As training target, minimize the Huber loss between the reconstructed polylines and the input polylines EMPM AALA + XLCL + CT. If not specified otherwise, we set dA dL Xn =

---

(8)
(b)
(uskk]
Encoder
Shared
Shared
local
[mask]
Encoder
local
[mask]]
Shared
decoder
encoder
decoder
[mask]
Encoder
[mask]
Learned queries
Figure 2:
Adaptive decoding for masked polyline modeling with late and early fusion encoders. (a) Late fusion with modality-specific encoders for agents (Encoder"), lanes (Encoder"), and traffic lights (Encoder").. (b) Early fusion with shared encoder for all modalities. Compressed features decoded using learned query tokens.
Experiments
4.1 Comparison of self-supervised pre-training methods for joint motion prediction
In this experiment, we compare our method with recent self-supervised pre-training methods for autonomous driving using the WOMD dataset. Specifically, we compare with contrastive learning via Pre'TraM [15] and masked autoencoding with Forecast-MAE [17] and Traj-MAE [16].
Motion prediction models. We pre-train and fine-tune well-established Scene Transformer [20] models the WOMD training split. We use the publicly available implementation by [21] with 3 mppelic encoders (i.e., for agents, lanes, and traffic lights). Adapted to the complexity modality, we encode traffic light features with 1, agent features with 3, and lane features with 6 transformer blocks.
Pre-training, For all methods, we add pre-training decoder for late fusion models (see Figure 2) with 3 transformer blocks and the hyperparameters described in Section 3.2. For our method, additionally add two projectors for scene-level representations described in Section 3.1. For Pre- TraM, we follow its trajectory-map contrastive learning configuration and add two linear projection layers as projectors for trajectory and map embeddings. For Forecast- MAEE we use masking ratio of 60% and reconstruct positions of lane polylines and past trajectories by minimizing the MSE loss. We exclude future trajectories, since self-supervised learning by design does not use the same labels as the intended downstream task (ef. [1)). For Traj-MAE, we use a masking ratio of 60% and reconstruct positions of lane polylines and past trajectories by minimizing the cor- responding Huber loss. For our method,
minimize the joint loss of our proposed objectives
LoimtMotion = ACME + LMPM- We set ACME 0.01 and following [31] the weight of the redundancy reduction term Aped = 0.005.
Fine-tuning. For all methods, we replace the pre-training decoder with a shared global decoder and learned anchors for
motion modes. We initialize the modality-speeiffe encoders with the learned weights from pre-training and do not freeze any weights during fine-tuning. We fine-tune the model using its joint configuration and hard loss assignment. Accordingly, the loss is computed for the best scene-wide joint prediction mode. As post-processing, we follow [36] and adjust the confidences of redundant predictions.
Training time, hardware, and optimizer. For all methods, we perform pre-training for 10 hours and fine-tuning for 23.5 hours using training server with A100 GPUs. For pre-training and fine- tuning, AdamW [37] with initial learning rate of le-4 and step learning rate scheduler with reduction rate of 0.5 and step size of 25 epochs.
Results. Table 1 shows the results of this experiments. Explicit scene-level objectives (i.e., Pre- TraM and JointMotion) lead to better and more balanced performance across all agent types, while implicit scene-wide masked autoencoding with Forecast-MAF or Traj-MAE tends to focus more on the pedestrian class than on the others (see mAP scores). Therefore, likely that explict scene- wide objectives enforee learning interactions between varying agent types more. Traj-MAE and our method without the objective of connecting motion and environments (JointMotion w/o CME)
5

---

achieve better scores than Forecast-MAE. Consequently, reconstructing individual elements of poly- lines improves representations more than reconstructing whole polylines. The superior performance of JointMotion w/o CME compared to Traj-MAE indicates that extending the motion feature set and reconstructing traffic light states as well further improves learned representations. Our method with- masked polyline modeling (JointMotion w/o MPM) performs on par with PreTraM, indicating that redundancy reduction can replace negative examples for scene-level similarity learning.
Pre training
mAPt
minADE L
minFDE L
avg
eye
ped
veh
eye
veh
cye
ped
veh
None
0.1596
0.1465
0.1821
0.1504
1.522
0.698
1.486
3.653
1,684
3.476
Forecast MAE(17]
0.1592
1.652
Trij TIH-MMFF [16]
0.1677
0.1492
0.1917
0.1623
1.421
0.708
1.338
3.320
1.647
3,090
PreTraM [15]
0.1689
0.1724
JoiniMation wlo MPM
0.1689
0.1762
0.1777
0.1528
1.478
0.684
1.406
3.507
1,608
3.287
JointMotion
0.1784
0.1652
0.1903
1.457
1.630
JoiniMotion
0.1940
0.1970
0.1964
0.1886
1,343
0.677
1,288
3.095
1,583
2,941
Table 1: Comparison of self-supervised pre-training methods for joint motion prediction. All methods are used to pre-train Scene Transformer models [20] on the Waymo Open Motion dataset and evaluated on the validation split. Agent types: cyclist (cyc), pedestrian (ped), and vehicle (veh). Best scores are bold, second best are underlined.
Overall, pre-training with both proposed objectives (i.e., JointMotion) leads to the best scores across all agent types. This shows that the combination of our objectives works best. Figure further highlights the complementary nature of our two objectives. Specifically, the reconstruction of past traffic light sequences are learned very similar with both configurations. The reconstruction loss for past agent motion converges more slowly with JointMotion pre-training, but reaches similar values as well. However, with our scene-level objective, the lane reconstruction loss likely converges to higher value. We hypothesize that models pre-trained with our scene-level objective tend to focus more on the overall lane structure than on specific details of individual lane polylines.
10
10
-10
6
-
-1
Step
Step
Step
2
-104
-104
-10
Figure 3: Loss plots our complementary pre-training objectives. The green curve represents JointMotion w/o CME, while the blue curve represents JointMotion. Consistent with the remainder of the document, L stands lanes, TL. stands for traffic lights, and stands for agents.
mAP
Pre-truining
Fine-tuning
Training from scratch
0.2
0.15
Time in hours
Figure 4: Accelerating and improving training via SSL. Scene Transformer models pre-trained with JointMotion achieve higher mAP scores on WOMD than models trained from scratch.
Figure 4 shows that Scene Transformer models pre-trained with our method achieve higher mAP scores on WOMD than models trained from scratch, even in shorter wall training time.

---

4.2 Comparing scene-level self-supervision methods
In this experiment, we further compare the scene-level objectives PreTraM [15] and JointMotion. We train Scene Transformer, HPTR, and joint configuration of Wayformer cover all common types of environment representations in motion prediction (i.e., scene-centric, pairwise relative, and agent-centric).
Experimental setup. For Scene Transformer, we use the same configuration as in the previous ex- periment. For HPTR, we analogously add modality-speciifi encoders and use a shared decoder for k = 6 motion modes. For the joint configuration Wayformer, we follow [24] and use shared encoder for early fusion, which compresses the multi-modal input to 128 tokens. We concatenate multiple such agent-centric embeddings with positional and rotation information into common reference frame and use a shared decoder to predict joint motion modes. For pre-training the Way- former model, we add our decoder for early fusion configurations (see Figure 2). The Wayformer model is not pre-trainable with Pre'rraM since an instance-level objective is required to decode the modality-speciinn tokens from fused representations (cf. Scction 3.2). For all models, we employ the same hardware, training time, optimizer, and learning rate scheduling as in the previous experiment. Evaluation is performed on the interactive validation split of WOMD and AV2.
Results. Table 2 shows the results of this experiment. Our method consistently outperforms Pre- TraM using different models with varying environment representations and fusion mechanisms. Our method improves all models, while the improvement of the scene-centric Scene Transformer model (c.g.. 12% lower minFDE) is most significant and the improvement of the agent-centric Wayformer model least significant (e.g., 3% lower minFDE). Hence, the improvements are inversely propor- tional to the sample efficiency of the models. In scene-centric modeling, one sample is generated per scene, whereas in agent-centric modeling, one sample is generated for each traffic agent within scene. This aligns with the finding that fine-tuning with more samples generally reduces the value of pre-training [38].
Datoset
Model (config)
Pre-training
minFDE L
minADE.1
MR. J
OR
3.6715
1.5255
0.7372
Scene Transformer
PreTraM [15]
3,6508
-0.57%
1.5415
1.05%
0.7385
0.18%
0.2915
1.64%
JoinMotion
3,2400
-11.5%%
1,3830
-9.36%
0,7090
-3.82%
0.2847
-0.73%
WOMD
None
2.6003
1.1682
0.6030
0.2331
HPTR
PreTraM [15]
2. 2,5049
-3.66%
1,0981
-5. -5.99%
0.5863
-2.78%
0.2345
0.60%
JointMotion
2.4006
-7.68%
1.0564
0.5591
-7.28%
0.2297
-1.46%
Wayformer (joint)
Nore
2.3529
1.0209
0.5461
0.2273
JointMotion
2,2823
-3.00%
0.9939
-2.64%
0.5270
-3.50%
0.2143
-5.72%
AV2
HPTR
JointMution WOMD
Nooc
2,2550
2.1530
-4.53%
1.1380
1.1370
-0.9%
0.1025
0.0988
3.75%
Table 2: Comparing scene-level self-supervision methods. All metrics are computed using the Waymo Open Motion interactive (WOMD) and Argoverse 2 Forecasting (AV2) validation splits. Best scores are bold.
Unlike PreTraM, the proposed adaptive pre-training decoder combined with our instance-level ob- jective enable our method to adapt to models with early fusion mechanisms (c.g., Wayformer). Specifically, modality-specifft masked polyline modeling with learned queries in the same number as input tokens enables our method to decode modality-speciffe tokens (i.e., agent, lane, and traffic light) from compressed latent representations. This is particular relevant since the current state- of-the-art methods on the Argoverse I Forecasting (ProphNet [11] via AiP tokens) and Argoverse 2 Forecasting (QCNeXt [39] via query-centric modeling) benchmarks rely on fusion mechanisms with compressed latent representations as well.
Furthermore, our pre-training leads to comparable improvements on the interactive and the regular validation splits (cf. Table 1), while pre-training with PreTraM leads to smaller improvements on the interactive validation split. We hypothesise that with additional instance-level objective, more fine-grain fine-grained trajectory details are learned, which is more important for close trajectories of
7

---

interacting agents. The lowest block shows that our method leads to transferable representations.
Specifically, pre-training on WOMD improves fine-tuning on AV2.
4.3 Comparison with state-of-the-art methods for joint motion prediction
In this experiment, we compare our method with state-of-the-ar methods for joint motion prediction in autonomous driving.
Experimental setup. We pre-train HPTR (configured as in Section 4.2) for 10 hours using JointMo- tion and fine-tune for 100 hours. We evaluate the methods in test and validation splits of WOMD. We use the official challenge website compute performance metries.
Results. Table 3 presents a comparative analysis of various state-of-the-ar methods for joint motion prediction on interactive splits of WOMD. In the test split, we compare four different approaches. The Scene Transformer has mAP of 0.1192 and outperformed by the other methods in all met- rics. GameFormer achieves higher mAP of 0.1376 and exhibits competitive performance in terms of minADE and minFDE. MotionDiffuser and JointMotion (HPTR) are close contenders, with Mo- tionDiffuser performing slightly better in all of the metrics compared to JointMotion (HPTR).
For the validation split, the results are similar. GameFormer (joint) scores mAP of 0.1339, with MotionLM (single replica) outperforming it in terms of mAP, achieving 0.1687. JointMo- tion (HPTR) presents an improved mAP of 1761 compared to its counterparts. Following [21], compare against the single replica model of MotionLM and show its ensembling version for reference. MotionL.M (ensemble) demonstrates superior overall performance, expected given the increased modeling capacity.
Split
Method (config)
Venue
mAP
minADE
minFDE І
MR J
OR
Scene Transforner (joint) [20]
ICLR'22
0.1192
0,9774
2.1892
0.4942
0.2067
GameFormer (joint) [40]
ICCV"23
0.1376
0,9161
1.9373
0.4531
0.2112
Test
MutionDifluser [24]
CVPR'21 23
0.1952
0.8642
1.9482
0,4300
0.2004
JointMotion (HPIR)
0.1869
0.9129
2.0507
0.4763
0.2037
GameFormer (joint) [40]
ICCV'23
0.1339
0.9133
1,9251
0.4564
MotionI.M M (single replica) (23)
ICCV*23
0.1687
1.0345
2.3886
0.4943
Val
JointMotion (HPTR)
0.1761
0,9689
2.2031
0.4915
0.1990
Motionl M (ensemble)
ICCV*23
0.21.50
0.8831
1,9825
0.4092
Table 3: Comparison with state-of-the--at methods for joint motion prediction. All methods are evaluated on interactive splits the Waymo Open Motion dataset. Following [21], we compare against single replica versions of joint prediction methods, ensemble versions are shown for refer- ence. Best scores are bold, second best are underlined.
5 Conclusion
In this work, we introduced SSL method for joint motion prediction of multiple agents within a traffic scene. Our SSL framework integrates scene-level and instance-level objectives that operate complementarily, enhancing the training speed and accuracy motion prediction models. Notably, JointMotion outperforms recent contrastive and autoencoding methods for pre-training in motion prediction. Moreover, our method generalizes across different environment representations, infor- mation fusion mechanisms, and dataset characteristics. Our evaluations demonstrate significant per- formance improvements over non-ensembling joint prediction methods or point-prediction variants of marginal-prediction architectures, underscoring the robustness and effectiveness of our proposed method.
Limitations. Our method is label-efficient as SSL methods in computer vision, which gen- erate supervisory signals from raw images (e.g.. [3]). Our pre-training does not include the down- stream labels for motion prediction (i.c., future motion), but relies past motion and map data.

---

Acknowledgments
The research leading to these results is funded by the German Federal Ministry for Economic Af- fairs and Climate Action within the project "NXT GEN AI METHODS Generative Methoden für Perzeption, Prädiktion und Planung". Furthermore, we acknowledge the financial support by the German Federal Ministry of Education and Research (BMBF) within the project HAlBrid (FKZ 011S21096A).
References
[1] R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bor- des, A. Bardes, G. Mialon, Y. Tian, et al. A Cookbook of Self-Supervised Learning. arXiv:2304. 12012000..2210,, 2023.
[2] Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, Neelakantan, Shyam, Sastry, A. Askell, et Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[3] T. Chen, S. Korbllihh, M. Norouzi, and G. Hinton. A Simple Framework for Contrastive Learn- ing of Visual Representations. In International Conference on Machine Learning (ICML), 2020.
[4] J.-B. Grill, F. Strub, Altché, C. Tallec, Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, al. Bootstrap Your Own Latent A New Approach to Self-Supervised Learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[5] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging Properties Supervisdd Vision Transformers. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[6] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked Autoencoders Are Scal- able Vision Learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[7] C. Feichtenhofer, Y. Li, He, et al. Masked Autoencoders As Spatiotemporal Learners. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[8] S. Casas, C. Gulino, S. Suo, K. Luo, R. Liao, and R. Urtasun. Implicit Latent Variable Model for Scene-Consistent Motion Forecasting. In European Conference on Computer Vi- sion (ECCV), 2020.
[9] H. Nishimura, J. Mercat, B. Wulfe, McAllister, and Gaidon. RAP: Risk-Aware Pre- diction for Robust Planning. In Conference Robot Learning (CoRL), 2023.
[10] Ó. S. Taş. P. H. Brusius, and C. Stiller. Decision-theoretie MPC: Motion Planning with Weighted Maneuver Preferences Under Uncertainty. arXiv: 2310. 2023.
[11] X. Wang, T. Su, Da, and Yang. ProphNet: Efficient Agent-Centric Motion Forecasting Proposals. In IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
With
[12] N. Nayakanti, R. Al-Rfou, A. Zhou, K. Goel, K. S. Refaat, and B. Sapp. Wayformer: Motion Forecasting via Simple & Efficient Attention Networks. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
[13] J. Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, 2018.

---

[14] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML), 2021.
[15] C. Xu, T. Li, C. Tang, L. Sun, K. Keutzer, M. Tomizuka, A. Fathi, and W. Zhan. PreTraM: Self-supervised Pre-training via Connecting Trajectory and Map. In European Conference on Computer Vision (ECCV), 2022.
[16] H. Chen, J. Wang, K. Shao, F. Liu, J. Hao, C. Guan, G. Chen, and P.-A. Heng, Traj-MAE: Masked Autoencoders for Trajectory Prediction. In /EEE/CVF International Conference on Computer Vision (ICCV), 2023.
[17] Cheng, X. Mei, and M. Liu. Forecast-MAE: Self-supervised Pre-training for Motion Fore- casting with Masked Autoencoders. In IEEE/CVF International Conference Computer Vision (ICCV), 2023.
[18] Y. Yang, Q. Zhang, T. Gilles, N. Batool, and J. Folkesson. RMP: A Random Mask Pretrain Framework for Motion Prediction. IEEE International Conference on Intelligent Trans- portation Systems (ITSC), 2023.
[19] P. Bhattacharyya, C. Huang, and K. Czarnecki. SSL-Lanes: Self-Supervised Learning for Motion Forecasting in Autonomous Driving. In Conference Robot Learning (CoRL), 2022.
[20] J. Ngiam, B. Caine, V. Vasudevan, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, et al. Scene Transformer: unified architecture for predicting multiple agent trajectories. In International Conference on Learning Representations (ICLR), 2022.
[21] 7. Zhang, A. Liniger, C. Sakaridis, F. Yu, and L. Van Gool. Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding, In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[22] A. Cui, S. Casas, K. Wong, S. Suo, and R. Urtasun. GoRela: Go Relative for Viewpoint- Invariant Motion Forecasting. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
[23] A. Seff, B. Cera, D. Chen, M. Ng. A. Zhou, N. Nayakanti, K. S. Refaat, R. Al-Rfou, and B. Sapp. MotionLM: Multi-Agent Motion Forecasting as Language Modeling, In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[24] C. Jiang, A. Comnman, C. Park, B. Sapp, Zhou, D. Anguelov, et al. MotionDiffusser: Control- lable Multi-Agent Motion Prediction Using Diffusion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[25] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling. In International Conference on Learning Representations (ICLR), 2022.
[26] S. Shi, L. Jiang, Dai, and B. Schiele. Motion Transformer with Global Intention Localiza- tion and Local Movement Refinement. In Advances Neural Information Processing Systems (NeurIPS), 2022.
[27] S. Shi, L. Jiang, D. Dai, and B. Schiele. MTR++: Multi-Agent Motion Prediction with Sym- metric Scene Modeling and Guided Intention Querying. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2024.
[28] Q. Sun, X. Huang, J. Gu, B. C. Williams, and H. Zhao. M21: From Factored Marginal Trajec- tory Prediction to Interactive Prediction. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
10

---

[29] W. Luo, C. Park, A. Commaan, B. Sapp. and D. Anguelov. JFP: Joint Future Prediction with Interactive Mueti-Agent Modeling for Autonomous Driving. In Conference on Robot Learning (CoRL), 2023.
[30] E. Fini, P. Astolfi, A. Romero-Soriano, J. Verbeek, and M. Drozdzal. Improved baselines for vision-language pre-training. Transactions on Machine Learning Research, 2023.
[31] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny. Barlow Twins: Self Seff-Super Learning via Redundancy Reduction. In International Conference on Machine Learning (ICML), 2021.
[32] R. Wagner, Ö. S. Taş, M. Klemp, C. Fernandez, and C. Stiller. RedMotion: Motion Prediction via Redundancy Reduction. Transactions on Machine Learning Research, 2024.
[33] T. Q. Nguyen and J. Salazar. Transformers without tears: Improving the normalization of self- attention. In Proceedings of the 16th International Conference on Spoken Language Transla- tion, 2019.
[34] I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The Long-Document Transformer. arXiv:2004.00 0500 2020.
[35] J. Su, Y. Lu, S. Pan, B. Wen, and Y. Liu. RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomputing, 568, 2024.
[36] Konev. MPA: MultiPath++ Based Architecture for Motion Prediction. WwXiv:226.100411 2022.
[37] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
[38] B. Zoph, G. Ghiasi, T.-Y. Lin, Y. Cui, Liu, D. Cubuk, and Q. Le. Rethinking pre-training and self-training. Advances in neural information processing systems, 2020.
[39] Z. Zhou, Z. Wen, J. Wang, Y.-H. Li, and Y.-K. Huang. Qenext: A next-generation framework for joint multi-agent trajectory prediction. arXiv preprint arXiv:2306. 10508, 2023.
[40] Z. Huang, H. Liu, and C. Lv. GameFormer: Game-theoretic Modeling and Learning of Transformer-basee Interactive Prediction and Planning for Autonomous Driving. IEEF/VVF International Conference on Computer Vision (ICCV), 2023.
[41] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, et al. Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[42] B. Wilson, W. Qi, T. Agarwal, Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hart- nett, J. K. Pontes, et al. Argoverse 2: Next Generation Datasets for Self Self-Driving Perception and Forecasting. arXiv:2301.0045 2023.
Appendix
Evaluation metrics. Following [41], we use the mean average precision (mAP), the average dis- placement error (minADE), the final displacement error (minFDE), miss rate (MR), and overlap rate (OR) to evaluate motion predictions. All metrics are computed using the minimum mode for = modes. Accordingly, the metrics for the mode closest to the ground truth are measured. We use the official challenge website compute metrics on WOMD, For the AV2 dataset, we use the evalua- tion software provided by [21]. Therefore, joint modes (best scene-wide mode) evaluated on the interactive splits and marginal modes (best mode for each agent individually) on the regular splits.
11

---

The Waymo Open Motion Dataset (WOMD) [41] is comprised of over 1.1 million data points extracted from 103,000 urban or suburban driving scenarios, spanning 20 seconds each. The state of object-agents includes attributes like position, dimensions, velocity, acceleration, orientation, and angular velocity. Each data point captures second of past followed by seconds of future data. We resample this time interval with 10Hz.
The Argoverse Motion Forecasting Dataset (AV2) [42] is comprised of 250,000 urban or sub- urban driving scenarios each spanning 11 seconds, with 5 seconds of past and a seconds of future data. The datasets entails interactions in over 2,000 km of roadways across six geographically di- verse cities.
12


Reviews:
[{"review_id": "DzeQEfVWI7", "reviewer": "Reviewer_ffHe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents JointMotion, a self-supervised pre-training method for joint motion prediction in self-driving vehicles. The main idea is to improve the prediction accuracy of traffic agents' future motion by learning joint scene representations and refining learned representations through complementary scene-level and instance-level objectives.", "summary_of_recommendation": "I recommend for a weak accept of this paper.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "* Could you kindly elaborate on why explicit scene-level objective methods typically tend to yield better and more balanced performance across all agent types in comparison to implicit methods?\n* Could you please explain why there is a worse performance of OR (+3.75%) when pre-trained with WOMD and fine tune on AV2\n* There is a typo in line 196: two without.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NW70VmzHw5", "reviewer": "Reviewer_ZYgU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes joint self-supervision learning for joint motion prediction. It employs scene-level self-supervised learning to connect motion and environment information from past trajectories of each agent and HD map polylines. The scene-level self-supervised learning is further combined with instance-level self-supervised learning to reconstruct masked trajectories and map polylines to improve prediction accuracy.", "summary_of_recommendation": "The paper has some drawbacks but they can be probably well addressed in the rebuttal. Hence, my rating is a weak accept", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "Overall, the paper shows the potential of exploring self-supervised learning at both the scene level and instance level to combine motion and map information for joint motion prediction. But the weaknesses mentioned above should be addressed.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LmwbnG7xwb", "reviewer": "Reviewer_XhvA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper JointMotion brings another way to employ data-efficient self-supervised learning (SSL) pre-training method to enhance existing autonomous driving joint motion prediction models. Unlike previous works, in this work, SSL is applied in two-fold to: (1) non-contrastive similarity learning for past motion sequences and environmental context at the scene level, creating robust joint scene representations (in section 4 this is also called Connection motion and environment (CME)); and (2) masked autoencoding and adaptive decoding to refining multimodal polyline representations, including motion, lanes, and traffic lights at the instance level. The authors’ experiment show this method outperforms existing SSL methods and is able to generalize across backbone models.", "summary_of_recommendation": "The contribution is incremental. The article needs multiple clarifications.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1. In section 1 contribution c: by saying your method can generalize across datasets, how many different datasets did you experiment? I only see WOMD and Argoverse 2. In addition, saying a model that can work on two datasets is generalizable is not rigor, unless you are talking about zero-shot generalizability, which does not seem the case in your work.\n2. Page 2 line 27: you listed reference 11 (Motion Transformer) as an example of marginal prediction model without interaction modeling. However, in section 3.1 of that paper, the author stated their model perform interaction prediction as part of the scene context encoding. \n3. Page 5 line 181: typo `ration`->`ratio`.\n4. Page 5 line 184, do you intent to include $\\lambda_{red}$ in the total loss defined at line 183?\n5. Page 5 line 196, there is a typo where you wrote 2x `without`.\n6. Page 6 line 205: how do you conclude lane reconstructions are more accurate in this case? Is figure 3 a plot of the training time loss? Indeed the blue curve decreases slower but they do not seem fully converged yet. You can say without scene-level objective, the training loss of lane reconstruction converges slower than that of agent state and traffic light but I don’t think it is evident enough to conclude lane reconstruction is less accurate.\n7. You have experiments ablating the CME from JointMotion. Do you also have experiments ablating masked autoencoding and compare with the scene level SSL method PreTraM?\n8. Page 6 figure 4: I’m a little confused by this figure. If I understand correctly, the orange curve and blue curves are meant to represent the $mAP$ progress under the same training time? First, your x-axis is step rather than time. Second, the orange curve (train from scratch) takes $1.8e5$ steps whereas the blue curve (pre-train + fine-tune) takes $0.8e5$ steps for pre-train plus $1.8e5$ steps for fine-tuning. Only the fine-tuning part should have taken the same time as the entire orange part given the model is the same, right? Are you sure the train-from-scratch configuration is given the same amount of time as the pre-train-then-fine-tune configuration?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ### Strengths:

- Re: Methodology: R-ffHe and R-ZYgU mention innovation in the scene-level self-supervised learning (SSL) that connects motion and environment without requiring negative examples, enhancing scene-wide representation learning; they highlight effective application across different environment representations and information fusion mechanisms, and they highlight that the instance-level SSL reconstructs masked polylines for motion sequences, lanes, and traffic lights, refining detailed representations. R-ffHe mentions that the Adaptive Pre-Training Decoder enables generalization across various datasets and environment representations, showing adaptability in models with early fusion mechanisms.
    
- Re: Experiments: All reviewers cite the manuscript for having extensive experiments that demonstrate that JointMotion outperforms recent SSL methods and state-of-the-art (SOTA) methods in joint motion prediction. They highlight significant improvements over non-ensembling joint prediction and joint-prediction variants of marginal-prediction architectures. They suggest that the method demonstrates generalizability across different backbone models.
    
- Re: Clarity: R-ZYgU mentions that the manuscript is generally well-written, allowing readers to follow through the sections and understand the methodology and results.
- Re: Contribution: R-XhvA highlights the work for combining scene-level similarity learning and masked autoencoding, which they say suggests complementary properties and contributing to the field.
    

### Weaknesses:

- Re: Methodology:  R-ZYgU and R-XhvA express concerns over some methodological details that require familiarity with referenced models, suggesting a need for summarizing these details in supplementary materials. They also mention a lack of clarity in the pre-training procedures and the relationship between contributions and improvements.
- Re: Experiments: R-ZYgU and R-XhvA cite concerns over inconsistent experimental settings in Tables 2 and 3. R-XhvA expressed concerns over missing ablation studies for masked autoencoding from JointMotion, affecting the robustness of the results.
- Re: Clarity: R-ZYgU and R-XhvA cite concerns over the lack of clarity in the Figure 1 and Figure 3 descriptions; they mention that Figure 4’s training time comparison lacks clarity and might mislead readers. R-ffHe and R-XhvA assert that major revisions needed to improve organization and clarity and that specific sections, such as experiment setup and pre-training procedures, need better descriptions.
- Re: Novelty: R-ffHe and R-XhvA mention issues in originality, mentioning that scene-level similarity learning is not considered a major contribution due to extensive prior research and that combining existing techniques is seen as incremental rather than groundbreaking.
    
- Re: Limitations: All reviewers point out that the Limitations section is not well addressed, that the impact is considered incremental with no significant advances in robotics or machine learning, and that the reported improvements fall short in various SOTA metrics.

### Post-rebuttal Meta Review Statement

I applaud the reviewers and authors for engaging in a productive and efficient discussion that resulted in several improvements to the paper and a notable increase in a reviewer score.

While concerns still remain for this work (e.g., incremental contribution and lacking detailed analysis of convergence properties, discussed by R-XhvA), I do feel as if the stated strengths of the manuscript, in sum, sufficiently outweigh the limitations.

I recommend for Accept as Poster.

I encourage the authors to follow through with all the promised changes and to incorporate all additional reviewer requests, to improve the paper even further.
Decision: Accept