SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction
Authors: Yang Zhou
Hao Shao
Letian Wang
Steven L. Waslander
Hongsheng Li
Yu Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Motion Prediction
Trajectory Prediction
Autonomous Driving
Self-Supervised Learning
Primary area: applications to robotics, autonomy, planning

Abstract: Predicting the future motion of surrounding agents is essential for autonomous vehicles (AVs) to operate safely in dynamic, human-robot-mixed environments. However, the scarcity of large-scale driving datasets has hindered the development of robust and generalizable motion prediction models, limiting their ability to capture complex interactions and road geometries. Inspired by recent advances in natural language processing (NLP) and computer vision (CV), self-supervised learning (SSL) has gained significant attention in the motion prediction community for learning rich and transferable scene representations. Nonetheless, existing pre-training methods for motion prediction have largely focused on specific model architectures and single dataset, limiting their scalability and generalizability.
To address these challenges, we propose SmartPretrain, a general and scalable SSL framework for motion prediction that is both model-agnostic and dataset-agnostic. Our approach integrates contrastive and reconstructive SSL, leveraging the strengths of both generative and discriminative paradigms to effectively represent spatiotemporal evolution and interactions without imposing architectural constraints. Additionally, SmartPretrain employs a dataset-agnostic scenario sampling strategy that integrates multiple datasets, enhancing data volume, diversity, and robustness.
Extensive experiments on multiple datasets demonstrate that SmartPretrain consistently improves the performance of state-of-the-art prediction models across datasets, data splits and main metrics. For instance, SmartPretrain significantly reduces the MissRate of Forecast-MAE by 10.6\%. These results highlight SmartPretrain's effectiveness as a unified, scalable solution for motion prediction, breaking free from the limitations of the small-data regime.

Published as a conference paper at ICLR 2025
SMARTPRETRAIN: MODEL-AGNOSTIC AND DATASET- AGNOSTIC REPRESENTATION LEARNING FOR MOTION PREDICTION
Yang Zhou'
Hao Shao Letian Wang"
Steven L. W
Hongsheng L;245
Yu Liu
Time Research
CUHK MMLab
Uniiversity of Toronto
CCPII under InnoHK
Shanghai Artificial Intelligence Laboratory
ABSTRACT
Predicting the future motion of surrounding agents is essential for autonomous vehicles (AVs) operate safely in dynamic, human-robot-mixed environments. However, the scarcity of large-scale driving datasets has hindered the develop- ment of robust and generalizable motion prediction models, limiting their ability to capture complex interactions and road geometries. Inspired by recent advances in natural language processing (NLP) and computer vision (CV), self-supervised learning (SSL) has gained significant attention in the motion prediction commu- nity for learning rich and transferable scene representations. Nonetheless, exist- ing pre-training methods for motion prediction have largely focused on specific model architectures and single dataset, limiting their scalability and generalizabil- ity. To address these challenges, we propose SmartPretrain, general and scalable SSL framework for motion prediction that both model-agnostic and dataset- agnostic. Our approach integrates contrastive and reconstructive SSL, leverag- ing the strengths of both generative and discriminative paradigms to effectively represent spatiotemporal evolution and interactions without imposing architec- tural constraints. Additionally. SmartPretrain employs a dataset-agnostic scenario sampling strategy that integrates multiple datasets, enhancing data volume, diver- sity, and robustness. Extensive experiments on multiple datasets demonstrate that SmartPretrain consistently improves the performance of state-of- of-the-art prediction models across datasets, data splits and main metrics. For instance, SmartPretrain significantly reduces the MissRate of Forecast-MAI by 10.6%. These results highlight SmartPretrain's 's effectiveness prediction, breaking free from the limitations of the small-data regime. The code is available
unified, scalable solution for motion
http:///ghho///9//t//a/ooaeaa 1999/SmartPretrain
INTRODUCTION
Motion prediction, predicting the future states of space-sharing agents nearby (e.g., vehicle, cyclist, pedestrian) is crucial for autonomous driving systems (Hu et al. [023 Shao al. 2023b 2024) to safely and efficiently operate in the dynamic and human-robot-miin mation, including surrounding agents' states and high-definition maps (HD maps), provides critical geometric and semantic information for motion behavior, as agents' behaviors are highly dependent interactions with surrounding agents and the map topology. For example, agents' interactive cues, such as yielding, would influence other agents' Decision-making and vehicles usually move in drivable areas and follow the direction of lanes. Thus, designing (Chai et al. 2219a Gao et al.
environment. Context infor-
2020
Liang et al. 2020 Wang et al.) 2022) and learning (Salzmann et al.) 2020; Nggiam et al,
2021
Varadarajan et al. 2022 Zhao et al. 2021) scene representation that captures rich motion and context information has long been a core challenge for motion prediction.
"Equal contribution.
Project lead.

---

Published as a conference paper at ICLR 2025
However, while natural language processing (NLP) and computer vision (CV) communities have repeatedly demonstrated the power of large-scale datasets, existing motion prediction works have been operating in "small-data regime", due to the expensive and laborious nature of collecting and annotating driving trajectory data. For example, popular motion prediction datasets such as the Argoverse (Chang al., 2019). Argoverse 2 (Wilson al.) (2023) and Waymo Open Motion Dataset (WOMD) (Sun et al. (020) datasets only include 320K, 250K, 480K data sequences, respec- tively, significantly less than the common-pinn -aannoon-prccceeeee data scale in the NLP and CV community (c.g., ChatGPT-3 trained on the 400B-token CommonCrawl dataset (Raffel et al., 2020), Vision Trans- former trained on the 303M-image JFT dataset (Sun et al., 2017)). The scarcity of trajectory data prohibits the models from learning rich and transferable scene representation, and thus restrains their performance and generalizability. To this end, one line of work has sought to mitigate the scarcity of real driving motion data, by leveraging synthetic trajectory data generated via prior knowledge and manually designed rules (Li et al.) [024| Azevedo et al.) (022). However, major drawback of using synthetic data is the "reality gap the distribution of artificially generated data often differs from real-world distributions, leading to gap between synthetic training and real-world per- formance. Additionally, generating high-fidelity simulations requires considerable computational resources and careful design to capture complex agent interactions accurately, limiting scalability.
In addition to data scaling, the NLP and CV communities have developed remarkable advances in self-supervised learning (SSL). As evidenced by the success of models such as BERT (Devlin al.) 2019) and Masked Autoencoders (He et al. 2022), SSL is demonstrated to acquire expressive repre- sentations/features by pre- pre-training on unlabelled data, and thus enhances downstream performance after fine-tuning To this end, SSL has recently received increased attention in the motion prediction community. However, designing truly general and scalable SSL pre- pre-training strategies for motion prediction has been non-trivial. We elaborate on the challenges by categorizing existing works into two primary approaches: generative SSL and discriminative SSL.
In the domain of generative SSL, pretext tasks designed to learn context-rich scene rep- resentations from real data. Various traffic-related pretext tasks have been proposed, such as classification for maneuver or forecasting success (Bhattacharyya al.] (022), map graph re- lationship prediction (Azevedo et al. 2022). and traffic event detection (Pourkeshavarz et al.) 2023). Moreover, extensive efforts have been made to replicate the success of Masked Au- toencoder techniques seen in NI.P and CV, exploring diverse traffic-specific masking strate- gies (Yang al.. 2023 Chen al., 2023 Lan al., 2004]] Cheng et al.. 2023). However, these pre-training strategies usually impose restrictions on the model architecture they start by proposing pre-training strategies, whose realization then necessitates the model generate certain types of features. For example, the detection for maneuvering, graph relationships, and traffic events require explicit formulation of these concepts/features within the model. MAE approaches demand that each trajectory or map segment must have an explicit feature rep- resentation enable reconstructive pre-training. In approaches where only trajectories are masked and reconstructed (Yang et al.] 2023). the method remains simple and flexible but es- sentially acts data augmentation technique. While many works focus on aggregating agent embeddings and provide explicit access to them, explicit map embeddings are not always avail- able (Zhou et al. 2023 Tang et al., 2024). Thus the map reconstruction pre-training strategy in MAE could be less general. Due to such inflexibility, these SSL strategies can usually only apply to specific models or, in most cases, their single specific model.
In discriminative SSL, contrastive learning has emerged as promising technique for motion prediction tasks, where methods aim learn discriminative features by contrasting the trajec- tory and map embeddings among positive and negative examples (Xu et al. 2022). However, it is limited to models based on rasterized map representations, which is shown to have a signifi- cant performance gap compared to more recent transformer -bssed or graph-based models that incorporate vectorized map (Liang et al. 2020 Gao et al. 2020). Applying CL to vectorized representations remains an which complicates the definition of meaningful positive and negative samples and requires so-
area, due to the irregular structure of vector data,
phisticated sampling strategies.
Summing up. due to the inherently complex and multimodal input representation for motion pre- diction, existing SSL pre-training strategies fall short in their applicability to general model archi- tectures and input representations. Moreover, the discrepancies among different popular motion prediction datasets have further prevented these methods from leveraging multiple datasets to break free from the small-data regime. As shown in Fig. [ such limitations detract from the true power
2

---

Published as a conference paper at ICLR 2025
Existing Pre-training Pipelines
Our Pre- Pre-Training Pipeline
Model-Specific Design
Infiexible to Other Models
Model-Agnostic Design
Generally Applicable Different Models
Leverage Muttiple Datasets
Pipeline A
Model A
Model A
Model
Our Pipaline
Pipeline B
Madel
Model
Model
Figure I: Illustration comparing existing trajectory prediction pre-training pipelines with ours. Our pipeline can unlock performance gains from all models as it is model-agnostic, while most existing pipelines are model-specific and inflexible.
of SSL, which lies in its ability to provide unified and scalable strategy for representation learning across diverse model architectures, input representation, and data sources.
In this paper, we propose SmartPretrain, general and scalable SSL representation learning ap- proach for motion prediction. To fully unleash the power of SSL, SmartPretrain is specifically designed to be agnoss-g and dataset-agnostic, providing unified solution that is broadly applicable across different model architectures and diverse data sources, thus escaping from the suboptimal representation learning of the small-data regime. Our key designs are twofold:
Model-Agnostic Contrastive and Reconstructive SSL: SmartPretrain proposes novel model- agnostic SSL framework, which combines the best of the two worlds of generative and dis- crimitive SSL. In a nutshell, we simultaneously reconstruct trajectories and contrast trajectory embeddings from different agents and time windows, learn spatiotemporal evolution and in- teraction happening in the scene. These snteectory-fouusee SSL pretext tasks avoid restrictions on the model architecture and map representation, making SmartPretrain
range of model designs, whether raster- transformer-base or adaptable wide
Dataset-A gnoostic Scenario Sampling: We propose a dataset-agnostic scenario sampling strat- egy that scales effectively by integrating multiple datasets, despite inherent discrepancies. To achieve this, we standardize data representations, ensure high-quality inputs, and maximize data volume and diversity. This allows SmartPretrain leverage broader range of driving scenarios, enhancing generalization and robustness.
Extensive experiments on applying SmartPretrain to multiple state-of-the-a prediction models and multiple datasets show that, SmarPretrain consistently improves all considered models, on down- stream datasets, data splits and main metries. For instance, SmartPretrain significantly reduces the minFDE, minADE, MR of QCNet by 4.9%, 3.3%. 7.6%, and Forecast-MAS by 4.5%. 3.1%, 10.6%. SmartPretrain exhibits superior performance when compared existing pre- pre-training meth- ods. Comprehensive ablation studies are conducted to analyze each component of our pipeline. such as scaling with multiple datasets, and the effect of the two proposed pretext tasks. To the best of our knowledge, SmartPretrain is the very first work that conducts SSI. pre-training leveraging multiple datasets and can be applied range of models, for motion prediction in the driving domain.
RELATED WORK
2.1 TRAJECTORY PREDICTION
Traditional methods for motion prediction primarily use Kalman filtering (Kalman & Others 1960) with physics and maneuver priors from HD-maps to predict future motion states (Houenou al. Xie et al. 2017] Shao et al., 2023a), or caapling-or-optimizztion
planning algorithms with manually specified learned reward functions to generate future trajectories (Wang et al.) 2021 2023a Schwarting et al.) (019 Li et al., 2022). With the rapid development of deep learning, recent works utilize data-driver approaches for motion prediction. Generally, these methods fit into
2013
three different architectures involving rasterized images and CNNs (Cui et al.] [019 Chai et al.)
2019b), vectorized representations and GNNs (Zhou al.] 2022) 2023 Park et al. 2023 Tang
et al., (204) and transformers (Wang et al. 2023b Nayakanti et al. 2023; Shi et al. 2024). )) To
represent scene information, raster-based methods utilize CNNs rasterize scene context into
3

---

Published as a conference paper at ICLR 2025
eyr--eye-view image, while the other two architectures represent each entity of the scene as vector following Gao et al. (020). For the multimodal trajectory outputs, in addition to the standard one- stage prediction pipelines which output trajectories directly, there
now two-stage prediction-
refinement pipelines (Ye et al., 2023] Choi et al., 2023] Zhou et al., 2023 Shi et al. 2024 Zhou
ct al. 2024) where coarse trajectonis are first proposed and then used for trajectory refinement.
2.2 SULERVUPER LEARNING IN TRAJECTORY PREDICTION
Self-supervised learning (SSL) methods have been applied widely in both visual understand- ing (Chen et al. 2020 Grill et al. 2020; Caron et al. 2021 He et al. 2022] Benaim et al. 20200
Bertasius al. 2021 Feichtenhofer et al. 2022] Lin et al. 2023 2024 Tang et al. (025)) natural
language processing (Devlin et al. 2019 Brown 2020: Touvron et al. 2023) and multimodal rep- resentation learning (Qu al.] 2025). SSL aims to learn informative and general representations via carefully designed pretext tasks, which can be fine-tuned for downstream tasks with supervision. Recently, the trajectory prediction community has made significant strides in incorporating SSL techniques. These methods utilize pretext tasks to pre-train models, allowing them to learn valuable representations that be fine-tuned for enhanced trajectory prediction performance. Existing pre- training pipelines for trajectory prediction can be divided into three different categories, involving augmented synthetic data (Yang et al. 2023 Li et al. (222), contrastive learning (Xu al. 2022
Bhattacharyya et al. 2022 |022|Azeveer Azevedo et al.
2022 Pourkeshavrzz et al., (023) and generative masked
representation learning (Chen et al. 2023 Cheng et al. 2023 Lan al. 2024)).
Methods with synthetic data generate trajectory and map data with prior knowledge and manually designed rules. For example, Li et al. (2024) generates driving scenarios with a map augmentation module and a model-based planning model. Contrastive learning methods aim to align and differ- entiate embeddings by comparing them with positive and negative examples, capturing high-level semantic relationships. For example, PreTraM (Xu et al.) 2022). a raster- raster-based approach, uses con- trastive learning to model the relationships between trajectories and maps, as well as between the maps themselyes. On the other hand, generative masked representation methods leverage the trans- former (Vaswani et al. 2023) to learn token relationships by reconstructing randomly masked HD map context and trajectories, inspired by the broad success of Masked Autoencoders (MAE) (He) et al. 2022). For example, Forecast-MAN ments as individual tokens, applies the random mask at the token level, and feeds the masked tokens into the transformer backbone for reconstruction.
(Cheng et al. 2023) treats agents' histories and lane seg-
Existing SSL pipelines have two major limitations: 1) they pre-train on a single trajectory prediction dataset due to the varied formats in different datasets and 2) they rely specific model architectures and map embeddings, limiting their adaptability to general models. Additionally, these pipelines struggle to extend to advanced GNN-based approaches, which integrate all inputs into unified graph, preventing the use of explicit map embeddings for pretext tasks. To overcome these chal- lenges, propose SmartPretrain, model-agnostic and dataset-agnostic solution. It can flexibly apply to various models and datasets, regardless of model architecture or dataset format.
3 METHODOLOGY
3.1 PROBLEM FORMULATION MOTION PREDICTION MEETS SELF-SUPERVISES LEARNING
Typical motion prediction task can be defined as follows: taking the observed states of the target agent S, [sTT++ 8-T, .2. Su] IR Th over the last Tí, steps, we aim to predict its future states 8 = [81, 82, .,, ST,] 8T, E RTyx2 for Tr future steps as well as the associated probabilities p. Naturally, the target agent will interact with context c, including observed states of surrounding agents, and the HD map. The typical motion prediction task is formulated as (sy, p) f( c), where of denotes the prediction model. Generally, motion prediction models are structured in an encoder-decoder architecture with two stages:
Encoding (Sp, c): encoder Sene embeds and fuses sa and to capture the evolution and interaction in the traffic scene, and generate trajectory embedding z;
Decoding (sy, p) = (z): a decoder ldee decipher z to generates multiple possible future trajectories 8y and corresponding probabilities p.
4

---

Published as a conference paper at ICLR 2025
Online Branch
Trajectory Decoder
AV
Model
Projector
Temporal
AV2
Sampling
EMA
Contrastive Predictor
Sampling
Sub-scenario
Updato
W
Momentum Branch
Reconstruction Loss
Repel
Model
Projector
Trajectoriss of Sub-scenario e'
Repel
Dataset
Scenario
Sub-scenario.t
Contrastive Loss
Figure 2: Overview of our model-agnostic and Cataset-agnosti pre-training pipeline. We begin by randomly sampling training scenario from mixed datasets. From this scenario, two sub-scenarios with different temporal timelines are randomly sampled and fed into two model branches to generate trajectory embedding in the scene. Two model-agnostic pretext tasks, trajectory contrastive learning and reconstructive learning, are introduced to learn transferable and robust representations.
SSL takes a pivot from the typical motion prediction learning by introducing self-supervised pre- training phase. The objective to pre-train the encoder Sene with pretext tasks, learn more transferable and generalized embeddings z. Afterward, the encoder fene and decoder Satee are fine- tuned together on the actual motion prediction task. the literature, the challenges and focus have been on designing pretext tasks that scale well and effectively enhance downstream performance.
3.2 SMARTPRETRAN SSL FRAMEWORK
We propose SmartPretrain, a model-agnostic and dataset-agnostic pre-training framework for mo- tion prediction, that can be flexibly applied to range of models regardless of their architectures, and leverage various datasets despite their differences in data formats. As illustrated in Fig. 2 Smart- Pretrain is composed of two parts: 1) dataset-agnostic scenario sampling strategy for constructing representative and diverse pre-training data, and 2) a model-agnostic SSL strategy, consisting two trajectory-focutt focused pretext tasks, trajectory contrastive learning (TCL) and trajectory reconstruction learning (TRL), which together shape the pre- teeeitrining to improve performance on downstream tasks.
3.2.1 AGNOSTIC SAMPLING
In the dataset-agnostic sampling process, aim to create positive and negative pairs for contrastive learning and construct trajectories for reconstructive learning, while ensuring data consistency across various datasets.
Data Sampling for Contrastiv/Reconstrutiiee SSL. Starting with formulating positive and neg- ative samples for contrastive learning, intuitive design would be contrasting different agents' trajectories within the same scenario. However, this approach may lead to suboptimal performance learning spatial context among agents without sufficient temporal
due 1) overemphasis modeling and 2) limited variability in positive samples, as they contain the same features. To this end, we propose temporal sampling strategy create sample pairs that capture both the spatial context and the temporal evolution of the agents. Specifically, we mix multiple datasets to form comprehensive data bank, and randomly sample scenario with time horizon 7 Th Ty. Next, we temporally sample two sub-scenarios, which have the same temporal horizon Tū, but start at different time f. and L' respectively. To prevent information leakage during sub-scenario sampling, single scenario do not overlap, which could compromise
we ensure that the two sub-scenarios the training of the pretext tasks. The sub-scenarios then later used to construct positive trajectory pair from the same agent across different time, and negative pairs from different agents or different time. Note that, beyond contrastive learning, the sub-scenario starting at will also be used for reconstructive learning, as its temporal horizon is designed to align well with the input horizon Th, of the target downstream dataset.
Maintaining Aataset-Agnosticissn To leverage multiple datasets with varied configurations and achieve data scaling, introduce three key designs:
5

---

Published as a conference paper at ICLR 2025
Standardizing Representations: Due varying formats across datasets, HD map's sap's resolution and agents' trajectory horizon often differ significantly, causing inconsistency. To address this, standardize the map contexts and trajectories representation into unified format. Specif- ically, we fix the resolution of HD maps using linear interpolation or downsampling to ensure consistency between consecutive points. Additionally, to handle differences in scenario hori- zons, we apply zero- poodppdding to scenarios with shorter horizons to align their length with the desired time horizon T', The padded trajectory steps are masked as invalid during pre-training to ensure they do not affect the learning process.
Ensuring Data Quality: Different datasets also vary in data quality and scenario complexity, often containing incomplete trajectories due to perception limitations or agents' entering/leaving the scene. To improve consistency and ensure higher data quality, we include only complete trajectories during pre-training, filtering out incomplete ones from the training pipeline.
Maximizing Data Volume and Diversity: The number of scenarios varies significantly among datasets. We found that directly mixing all available training data, rather than balancing the number of trajectories from each dataset, provides the best downstream results.
3.2.2 MODEL- CONTRASTIVE AND RECONSTRUCTIV SSL
We propose model-agnostic SSL strategy that consists of two pretext tasks: 1) trajectory con- trastive learning task (TCL), that enriches learned trajectory embeddings by contrasting them across agents and time windows; 2) a trajectory reconstruction learning task (TRL), that aligns more closely with the primary goal of motion prediction, better shaping the direction of pre- vee-training. Note that both pretext tasks are designed to ensure model-agnostice via their trajectory- -ocuus:: they only contrast and reconstruct agents' trajectory embeddings, rather than any other embeddings such as map customized embeddings. Thus they remove restrictions on the model architectures and map representations, and can be applied to much wider variety of motion prediction models.
Embedding Generation. With 2 sampled sub-scenarios, we then generate embeddings for all the trajectories they contain. Specifically, we follow a self-training strategy from existing SSL litera- ture (He et al. [020 Caron et al.] 2021 I to bootstrap performance and avoid model collapse. Specif- ically, the two sub-scenarios are fed into two identical architecture branches, an online branch and momentum branch. Within each branch, the input sub-scenario is first passed through the motion prediction model to generate all trajectories' embeddings, which are then fed to a projector for fur- ther encoding modification. During pre- trainingg, the online branch is continuously updated, while the momentum branch is occasionally updated using an exponential moving average mechanism. The embeddings from the online branch are also additionally passed through contrastive predictor to generate the final embeddings for contrastive learning.
Trajectory Contrastive Learning (TCL). TCL is designed to learn rich trajectory representations by contrasting trajectory embeddings across spatiotemporal dimensions. Specifically, considering that the sampled scenarios in mini-batch consist of N agents, we now have two sets of embeddings {z,,e 1 and (z),,' IN 1 generated from the online branch and momentum branch respectively. We define the contrastive loss to pull closer positive samples, and repel away negative samples:
N
exp (r(zi,t, 21,1))/7)
= N
log
(1)
1 Eر=1y exp (r(z.,,, 25,1/T) + E exp (r(zi,t, (a,c,,,,,,,)) )//))
where denotes the cosine similarity measure, and denotes the temperature hyper-parameter. Specifically, the numerator term represents the similarity of positive sample pairs (Z,,,, 2/,r)) namely trajectory embeddings from the same agent in different timelines. The denominator term consists of two types of negative pairs: 1) intra-repelling pairs (Z,,e. z,.:): the trajectory embeddings from other agents of the same sub-scenario; 2) inter-repelling pairs (z,,e, z trajectory embeddings from different sub-scenarios. Through this objective, the similarities of the same agent's embeddings are maximized, and those of the other pairs are minimized, thereby refining the model's ability to capture meaningful contextual relationships and temporal dynamics.
Trajectory Reconstruction Learning (TRL). While contrastive learning is a discriminative task that helps features distinguish motion and contextual differences among trajectories, the ultimate goal of motion prediction is a regression task. Therefore, the features learned only through con- trastive learning may not necessarily align with or benefit the needs of motion prediction. To this

---

Published as a conference paper at ICLR 2025
end, propose trajectory reconstruction learning, which is designed to bring the learned represen- tation as close as possible to the needs of motion prediction, by sharing a similar training objective.
Specifically, recall that in Sec. 3.2.1 we sampled scenario with horizon length T, and generated the trajectory embeddings {z,,) for sub-scenario with horizon length Th and start time We then pass { J.V to a trajectory decoder, to reconstruct the trajectory segments from the sub- scenario with starting time L'. The reconstruction loss L, is then simply designed as the average L1 distance between the reconstructed trajectories and the ground truth. Interestingly, the proposed reconstruction loss can be viewed jectory prediction task. When equals O, the input segment exactly the observed states S,, of the downstream trajectory prediction task. The task regresses the trajectory with Tn, steps started from of the future states 8,, based on the observed states 8).-
generalized form of regression loss for the downstream tra-
3.2.3 TRAINING DETAILS
Pre-training Stage. The overall pre-training scheme integrates trajectory contrastive learning and reconstruction. The combined loss function is formulated as follows:
C - Le Xhr:
(2)
where is hyper-parameter balancing the contribution of both tasks and we set À as 1.0.
Finetuning Stage. After pre-training on a specific model, we initialize the model's encoder fenc with pre-trained weights, and fine-tune the whole model the downstream trajectory prediction task, using the model's s original prediction objective and training schedules.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We train and evaluate our method on three large-scale motion forecasting datasets: Ar- goverse (Chang et al. 2019), Argoverse 2 (Wilson al., 2023) and Waymo Open Motion Dataset (WOMD) (Sun et al. 2020). Argoverse contains 333k scenarios collected from interactive and dense traffic. Each scenario provides the HD map and 2 seconds of historic trajectory data, predict the trajectory for the next seconds, sampled at 10Hz. The training, validation, and test set of Argoverse set to 205k, 39k and 78k scenarios, respectively. Argoverse 2 extends the historic and prediction horizon to 5 seconds and 6 seconds respectively, sampled at 10Hz. The data is split into 200k, 25k, and 25k for training, validation, and test, respectively. For WOMD, the dataset provides second of historic trajectory data and aims to predict the trajectory for 8 seconds into the future, sampled 10Hz well. It contains 487k training scenes, 44k validation scenes and 44k testing scenes. Notably, our pre-training pipeline only utilizes the training splits of these datasets.
Baselines. As mentioned previously, our pre-training pipeline be seamlessly integrated into most existing trajectory prediction methods. In our experiments, we consider four popular and advanced methods the prediction backbone evaluate how our SmartPretrain further improves performance: HiVT (Zhou et al. (2022), HPNet (Tang et al. 2024). Forecast-MAE (Cheng et al.) 2023) and QCNet (Zhou et al.) 2023). We use their official open-sourced code for implementation.
Metrics. Following the official dataset settings (Chang et al. 2019), we evaluate our model using the standard metries for motion prediction, including minimum Average Displacement Error (minADE), minimum Final Displacement Error (minFDE), and Miss Rate (MR). While the prediction model forecasts up to 6 trajectories for each agent, these metries evaluate the trajectory with the minimum endpoint error, as signal of best possible performance from the multi-modal predictions.
Implementation Details. We implement the projector and contrastive predictor as a 2-layer MLP with batch normalization, and the trajectory decoder also as a 2-layer MLP but with layer normal- ization to better fit its sequence nature. We use AdamW W to optimize the online branch. In the momentum branch, the weights of the motion prediction encoder and the projector are initialized to be identical to that of the online branch, and updated via exponential moving average (EMA) strategy. We use a momentum value of 0.996 and increase this value to 1.0 with a cosine schedule. We conduct single dataset pre-training with 8 Nvidia A A000 40GB GPUs and data-scaled pertaining with 32 GPUs, both for 128 epochs. For fine-tuning, we use 8 GPUs with the models' original training schedules.
7

---

Published as a conference paper at ICLR 2025
Downetream
Datases
Method
Pre-train Dataset
minFDE:
Validation Set
minADE
MR L
minFDF L
minADE
Test
MR
HiVT
0,969
0.661
0.092
1.169
0.774
0.127
HIVT w/
Argo
0.940
0.647 (:2.19)
(-4.3%)
(-2.0%)
0.963 (-1449)
0.122 (-3.9%)
Argovense
HIVT w/ Ours
All
0,929 (-4.1%)
0.644 (-2.6%)
0.086 (-6.58)
1.135 (-2.9%)
(-1.9)
0.119 (-6.3%)
HPNet
0.871
0.688
0.009
1,009
0.761
0.107
Argo
0.854 (-2.0%)
(:1.1))
(-5.8%)
0.758 (-0.45)
(-3.%%
QCNet
1.253
0.720
0.157
1.241
0.636
0.154
NCN w/ Ours
Argo 2 All
1:212 (33.3%%)
0.202 (-2.59%
0.148 (-5.89))
1.222 (-1.5%)
0.625 (c1.796)
(:5.295)
Angovense
QUNer
999)
0.606
(-3.1%)
0.618
0.142
Forocust-MAF
1,436
0.811
0.189
1.427
0.727
0.187
Forocast-MAN w/ Ours
Argo 2
1.372 (-1.5%)
0.786 (-3.1%)
0.069 (- 10.6%)
1.390 (-2.6%)
0.713 (-1,9%)
0.171 (-8.5%)
We evaluate the performance of SmartPretrain with multiple state-of-the- prediction models, reporting results on the validation and test sets of the Argoverse and Argoverse datasets. SmarttPrerriin consistently enhances all models across datasets, data splits and main metrics. Addi- tionally, pre-training with more datasets results in further performance improvements.
Table 1:
Dataset
Backhone
Pre-training Method
minFDE
minADE 1
MR L
Without Pre-training
1.466
0.811
0.189
Angoverse
Forecast
Forocast-MAE
1,409 (-1.9%
0.801
(-1.2%)
0.178 (-6.2%)
Ours
1.372 (-4.5%)
0.786 (-3.19)
0.169 (-10.6%)
Table 2: Performance comparison with existing pre-training method Forecast-MAE.
4.2 QUANTITATIVE RESULTS
Performance of Applying SmartPretrain to Multiple Models. As shown in Table we first report the performance when we apply SmartPretrain to multiple state-of-the-art prediction models, on the validation and test set of Argoverse and Argoverse 2. We consider two pre-training settings: pre- training only on the single downstream dataset, and pre-training on all three datasets. Specifically, we pre-train HiVT and QCNet with both two settings, and only pre-train HPNet and Forecast- with the single downstream dataset due to compute constraints. SmartPretrain can consistently improve all considered models, on downstream datasets, data splits and main metrics. For instance, SmartPretrain can significantly reduce the minFDE, minADE, MR of QCNet on validation set by 4.9%, 3.3%. 7.6% respectively. Besides, Pre-training with all datasets also shows consistently higher improvement compared to pre-training with only one dataset.
Performance Comparison with other Pre-training Method. We also compare SmarPretrain with other pre-training methods. For fair and meaningful comparison, we look for methods that have open-source code. However, to the best of our knowledge, only Forecast-MAF: was open-sourced the time of this paper's 's submission. Specifically, Forecast-MAE proposes motion prediction backbone and pre-training strategy. We then apply SmartPretrain to Forecast-MAE's backbone, and compare with Forecast-MAE's pre-training strategy, on the Argoverse dataset. As in Table our pre-training method shows a larger improvement than Forecast-MAE's pre-training method by a substantial margin (e.g. improvement of 4.5% vs 1.9% on minFDE).
These results demonstrate that our pre-training pipeline: 1) can be flexibly applied to wide range of motion prediction models; 2) consistently improves performance through pre-training and data scal- ing; and 3) delivers stronger performance enhancements compared to existing pre-training methods.
4.3 ABLATION STUDIES
We conduct comprehensive ablation studies to analyze the impact of different components, including the data scale of pre-training, the two proposed pretext tasks, and associated hyperparameters and configurations. For efficient evaluation, we use HiVT as the prediction model and Argoverse for both pre-training and fine-tuning, reporting performance the Argoverse validation set.
Ablations on Pre-training Datasets. SmartPretrain is designed to be agtaset-agnostic and thus can leverage multiple datasets for pre-training. Here introduce four pre-training settings: 1) No Pre- Training: the model is trained from scratch on the downstream task without pre- pre-training, 2) Baseline Pre-Training: the model is pre-trained on single dataset, which is the same as the downstream dataset; 3) Transfer Pre-Training: the model is pre-trained on a single dataset different from the downstream dataset; 4) Data-Scaled Pre Pre-Training: the model is pre-trained on multiple
8

---

Published as a conference paper at ICLR 2025
HiVT on
Pre-training Dataset
Validation Set
Argoverse
Argoverse
Argoverse 2
WOMD
minFDE 4
minADE
MR 4
No Pre-training
0.969
0.661
0.092
Baseline Pre-training
x
0.940 (-3.0%)
0.647 (-2.1%%
0.088 (-4.3%)
x
0.951 (-1.9%)
0.653 (-1.2%)
0.090 (-2.2%)
Transfer Pre-training
x
0.946 (-2.4%
0.652 (-1.4%)
0.089 (-3.3%)
0.935 (-3.5%)
0.645 (-2.4%)
0.087 (-5.4%)
Data-Scaled Pre-training
0.935 (-3.5%)
0.645 (-2.4%)
0.086 (-5.4%)
0.929 (-4.1%)
0.644 (-2.6%
0.086 (-6.5%)
QCNet on Argoverse
Pre-training Dataset
Validation Set
Argoverse
Argoverse
WOMD
minFDE
minADE
MR 4
No Pre-truining
1,253
0.720
0.157
Baseline Pre-training
x
x
1.212 (-3.3%)
0,702 (-2.5%)
0.148 (-5.7%)
V
x
x
1,226 (-2.2%)
0.712 (-1.1%)
0,151 (-3.8%)
Transfer Pre-training
x
x
/
1,225 (-2.2%)
0,708 (-1.7%)
0.149 (-5.1%)
x
1,201 (-4.2%)
0.702 (-2.5%)
0.143 (-8.9%)
Data-Scaled Pre-training
1.199 (-4.3%%
0.698 (-311%)
0.145 (-7.6%)
1.191 (-4.9%%
(-3.3%)
0.145 (-7.6%)
Table 3: Ablations on Pre-training Datasets: All pre-training strategies effectively improve predic- tion accuracy compared to training from scratch. However, transfer pre-training shows the smallest improvement, likely due to data distribution mismatch, while data-scaled pre-training, leveraging multiple datasets, provides the most substantial performance enhancement by enabling the model to learn more generalized and robust features.
Pretext Tasks
Validation Set
Dataset
Backbone
Contrastive
Reconstructive
minFDE
minADE
MR
X
x
0,969
0.661
0.092
x
0,958 (-1.1%)
0.656 (-0.8%)
0.090 (-2.2%)
Argoversc
HiVT
X
0,952 (-1.8%)
0.652 (-1.4%)
0.090 (-2.2%)
0.940 (-3.0%)
0.647 (-2.1%) 1%)
0.088 (-4.3%)
Table 4: Ablations on the two pretext tasks: each task can effectively improve prediction accuracy in isolation, while combining both tasks yields the largest improvement.
datasets. We then analyze these pre-training settings by applying them to HiVT and QCNet, and fine-tunc/evaluate them on their original downstream dataset.
The results, presented in Table 3 indicate that: (1) all pre-training strategies effectively improve prediction accuracy compared training from scratch, demonstrating the effectiveness of utilizing pre-training to learn transferable features; (2) transfer pre-training yields the smallest performance improvement, likely due the distribution mismatch between the pre-training and downstream datasets, resulting in relatively less relevant learned representations: and (3) data-scaled pre-training, which leverages multiple datasets, provides the most substantial performance enhancement, as the increased diversity in training data helps the model learn more generalized and robust features that benefit the downstream task.
To further demonstrate the effectiveness of utilizing pre-training datasets of our SmartPretrain, we the additional datasets in the pre-training phase. We explored also conduct ablation how two pre-training settings: 1) We pre-train with the standard motion prediction task on additional datasets and then finetuned on the downstream target dataset. 2) directly train the downstream target dataset and additional datasets with the standard motion prediction task. The results and discussion provided in Appendix A.T due to limited space.
Ablations on Pretext Tasks. Recall that our SSL pre-training framework consists of two pretext tasks: trajectory contrastive learning (TCL) and trajectory reconstruction learning (TRL). Table 4 presents an ablation study on them, where we apply them to HiVT on the Argoverse dataset. We observe that each pretext task, when applied in isolation, improves serformannce-reducinn minFDE by 1.1% for TCL. and 1.8% for TRL. However, combining both tasks yields the largest improvement, reducing minFDE by 3%.

---

Published as a conference paper at ICLR 2025
0.98
0.98
0.969
minFDE
minFD
0.959
0.96
0.950
0.945
0.96
0.952
0.946
0.941
0.940
0.940
0.94
0.94
32
64
96
128
256
512
1024
2048
Pre-training Epochs
Pre-training Batch sizes
Figure 3: Ablation study on pre-training epochs and batch sizes. Larger pre-training epochs and
batch sizes enhance performance, while diminishing returns are observed beyond certain levels.
Backbone
Calegory
Reconstructio Target
minFDE
minADE
MR J
No reconstruction
None
0.958
0.656
0.090
Reconstraction with historical information
Trajectory of the ingot sulh-scenario
0.956 (-0.258)
0.655 (-0.2%)
0,000 0.000 (-0.0%)
HIVT
Trajectory
scenario
(-1.0%
(-0.6%)
(-1.1%)
Reconstraction with predictive information
Complementary tnjjectory of the inpur sub-scenurio
0.940 (-1 (-1.9%)
0.648 (-1.2%)
0.087 (-3.3%)
Trajectory of other sub-scenario
0.940 (-1.9%)
0.647 (-1,4%)
0.088 (-2.2%)
Table 5: Ablation study different reconstruction targets. Pre-training with reconstruction tasks boosts prediction performance, with predictive targets showing greater improvement.
Ablations Pre-training Epochs and Batch Sizes. Fig. shows the ablation studies HiVT and the Argoverse dataset. We observe that larger pre-traming epochs and batch sizes improve performance. Larger epochs allow the model to undergo extended training, which helps it learn more nuanced and complex feature representations, ultimately Icading to better performance. Mcanwhile, larger batch sizes are particularly beneficial for contrastive learning, they provide greater number of negative samples within each training batch. However, a diminishing return is observed as we further increase the epochs and batch size, indicating gap between pretext tasks and the fine- tuning task, and suggesting that beyond a certain point, additional pre-training does not significantly enhance model performance and should be balanced with efficiency avoid unnecessary overhead.
Reconstretion Strategies. Our pre-training pipeline reconstructs the trajectories of the other sub- scenario t' given the input sub-scenario t. For the ablation study, explore more reconstruction targets, categorized into two groups. The first group incorporates predicting historical information, including 1) the trajectory of the input sub-scenario t and 2) the entire scenario. The second group focuses on predictive reconstruction, excluding any historical data, and includes 3) the complemen- tary trajectory steps of the input sub-scenario É and 4) the trajectory of the other sub-scenario t'. As shown in Table 5 pre-training with reconstruction tasks enhances prediction performance, with predictive targets yielding the most significant improvements.
Visulization. We present some visualization results fine-tuning Appendix A.2 categorized into four distinct scenarios: 1) trajectory alignment, 2) long trajectory prediction, 3) novel behavior gen- eration, and 4) smooth and safe trajectory synthesis. These results demonstrate the model's multi- modal trajectory predictions across various scenarios and highlight the enhanced generalization and robustness of the proposed method. We also show some reconstructed trajectories of pre-training in Appendix A.3 to illustrate the effectiveness of our pre-text task learning.
5 CONCLUSION
In this paper, we introduce SmartPretrain, a novel, model-agnostic, and dataset-agnostic self- supervised learning (SSL. framework designed to enhance motion prediction in autonomous driving. Through a combination of contrastive and reconstructive SSL techniques, SmartPretrain consistently improves the performance of state-of-the-arr models across multiple datasets. Our extensive exper- iments demonstrate significant improvements when applying SmartPretrain to various prediction models like HiVT, HPNet, Fore and QCNet. Additionally. the flexibility of our frame- work allows effective pre-training across multiple datasets, leveraging data diversity to improve accuracy and generalization. SmartPretrain outperforms existing methods, confirming the effective- ness of our approach. These results highlight SmartPretrain'ss 's scalability, versatility, and potential to advance motion prediction in driving environments.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Caio Azevedo, Thomas Gilles, Stefano Sabatini, and Dzmitry Tsishkou. Exploiting map information for self-supervised learning in motion forecasting, 2022. URL httLhttts///eexii.orgg abs/
0210.04617
Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T Freeman, Michael Rubinstein, Michal Trani, and Tali Dekel. Speednet: Learning the speediness in videos. In Proceedings of the JEEF/CVF conference on computer vision and pattern recognition, pp. 1922-9931, 2020.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Marina Meila and Tong Zhang (eds.), ICML, volume 139 of Proceedings of Machine Learning Research, pp. 813-824. PMLR, 2021. URL httphhtpp//bblll uni triirrrrerr.
de/ob/ htt//oof//// cmml / cm 2021 htm #Rertas USWT21
Prarthana Bhattacharyya, Chengjie Huang, and Krzysztof Czarnecki. Ssl-lanes: Self-supervised learning for motion forecasting in autonomous driving. In Karen Liu, Dana Kulic, and Jeffrey Ichnowski (eds.), CoRL, volume 205 of Proceedings of Machine Learning Research, pp. 1793- 1805. PMLR, 2022. URL URthttp::/ http///blp.uuu uni mami-triess teretrrrrroorr. de/ html #BhattacharyaaH222
dob /e////////e
/ocoorl/corrooooooooooooooooooeo cor 1022.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Yuning Chai, Benjamin Sapp. Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple prob- abilistic anchor trajectory hypotheses for behavior prediction. arXiv preprint arXiv: 19.009444 2019a.
Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple prob- abilistic anchor trajectory hypotheses for behavior prediction. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), CoRL, volume 100 of Proceedings of Machine Learning Re- search, pp. 86-99. PMLR, 2019 9b. URL http:://tb/lll.. papllpuun. commi-trier.ce de /d/ conf corl/ corl 2019 html #Cha SBA19
Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hart- nett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 8748-8757, 2019.
Hao Chen, Jiaze Wang, Kun Shao, Furui Liu, Jianye Hao, Chenyong Guan, Guangyong Chen, and Pheng-Ama Heng. Traj-mae: Masked autoencoders for trajectory prediction. In ICCV, pp. 8317- 8328. IEEE, 2023. ISBN 199-8-3503-0718-44 URL htttp://dble / wwwp:/ec//cccvc22eeo html #CherWSLHGCH2S
de/db/
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Jie Cheng, Xiaodong Mei, and Ming Liu. Forecast-mae: Seff-supervised pre-training for motion forecasting with masked autoencoders. In ICCV, pp. 8-3503-0718-4. URL httt:://bbpp: http://wlppp..ii.eeee www.uni-trier.. de/ db atb/conf/ccc/iccc200// html #.mn@@Chenn0223
IEEE, 2023. ISBN 979-
Sehwan Choi, Jungho Kim, Junyong Yun, and Jun Won Choi. R-pred: Two-stage motion prediction via tube-query attention-based trajectory refinement. In Proceedings of the IEEF/CVF Interna- tional Conference on Computer Vision, pp. 8525-8535. 2023.
Henggang Cui, Vladan Radosavljevic, Fang-Chin Cang-Chich Chou, Tsung- Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djurie. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 international conference on robolics and
automation (icra), pp. 2090-2096 IEEE, 2019.

---

Published as a conference paper at ICLR 2025
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.orr 0tt5/8810.04888
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-3555 -35988, 2022.
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 11525-
11533, 2020.
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2020. URL
05722
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Adam Houenou, Philippe Bonnifait, Véronique Cherfaoui, and Wen Yao. Vehicle trajectory pre- diction based on motion model and maneuver recognition. In IROS, pp. 2013. ISBN 978-1 178-1-4673-6358-7 URL iros2013. mtml #HouenouBCY13
IEEE, de/t:/ee/dbb///// http:/ddcooofrrrr/
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 77853-17866 2023.
Rudolph Emil Kalman and Others. A new approach to linear filtering and prediction problems. Journal of basic Engineering. 82(1):35 45, 1960.
Zhiqian Lan, Yuxuan Jiang, Yao Mu, Chen Chen, and Shengbo Eben Li. Sept: Towards efficient scene representation learning for motion prediction. In ICLR. OpenReview.net, 2024. URL http:/ / dblp. uni trierrrerrer http///c//oooo////ccc///////////
html #La00ML22
Chenran Li, Tu Trinh, Letian Wang, Changliu Liu, Masayoshi Tomizuka, and Wei Zhan. Effi- cient game-theoretic planning with prediction heuristic for socially-compliant autonomous driv- ing. IEEE Robotics and Automation Letters, 1(4):0024--0025. 2022.
Yiheng Li, Seth Z. Zhao, Chenfeng Xu, Chen Tang, Chenran Li, Mingyu Ding, Masayoshi Tomizuka, and Wei Zhan. Pre-training on synthetic driving data for trajectory prediction. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learn ing lane graph representations for motion forecasting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 541-556. Springer, 2020.
Yexiong Lin, Yu Yao, Xiaolong Shi, Mingming Gong, Xu Shen, Dong Xu, and Tongliang Liu. CS-isolate: Extracting hard confident examples by content and style isolation. In Thirly-seventh Conference on Neural Information Processing Systems, 2023. URLphttss:///pptt //pp//vveeee...oeo net dik..:kk=kkkkkkkm jsDFv
Yexiong Lin, Yu Yao, and Tongliang Liu. Learning the latent causal structure for modeling label noise. In The Tiihth-eighth Annual Conference on Neural Information Processing Systems, 2024. htt.:httts://opemm wwwp:neeet.oofteeeeom.mmm EBvq
12

---

Published as a conference paper at ICLR 2025
Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple efficient attention networks. In 2023 IEEE International Conference Robotics and Automation (ICRA). pp. 2980-2987. IEEE, 2023.
Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelof', Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified architecture for predicting multiple agent trajectories. arXiv preprint arXiv:2106.08417, 2021.
Dachee Park, Hobin Ryu, Yunseo Yang. Jegyeong Cho, Jiwon Kim, and Kuk-Jin Yoon. Lever- aging future relationship reasoning for vehicle trajectory prediction. In ICLR. OpenRe- view.net, 2023. URL http://cblp.. uni uni trier.rree de/ http///ooo/ccccccl2/000000
html #ParkYYKYY3
Mozhgan Pourkeshavarz, Changhe Chen, and Amir Rasouli. Learn tarot with mentor: A meta- learned self-supervised approach for trajectory prediction. In ICCV, pp. 8350-8359, IEEE, 2023. ISBN 979-8-3503-0718-4. URLhttp::/dbbl uni www.tr.rreeee//ooo.e.e... hor.eaaaaattttrrrssaaaaaaaaa
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, Jia Yuan Gu, Bin Zhao, Dong Wang, and Xuelong Li. Spatialvla: Exploring spatial representations for abs/ 22511.1531010
visual-language-action model, 2025. URL htttp::/arxiv.orrt
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 21(140):1-6) 2020.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron+4: Dynamically-feasible trajectory forecasting with heterogeneous data. In Computer Visio--ECC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pp. 683-700. Springer, 2020.
Wilko Schwarting, Alyssa Pierson, Javier Alonso-Mora Sertac Karaman, and Daniela Rus. Social behavior for autonomous vehicles. Proceedings of the National Academy of Sciences, 116(50): 24972-24978, 2019.
Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. In Conference on Robot Learning, pp. 726- 737. PMLR, 2023a.
Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li, and Yu Liu. Reason- net: End-to-end driving with temporal and global reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13723-13733, 2023b.
Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed- loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15120-15130, 2024.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr+:: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVE conference on com- puter vision and pattern recognition, pp.
2454, 2020.
Xiaolong Tang, Meina Kan, Shiguang Shan, Zhilong Ji, Jinfeng Bai, and Xilin Chen. Hpnet: Dy- namic trajectory forecasting with historical prediction attention. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15261-15270, 2024.
13

---

Published as a conference paper at ICLR 2025
Yiwen Tang, Zoey Guo, Zhuhao Wang, Ray Zhang, Qizhi Chen, Junli Liu, Delin Qu, Zhigang Wang, Dong Wang. Xuelong Li, and Bin Zhao. Exploring the potential of incoder-free architectures in 3d Imms, 2025. URL https: httptttss//axxi../or///// www.wiwwwwwww....aaa..oaaaap 220.22522200 25622096200
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 130771971, 2023.
Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Mul- tipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In 2022 International Conference on Robotics and Automation (ICRA), pp. 7814-7821. IEEE, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. httpp:::////
erg/abs/1700 03762
Letian Wang, Liting Sun, Masayoshi Tomizuka, and Wei Zhan. Socially-copaatble behavior design of autonomous vehicles with verification on real human data. IEEE Robolics and Automation Letters, 6(2):3411-3428, 2021.
Letian Wang, Yeping Hu, Liting Sun, Wei Zhan, Masayoshi Tomizuka, and Changliu Liu. Transfer- able and adaptable driving behavior prediction. arXiv preprint arXiv:2202.055444 2022.
Letian Wang, Jie Liu, Hao Shao, Wenshuo Wang, Ruobing Chen, Yu Liu, and Steven I. Waslander. Efficient reinforcement learning for autonomous driving with parameterized skills and priors. 2023a.
arXiv preprint arX5v:2305.044:
Xishun Wang. Tong Su, Fang Da, and Xiaodong Yang. Prophnet: forecasting with inhoor proposals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Efficient agent-centric motion
2023b.
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khan- delwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Ar- goverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint 1rXiv:2301.00493 2023.
Guotao Xie, Hongbo Gao, Lijun Qian, Bin Huang, Keqiang Li, and Jianqiang Wang. Vehicle trajec- tory prediction by integrating physics- and maneuver- based approaches using interactive multiple models. IEEE Transactions on Industrial Electronics, 65(7):5999 6008, 2017.
Chenfeng Xu, Tian Li, Chen Tang, Lingfeng Sun, Kurt Keutzer, Masayoshi Tomizuka, Alireza Fathi, and Wei Zhan. Pretram: Self-supervised pre-training via connecting trajectory and map. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), ECCV (39), volume 13699 of Lecture Notes in Computer Science, pp. 34-50. Springer. 2022. ISBN eccv2022- 39. html XULLSKKFF22 222
199822--7
URL httt::///blie uni-trier. de/db//eeeeee
Yi Yang, Qingwen Zhang, Thomas Gilles, Nazre Batool, and John Folkesson. Rmp: A random mask pretrain framework for motion prediction. In ITSC, pp. 3717-3723, IEEE, 2023. ISBN -78-1-6654-6888000 itsc2023. html #YangZGBF23
Maosheng Ye, Jiamiao Xu, Xunnong Xu, Tengfei Wang, Tongyi Cao, and Qifeng Chen. Bootstrap motion forecasting with self-consistent constraints. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 8504-8514, 2023.
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp. Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. In Con- ference Robot Learning, pp. 895-904. PMLR, 2021.
Yang Zhou, Hao Shao, Letian Wang, Steven L Waslander, Hongsheng Li, and Yu Liu. Smartrefine: A scenario-adaptive refinement framework for efficient motion prediction. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 15281-15290 2024.
14

---

Published as a conference paper at ICLR 2025
Zikang Zhou, Luyao Ye, Jianping Wang. Kui Wu, and Kejie Lu. Hivt: Hierarchical vector trans- former for multi-agen agent motion prediction. In Proceedings of the IEEE/CVE Conference on Com- puter Vision and Pattern Recognition, pp.
2022.
Zikang Zhou, Jianping Wang. Yung-Hui Li, and Yu-Kai Huang. Query-centric trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 17863- 2023.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ABLATION ON HOW TO PRE-TRAIN ADDITIONAL. DATASETS
We sincerely thank Reviewer TuBk for this contributive suggestion on our paper. The original discussion can be seen at: http:://ppenreviee.ee/ffmmmm
We use Argoverse 2 as additional data to pre-train HiVT and Argoverse as the downstream target dataset, we explored two pre-training settings:
We pre-train the model on Argoverse with the standard motion prediction task, and then fine-tune it to Argoverse.
We directly train the model on Argoverse and Argoverse 2 with the standard motion pre- diction task.
A minor design choice is that Argoverse has longer trajectory horizon than Argoverse. When pre-training on Argoverse we could either randomly sample trajectory segments from the full tra- jectory, or use fixed time window. For more comprehensive exploration, we explored both. For the fixed time window choice, considering Argoverse 2 data has 110 waypoints and Argoverse requires 50 waypoints (20 as inputs and 30 as outputs), we use Argoverse 2's original current timestep, and collect 20 historic waypoints as input and 30 future waypoints as output.
We show the results from the first approach (random window) in Table [ and the results from the second approach (fixed window) Table 7
Validation Set: Argoverse
Backbone
Pre-training Dataset
Finetuning Dataset
minFDE +
minADE.
MR +
Argoverse
0.969
0.661
0.092
HiVT
Argoverse 2
Argoverse
1.077
0.701
0.112
Argoverse Argoverse 3.359
2.092
0.636
Table Ablation study on motion prediction as the pre-text task for pre-training additional datasets (Argoverse 2 random window). Pre-training with the motion prediction task and then finetuning or directly training with mixed data does not boost prediction performance. They achieve worse prediction results than training from single dataset.
Validation Set: Argoverse
Backbone
Pre-training Dataset
Finetuning Dataset
minFDE &
minADE
MR |
Argoverse
0.969
0.661
0.092
HiVT
Argoverse (fixed)
Argoverse
1.078
0.697
0.112
Argoverse + Argoverse 2 (fixed)
1.214
0.762
0.133
Table 7: Ablation study on motion prediction as the pre-text task for pre- pre-training additional datasets (Argoverse fixed window). Pre-training with the motion prediction task and then finetuning or directly training with mixed data does not boost prediction performance. They achieve worse pre- diction results than training from single dataset.
Interestingly, for both approaches, poor performance is observed when directly motion pre- diction as the pretraining task, or directly train the model from a mix of the two datasets (especially when we random sample from the additional dataset). It could be presumably due to: the features learned by motion prediction are less transferable or robust, compared to the features learned from SSL tasks; 2) the trajectory distribution between different datasets is quite different, and could be pronounced when pre-training is performed on the standard prediction task.
16

---

Published as a conference paper at ICLR 2025
A.2 ADDITIONAL VISUALIZATION RESULTS
We present additional visualization results to demonstrate the effectiveness of our method. These results categorized into four distinct scenarios: (1) trajectory alignment, (2) long trajectory pre- diction, (3) novel behavior generation, and (4) smooth and safe trajectory synthesis.
Trajectory Alignment. As illustrated in Fig. error of the target agent's trajectories is consistently reduced after pre-training. With more datasets used in pre-training, the trajectories get closer the ground truth, showing enhanced generalization and robustness with multiple datasets pre-training.
showcase scenarios involving the prediction
No Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
No Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
No Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
Figure 4: Visulization Results of Trajectory Alignment. The blue arrows are the model's multi- modal trajectory predictions for the target agent, and the pink arrow is the ground truth future tra- jectory. After pre-traing pre-training, the predicted trajectories get closer to the ground truth.
17

---

Published as a conference paper at ICLR 2025
Long Trajectories. As illustrated in Fig. showcase scenarios involving long trajectory pre- dictions compared against ground truth data. Incorporating additional datasets during pre-training enhances the model's ' ability to capture fine-grained spatiotemporal interactions with objects within the scenario, thereby enabling more accurate trajectory predictions in the long term.
No Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
No Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
Figure 5: Visulization Results of Long Trajectories. The blue arrows are the model's multi-modal trajectory predictions for the target agent, and the pink arrow is the ground truth future trajectory. Our pre-training method enables more accurate trajectory predictions the long term.
18

---

Published as a conference paper at ICLR 2025
Behavior Generation. We present scenarios in which new behavior is learned through pre- training. Fig. 6, the original model struggles to align well with the ground truth across six modal- ity outputs. However, after pre- pre-training, particularly multiple datasets pre-training, the model quires new behaviors, resulting in the generation of more meaningful and skillful trajectories. For example, lane- lane-changing maneuvers (Fig. 6 top), lane-keeping (Fig. 6 middle) and trajectories that are closely synchronized with the ground truth (Fig. bottom..
No Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
No Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
No Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
Figure 6: Visulization Results of Behavior generation. The blue arrows are the model's 's multi-modal trajectory predictions for the target agent, and the pink arrow is the ground truth future trajectory. Our pre-training method can teach new behavior learned through multiple datasets to the model, thus generating more meaningful and skillful trajectories.
19

---

Published as a conference paper at ICLR 2025
Smoothness and Safety Synthesis. In Fig. illustrate scenarios where pre-training facilitates the generation of smoother and safer trajectories compared to models without pre-training. Coarse trajectories are improved into smoother and safer ones, demonstrating the generalizability and ro- bustness the learned features achieved through our multiple datasets pre-training.
No Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
No Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
Pre-training
Single Dataset Pre-training
Multiple Datasets Pre-training
Figure 7: Visulization Results Smooth and Safety Synthesis. The blue arrows are the model's multi-modal trajectory predictions for the target agent, and the pink anrow is the ground truth future trajectory. Our pre-training method contributes to learning smoother and safer trajectories, espe- cially through multiple datasets pre- training.
20

---

Published as a conference paper at ICLR 2025
A.3 RECONSTRUCTED TRAJECTORIES
We present visualization results reconstructed trajectories to demonstrate the effectiveness of our pre-text task learnig. As shown in Fig. & the reconstructed trajectories align closely with the ground truth, indicating the successful learning of our pretext task.
Input Lub-Scenario
Reconstructed Lub-Scenario t'
Input Sub-Scenario
Reconstructed Lub-Scenario
Input Sub-Scenario
Reconstructed Sub-Scenario t'
The blue arrows the trajec- tories of sub-scenario f, and the pink arrows are the trajectories of the entire scenario. Right: The blue arrows are the reconstructed trajectories of sub-scenario t', and the pink arrows are the ground truth trajectories of the sub-scenario f'. Through pre-training, the model successfully reconstructs the sub-scenario f' based sub-scenaric The promising results suggest that the reconstruction task has been effectively learned.
Figure 8: Visulization Results of Reconstructed Trajectories. Left:
21


Reviews:
[{"review_id": "p9l61Blu3L", "reviewer": "Reviewer_SCK1", "summary": "The work proposes a pretraining self-supervised learning framework that can be applied to many models, and trained on different datasets for motion prediction. The pretraining pipeline leverages momentum contrast and generates contrastive pairs by augmenting the same traffic scene with non-overlapping time horizon clips, for the contrastive loss of embeddings and trajectory reconstruction loss. It demonstrates better performance with the pretraining pipeline in two different datasets and various models.", "questions": "1.\tIn Fig 2, the figure labels the component as \"model\" for pretraining. However, since Section 3.1 (problem formulation) indicates that the contrastive loss is calculated on the encoded embeddings, should this component be referred to as the \"encoder\" instead? In the experiments with HiVT, HPNet, QCNet, and Forecast-MAE, are only the encoders of these models pretrained? For models like HPNet, which do not clearly differentiate between encoder and decoder architectures, how do you determine which parts of the model to use for pretraining to obtain latent embeddings? How might this selection influence the results?\n2.\tline 95-97, I find this claim unclear and potentially misleading. I disagree that MAE pretraining is inflexible; on the contrary, I think the masking pretraining is quite versatile. The masking techniques used in the papers you referenced - Rmp, Traj-mae, Forecast-mae, Sept - appear very similar, suggesting the masking concept is not limited and can be readily applied across various models.\n3.\tLine 104, abbreviation CL is not explained before.\n4.\tLine 266 mentions that only complete trajectories are used in pretraining, excluding incomplete ones. During pretraining phase, do you reconstruct single-agent trajectory or multi-agent? The same question applies to the prediction phase. Could you specify what percentage of trajectories remain after filtering for each dataset?\n5.\tLine 296, for eq.(1), why the case of i=j is not excluded in the second term of the denominator, as in this case, it equals to the positive pairs (numerator part) that try to maximize, so it this seems to conflict with the intent.\n6.\tLine 485, different reconstruction strategies (categories and reconstruction target of Table 5) are confusing. It is unclear how different options for reconstructing trajectories starting at t' are justified or implemented. What do you mean by historical information for the sub-scenarios of t'? Could you clarify these strategies?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. Good performance is achieved on the AV, AV2 datasets, with different models.\n3. The exploration of the concept of model-agnostic and dataset-agnostic is very good.", "weaknesses": "1.\tMany details are missing, which may hinder reproducibility. For instance, in the pretraining phase, it is unclear how the values for t and t' are selected for each dataset to avoid overlapping for the experiments. The horizons of the sub-scenario as input and reconstruction are also not specified; it would be helpful to know if these are consistent with the motion prediction settings (either input or output horizons?) used during fine-tuning. Additionally, the default lambda value for the loss function is not provided. Clarifications on which parts of each network are used for pretraining would be beneficial (refer to question 1 below). Will the code be made available as open-source?\n2.\tComputational cost is not shown and compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PXDTL9oMdc", "reviewer": "Reviewer_wPEn", "summary": "In this paper, the authors present SmartPretrain, a novel self-supervised learning (SSL) framework that is model-agnostic and dataset-agnostic. This framework aims to overcome the challenges associated with the scarcity of large-scale driving datasets for motion prediction and the reliance of existing SSL pre-training methods on specific model structures. SmartPretrain incorporates both contrastive and reconstructive SSL approaches and features a dataset-agnostic scenario sampling strategy that combines multiple datasets. Extensive experiments validate the effectiveness of SmartPretrain in motion prediction.", "questions": "Please see weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The field is worthy researching, and the motivations behind the method is clear.\n2.The paper is well-organized.\n3.The paper proposes to pre-train from composition of different sources data, which is rarely researched in motion prediction areas before.", "weaknesses": "1. While the method introduces a novel paradigm for trajectory prediction, the pretrain-finetune approach has been widely adopted across various domains like NLP and CV for years, making it less valuable. Consequently, the contribution of SSL for model training is incremental.\n2. The technical contribution of the dataset sampling strategy appears limited: aspects like standardizing representations, ensuring data quality, and maximizing volume and diversity are fundamental considerations when integrating different data sources.\n3. With only one SSL pre-training baseline and a single dataset in Table 2, it may be challenging to substantiate the proposed method’s advantages over other SSL approaches.\n4. If possible, additional visualization results from the authors would be highly valuable.\n5. The pre-training needs 32 Nvidia A100 40GB GPUs for 128 epochs, which takes abundant computational resources. However, the improvements are not that significant.\nIn short, the contributions of the papers are limited, especially the technical part. I think the paper cannot meet the standard of ICLR conference,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dq9cxKevv0", "reviewer": "Reviewer_ND2Q", "summary": "This paper introduces a self-supervised learning framework for motion forecasting that is both model-agnostic and dataset-agnostic. The approach unifies data samples from various motion forecasting datasets, such as WOMD, AV1, and AV2, making it feasible to pretrain models on large-scale, multi-source data. The framework incorporates both conservative learning and a reconstruction task, achieved through Trajectory Contrastive Learning (TCL) and Trajectory Reconstruction Learning (TRL). Experimental results demonstrate significant performance improvements across multiple architectures and datasets, validating the effectiveness of the proposed methods.", "questions": "1. In the experiment section (Table 1), it is noted that both HPNet and Forecast-MAE were not pretrained on all three datasets, reportedly due to \"compute constraints.\" This reasoning should be clarified further to help readers understand the specific limitations or challenges involved.\n\n2. The reconstruction task is commonly employed in motion forecasting pretraining frameworks, with two main approaches: predicting masked tokens (as in Forecast-MAE) or predicting masked tail trajectories (as in SEPT[2]). The proposed method follows a strategy similar to the latter, which has been shown to outperform the token prediction approach in [2]. It would be beneficial to emphasize the main distinctions of the proposed method from this established approach to further highlight its contributions.\n\n[1] Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoders\n\n[2] SEPT: Towards Efficient Scene Representation Learning for Motion Prediction", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-organized and easy to follow. The experimental results are thorough, covering various datasets and methods, and providing strong evidence for the method's effectiveness.\n\n2. While conservative learning and reconstruction tasks are common in self-supervised learning frameworks for motion forecasting, this submission introduces some innovative strategies that add value.\n\n3. The approach to unifying data representation from diverse sources could pave the way toward a foundation model for motion forecasting.", "weaknesses": "While the novelty of this submission may be somewhat limited and most techniques are already verified in many previous works, it does not present any clear weaknesses. For specific considerations, please refer to the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xCpSNXeAyo", "reviewer": "Reviewer_TuBk", "summary": "This paper offers a universal pipeline to pre-train on real-world motion data for trajectory prediction tasks. Combining popular self-supervised pre-training methods like contrastive learning and reconstruction learning, and with specific designs on motion data domain, this work successfully proposes a pre-training pipeline to learn general representations of *trajectories* in motion data, which can work regardless of the baseline motion prediction models or the motion datasets used. Extensive experiments have been performed on various commonly-used datasets (Argoverse 1/2, Waymo Open Motion Dataset) and state-of-the-art baselines (HiVT/QCNet etc.) to show the effectiveness of the pre-training pipeline proposed. A series of ablation studies clearly ablate the effectiveness of each module of the pre-training pipeline as well as the pre-training data involved.", "questions": "Please see the Weaknesses. I am looking forward to the authors' rebuttal and discussions on those issues.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The pipeline proposed in this work is model-agnostic, i.e., it can be easily extended to any encoder-decoder style motion prediction model.\n2. This work successfully combines different real-world datasets like Argoverse and Waymo, so it can significantly enlarge the available motion data that can be used for a specific motion prediction task, where data scarcity is a significant problem.\n3. Extensive experiments are conducted to prove its general effectiveness across different baseline models and datasets.\n4. Ablation studies are well designed to clearly show i) the effectiveness of each part of the pipeline ii) the influence of pre-training data.", "weaknesses": "1. The downstream motion prediction settings, though already very diverse, seem not being able to cover all necessary cases. For example, no methods fine-tuned on Waymo Open Motion Dataset (WOMD) are presented.\n2. The pre-training performance should be illustrated to prove that pre-training tasks can be done successfully. For example, can you show some examples of reconstructed trajectories?\n3. Direct data mixing in Data-scaled Pre-training is a natural choice, but might not be optimal. For example, WOMD has significant domain gap compared to Argoverse. In this case, a biased weight might be helpful in pre-training stage to lower the influence of WOMD-Argoverse domain gaps.\n4.  An ablation on how to utilize the additional data in the pre-training stage could be added to make Table 3 even more convincing. For example, in Transfer Pre-training and Data-scaled Pre-training, what would happen if the additional data is used to pre-train on the baseline model directly, or even directly to augment the training set for the baseline model?\n5. Some minor questions:\ni) Why to use L1 loss in TRL instead of L2, while the latter is the base for metrics used (MR, FDE, ADE)?\nii) The quantity of data that is complete / incomplete might be presented to help readers understand the quantity of additional data introduced through this work.\n6. The prediction metrics lack error bars. This is not a weakness, but a point that can be even improved. I understand that the motion prediction metrics can be unstable sometimes, but adding error bars onto the most important results would significantly improve the reliabilities of the pipeline proposed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)