SSL-Lanes: Self-Supervised Learning for Motion Forecasting in Autonomous Driving
Authors: Prarthana Bhattacharyya
Chengjie Huang
Krzysztof Czarnecki
Venue: CoRL 2022 Poster (2022)
Keywords: Motion Forecasting
Autonomous Driving
Self-Supervised Learning

Abstract: Self-supervised learning (SSL) is an emerging technique that has been successfully employed to train convolutional neural networks (CNNs) and graph neural networks (GNNs) for more transferable, generalizable, and robust representation learning. However its potential in motion forecasting for autonomous driving has rarely been explored. In this study, we report the first systematic exploration and assessment of incorporating self-supervision into motion forecasting. We first propose to investigate four novel self-supervised learning tasks for motion forecasting with theoretical rationale and quantitative and qualitative comparisons on the challenging large-scale Argoverse dataset. Secondly, we point out that our auxiliary SSL-based learning setup not only outperforms forecasting methods which use transformers, complicated fusion mechanisms and sophisticated online dense goal candidate optimization algorithms in terms of performance accuracy, but also has low inference time and architectural complexity. Lastly, we conduct several experiments to understand why SSL improves motion forecasting.

SSL-Lanes: Self-Supervised Learning for Motion Forecasting in Autonomous Driving
Prarthana Bhattacharyya
Chengjie Huang
Krzysztof Czarnecki
University of Waterloo, Canada
{p6bhatta, c. c.huang, k2czarme euraarroeeuuaateooooooca
Abstract: Self-supervised learning (SSL) is emerging technique that has been successfully employed to train convolutional neural networks (CNNs) and graph neural networks (GNNs) for more transferable, generalizable, and robust repre- sentation learning. However its potential in motion forecasting for autonomous driving has rarely been explored. In this study, report the first systematic explo- ration and assessment of incorporating self-supervision into motion forecasting. We first propose investigate four novel self-supervised learning tasks for motion forecasting with theoretical rationale and quantitative and qualitative comparisons the challenging large-scale Argoverse dataset. Secondly, we point out that our auxiliary SSL-based learning setup not only outperforms forecasting methods which use transformers, complicated fusion mechanisms and sophisticated online dense goal candidate optimization algorithms in terms of performance accuracy. but also has low inference time and architectural complexity. Lastly, we conduct several experiments to understand why SSL. improves motion forecasting.
Keywords: Motion Forecasting, Autonomous Driving, Self-Supervised Learning
Introduction
Motion forecasting in real-world urban environment is important task for autonomous robots. It involves predicting the future trajectories of traffic agents including vehicles and pedestrians. This is absolutely crucial in the self-driving domain for safe, comfortable and efficient operation. However, this is a very challenging problem. Difficulties include inherent stochasticity and multimodality of driving behaviors, and that future motion can involve complicated maneuvers such yielding. nudging, lane- channchanging,, turning and accelcration or decelcration.
The motion prediction task has traditionally been based on kinematic constraints and road map in- formation with handcrafted rules. These approaches however fail to capture long-term behavior and interactions with map structure and other traffic agents in complex scenarios. Tremendous progress has been made with data-driven methods in motion forecasting [3, 4, 5, 6, 7, 8, 9, 10]. Recent methods vector representation for HD maps and agent trajectories, including approaches like Lane-GCN [2]. Lanne-RCNN [11], Vector-Net [12), TNT [5] and Dense-TNT [6]. More recently, the enormous success of transformers [13] has been leveraged for forecasting in mm-Transformer [9], Scene transformer [8], Multimodal transformer [ [14] and Latent Variable Sequential Transformers [15]. Most of these methods however are extremely complex in terms of architecture and have low inference speeds, which makes them unsuitable for real-world settings.
In this work, we extend ideas from self-supervised learning (SSL) to the motion forecasting task. Self-supervision has seen huge interest in both natural language processing and computer vision [16] to make use of freely available data without the need for annotations. It aims to assist the model to learn more transferable and generalized representation from pseudo-labell via pretext tasks. Given the recent success of self-supervision with CNNs, transformers, and GNNs, are naturally motivated to ask the question: Can supe-supe learning improve accuracy and generalizability of motion forecasting, without sacrificing inference speed architectural simplicity?
Contributions: Our work, SSL-Lanes, presents the first systematic study on how to incorporate self-supervision in standard data-driven motion forecasting model. Our contributions are: (a)
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Bocline
Preseat Tisk Gonnowliinn disalisioo
theee (hrennkingg
odatane (ieewtiie)
Ego
Other agents
Pust traeectory
GT trgjctory
enjectery
Figure 1: Motion forecasting on Argoverse [1] validation. We show four challenging scenarios at intersections. The baseline [2] misses all the predictions. In the first row, our proposed lane masking successfully captures the right-turn. For the second row, predicting distance to intersection helps the most in capturing the left turn. In the third row, acceleration at an intersection is best captured by the model that is made to classify maneuvers of traffic agents. Finally, in the fourth row, classifying successful final goal states is the most effective at capturing the left turn. These tasks are trained with pseudo-labels which are obtained for free from data.
We demonstrate the effectiveness of incorporating self-supervised learning in motion forecasting. Since this does not add extra parameters or compute during inference, SSL-Lanes achieves the best accuracy-simplicity-ffiii
trade-off the challenging large-scale Argoverse [1] benchmark. (b) We propose four self-supervised tasks based on the nature of the motion forecasting problem. The key idea is to leverage easily accessible map/agent-level information to define domain- -specffc pretext tasks that encourage the standard model to capture more superior and generalizable represen- tations for forecasting comparison pure supervised learning. (c) We further design experiments to explore why forecasting benefits from SSL. We provide extensive results to hypothesize that SSL-Lanes learns richer features from the SSL training as compared a model trained with vanilla supervised learning.
Related Work
Motion Forecasting: Traditional methods for motion forecasting primarily use Kalman filtering [17] with prior from HD-maps to predict future motion states [18, 19]. With the huge success of deep learning, recent works use data-driven approaches for motion forecasting. These methods explore different architectures involving rasterized images and CNNs [3, 20, 21], vectorized repre- sentations and GNNs [12, 11, 22, 4, 7], point-cloud representations [23]. transformers [8, 9, 15, 14] and sophisticated fusion mechanisms [2], generate features that predict final output trajectories. While the focus of these works is to find more effective ways of feature extraction from HD-maps and interacting agents, they need huge model capacity, heavy parameterization, and extensive aug- mentations or large amounts of data converge build on them to incorporate prior knowledge the form of predefined candidate trajectories ob- tained from sampling or clustering strategies from training data. However the disadvantage of these methods is that their performance is highly related to the quality of the trajectory proposals, which becomes extra dependency. End-to-end solutions for optimizing end-points of these candidates trajectories are proposed by Pense-TNT [6] and HOME [26]. Dense- Conse-INTTT has state-of-the-an accu- racy with reasonable parameter budget, but its online dense goal candidate optimization strategy is computationally very expensive, which is unrealistic for real-time operations like autonomous driv- ing. Lately, ensembling techniques like MultiPtth+ [27] and DCMS [28] have been proposed and while they have high forecasting performance, major disadvantage is their high memory cost for training and heavy computational cost inference.
general solution. Other works [5, 10, 24, 25]
Self-supervised Learning: SSL. is a rapidly emerging learning framework that generates additional supervised signals to train deep learning models through carefully designed pretext tasks. In the

---

Forecasting Model (Sec. 3)
Agent Encoder (Grnc)
Interaction Madeling (A)
Trajectory Decoder (gaea)
GT
Supervised Loss faep
Predicted Trijectories
Map + Agents
Map Encoder
Succi Fastao
Pretest task I: Lane Masking (Sec. 4.1.1)
Pretext task 3: Maneuver Clacsiftcation (Sec. 4,1.3)
Lan
Festas
Tirn
Eatrope
Consticator
Mip-Awar Agrii Testrrs
Maneeser blwis from unageeviee cluccring
Pretext task Distance to Intersection (Sec. 4.1.2)
Pretext task 4: Success/Faiture Classification (See. 4.1.4)
Surocio X
MSE Lass
insied code
Sacceso Fabluc
Cress Ketropy
GT Nacemce
Map Frateoss
Product forezad closs
Agrnt Traterss
Figure 2: Illustration of the overall SSL-Lanes framework for self-supervision on motion forecast- ing through joint training. SSL-Lanes improves upon standard-motes forecasting baseline, that consists of an agent encoder, map encoder, interaction model and a trajectory decoder, trained using supervised loss Lau.. SSL-Lanes proposes four pretext tasks: (1) Lane Masking: which recovers feature information from the perturbed lane graphs. (2) Distance to Intersection: which predicts the distance (in terms of shortest path length) from all lane nodes intersection nodes. (3) Ma- neuver Classification: predicts the form of a 'maneuver' the agent-of-interest intends to execute (4) Success/Failure Classification: which trains an agent specialized achieving end-point goals.
image domain, various self-supervised learning techniques have been developed for learning high- level image representations, including predicting the relative locations of image patches [29]. jigsaw puzzle [30], image rotation [31], image clustering
[32], image inpainting [33], image colorization [34] and segmentation prediction [35]. In the domain of graphs and graph neural networks, pretext tasks include graph partitioning, node clustering, context prediction and graph completion [36, 37, 38, 39]. To the best of our knowledge, this is the first principled approach that explores motion forecasting for autonomous driving with supervviri..
3 Background
Problem Formulation: We are given the past motion of N actors. The i-th actor is denoted as of center locations over the past time-steps. We pre-process to represent each trajectory as sequence of displacements P: [Ap, L+1 L+1 Ap, App), where pt is the 2D displacement from time step I 1 to I. We are also given a high-definition (HD) map, which contains lanes and semantic attributes. Each lane composed of many consecutive lane nodes, with a total of M nodes. X E R M denotes the lane node feature matrix, where 2, = X =] is the F F-dimensional lane node vector. Following the connections between lane centerlines (i.e., predecessor, successor, left neighbour and right neighbour), we represent the connectivity within the lane nodes with 4 adjacency matrices {A,) JE sncffssc.. with A R MM This implies that if Af.gh 1, then node is an f-type neighbor of node Our goal is to forecast the future motions of all actors in the scene OUT {a v.). y!).... (o. V. "") I.... N). where T is our prediction horizon.
Standard Motion Forecasting Model: We briefly introduce a standard data-driven motion fore- casting framework, consisting of a feature encoder, interaction-modeler and prediction header.
Feature Encoding: We first encode the agent and map inputs similar to Lane-GCN [2]. The agent
encoder includes ID convolution with feature pyramid network, parameterized by Jene given by Eq. (1). For map-encoding, we adopt two Lane-Conv residual blocks, parameterized by θ = (Wo, o, W lea, W right, W pre,,e W ), where à {1, 4, 8, 16, 32), given by Eq. (2).
P: Jene (P:)
(1)
Y X Σ A + AL. X.. ge. + A NWW XW suc,
(E(l...))))
&
Modeling Interactions: Since the behavior of agents depends on map topology and social consis- tency, each encoded agent i subsequently aggregates context from the surrounding map features and

---

SSL Task
Property Level
Primary Assumption
Type
Lanc- Lane-Maskee Masking
Local mäp structure
Aux. auto-encoder
Map features
Distance to Intersection
Global map structure
Aux. regression
Mancuver Classification
Map-aware
Agent feature similarity
Aux. classification
Success/Failure Classification
agent features
Distance success state
Table 1: Overview of our proposed self-supervised (SSL.) tasks
its neighboring agent features, via spatial attention [40] as given by Eq. (3):
P; p. W M2A
Σ
(cconeat(p.. A... U3) W
/. = p. A2A + Σ ((concat(p.. A.g. P.s, W))
Here, y is the feature of the j-th node, p; is the feature of the i-th agent, 6 the composition of layer normalization and ReLU, and A. = MLP((, (P(Uu-0;)) where v denotes the (x, y) 2-D bird's s-eye-view (BEV) location of the agent the lane node. The parameters for map and agent feature aggregation represented by A (WMZA. MZA. W1, W2, WZA.. W3, Wa}.
Finally, we decode the future trajectories from the features p corresponding to the agents of interest as given by: Oped = (f))= I,.., N}. where Jas: the parameter- ized trajectory decoder. The parameters for the motion forecasting model are learned by minimizing the supervised loss (Lmp) calculated between the predicted output and the ground-truth future tra- jectories (0 as given by Eq. (4):
Trajectory Prediction:
Jenc: θ", A Jdse
arg min
Coup (Opele
L:T OCT
9eac, G.A.
SSL-Lanes
The goal of our proposed SSL-Lanes framework is improve the performance of the primary motion forecasting baseline by learning simultancously with various self-supervised tasks. Fig. shows the pipeline of our proposed approach, and Tab. summarizes the self-supervised tasks.
Considering our motion forecasting task and a self- supervised task, the output and the training process be formulated as:
Self-Supervision meets Motion Forecasting:
V S2 e v p,
arg min a1 (V, 14 (2) az (V 2x)
loss function of the self-supervised task, O parameterizes the corresponding task-specific layers, and (1, d2 E R²0 are the weights for the supervised and self-supervieed losses. the pretext task only focuses on the map encoder, then & = (θ} and S2 = {9ence A, 9dce Other- also define the following representations.
where, Lu((,))
wise, L = {geme, O, }} and Ω {9dce Henceforth, We will represent the primary task encoder as function fw, parameterized by w. Furthermore, given pretext task, which will design in the next section, the pretext decoder pe. function that predicts pseudo-labels and parameterized by e..-
4.1 Pretext tasks for Motion Forecasting
At the core of our SSL--Lanes approach is defining pretext tasks based upon self-supervised informa- tion from the underlying map structure and the overall temporal prediction problem itself (Tab. 1).
4.1.1 Lane-Masking
The goal of the Lane-Masking pretext task is to encourage the map encoder v = structure information addition to the forecasting task that being optimized. Specifically, we randomly mask (i.e., set equal to zero) the features of m. percent of nodes per lane and then ask the self-supervised decoder to reconstruct these features.
{ to learn local
1 ma
T O arg min
Σ [m1
Lane (pe. (f)(,
(6)
V. V.

---

Method
minADE,
minFDE,
MR.
minADE;
minFDE,
MR.
Baseline
1.42
3.18
51.35
0.73
1.12
11.07
Lane-Masking
1.36
2.96
49.45
0.70
1.02
8.82
Distance to Intersection
1.38
3.02
49.53
0.71
1.04
8.93
Maneuver Classification
1.33
2.90
49.26
0.72
1.05
9.36
Success/Failure Classification
1.35
2.93
48.54
0.70
1.01
8.59
Table 2: Motion forecasting performance on Argoverse validation with our proposed pretext tasks
Here, X is the node feature matrix corrupted with random masking, i.e., some rows of X corre- sponding to nodes v2 are set to zero. pe,, is a fully connected network that maps the representations the reconstructed features. Emse the mean squared enror (MSE) loss function penalizing the distance between the reconstructed map features pe. (lfv (X,, (s)))) for node v, and its actual features X;
4.1.2 Distance to Intersection
cotanne-to--nterseetti pretext task proposed to guide the map-encoder, V = ( to maintain global topology information by predicting the distance (in terms of shortest path length) from all lane nodes to intersection nodes. We aim to regress the distances from each lane node to pre- labeled intersection nodes annotated as part of the dataset. Given K labeled intersection nodes Vintersection { (Uintersectioo. lk 1).. search (BFS). Specifically, BFS calculates the shortest distance d, R for every lane node v; from the given set Vimersection- The target of this task is to predict the pseudo-labeled distances using a pretext decoder. If pe,, ([, (X, A f f)).;) ) is the prediction of node u,, and Lise is the mean-squared emror loss function for regression, then the loss formulation for this SSL pretext task is as follows:
) we first generate reliable pseudo labels using breadth-fire
O arg min 9,
M Σ Emse Cmme(ppe.. (pen (I A1)b))))
4.1.3 Maneuver Classification
We propose Maneuver Classification, and we expect it to provide prior regularization V (Jene: A). based on driving modes of agents. We aim to construct pseudo label to di- vide agents into different clusters according supervised clustering algorithms to acquire the maneuver for each agent. We find that us- ing naive k-Means (on agent end-points) or DBSCAN (on Hausdorff distance between entire trajectories [41]) leads to noisy clustering. We find that constrained k-means [42] on agent end-points works best to divide trajectory samples into C clusters equally. We define C imaintain-speed accelerate, decelerate, turn-left, turn-right, lane-change) and the clustering func- tion p. If pe... (fw (P;, X, Ay)) is the prediction of agent i's intention and E; (œ._er= V.,rr its ground-truth end-point, then the learning objective is to classify each agent maneuver into its corresponding cluster using cross-entropy loss Lae as:
their driving behavior and thus explore un-
O arg min Lae (pe. (P., (P). V.
(8)
4.1.4 Forecasting Success/Failure Classification
We propose a pretext task called Success/Failure Classification, which trains an agent specialized at achieving end-point goals and thus links directly to the forecasting task. We expect this to constrain v {Jene O A to predict trajectories distance away from the correct final end-point. Similar to maneuver classification, we wish to create pseudo-labels for our data samples. We label trajectory predictions as successful (c 1) if the final prediction (Iifpred Vi,peaa is within < 2m of the final end-point F,, and as failure (c = 0) otherwise. We choose 2 m as our threshold because it is also used for miss-rate calculation (Sec. 5). the pretext decoder predicts agent i's final-endpoint pe.. (f (P,, X, ))) and, given the ground-trute truth end-point E,, its success failure label is c;
5

---

Method
minADE;
minFDE,
MR
minADE;
minFDEg
MR;
b-FDEs
NN Map [1]
3.65
8.12
94.0
2.08
4,02
58.0
Jean [4]
4.24
68.56
0.98
1.42
13.08
2.12
Lane-GiN [2]
1.71
3.78
58.77
0.87
1.36
16.20
2.05
LaneRCNN [11]
56.85
0.90
1.45
12.32
2.15
TNT [5]
1.77
3.91
59.70
0.94
1.54
13.30
2.14
DenseTNT [6]
1.68
3.63
58.43
0.88
1.28
12.58
1.97
PRIME [24]
1.91
3.82
58.67
1.22
1.55
11.50
2.09
WIMP [7]
1.82
4.03
62.88
0.90
1.42
16.69
2.11
TPCN [23]
1.66
3.69
58.80
0.87
1.38
15.80
1.92
HOME [26]
1.70
3.68
57.23
0.89
1.29
8,46
1.86
mmnrransformer [9]
1.77
4.00
61.78
0.87
1.34
15.40
2.03
ColtiMoodlTTransfomoee [14]
1.74
3,90
60.23
0.84
1.29
14.29
1.94
LatentVariable Transformer [15]
0.89
1,41
16.00
SceneTrransoomee Transformer [8]
1.81
4.06
59.21
0.80
1.23
12.55
1.88
Success/Failure
1.63
3.56
56.71
0.84
1.25
13.26
1.94
Classification (Ours)
Table 3: Comparison of our (best) proposed model and top approaches on the Argoverse Test. The best results are in bold and underlined, and the second best is also underlined.
then the pretext loss can be formulated as:
min Loo (f (fv A,))), v
4.2 Learning
As all the modules are differentiable, we can train the model in an end-to-end way. We use the sum of classification, regression and self-supervised losses to train the model. Specifically, we use:
Lets + Lreg + Lierminal + Lss
(10)
For classification and regression loss design, adopt the formulation proposed in [2). Lieriinal 1 N Eim L2 L2((1)) (ITTpred vippeaa), (I b(,T)) is simple L2 loss that minimizes the distance between predicted final-endpoints and the ground-truth. This is because Lry is averaged across all time- points 1 T, and from a practical end user perspective, minimizing the endpoint loss is much more important than weighting loss from all time-steps equally. Our proposed pretext tasks contribute to Lg. During evaluation, we study each pretext task separately, and their corresponding loss formula- tions defined in Eq. (6), Eq. (7), Eq. (8), Eq. (9) are used for joint training.
Experiments
Dataset: Argoverse provides large-scale dataset, where the task is forecast seconds of future motions, given 2 seconds of past observations. It has more than 300K real-world driving sequences collected Miami (MIA) and Pittsburgh (PIT). Those sequences are further split into train, val- idation, and test sets, without any geographical overlap. Each of them has 205,944 ,4,,,2, 39,472, and 78, 143 sequences respectively. scene within the past 2 seconds history, annotated at 10Hz. It also specifies one actor of interest the scene, with type 'agent', whose future seconds of motion are used for the evaluation. The train and validation splits additionally provide future locations of all actors within 3 second horizon labeled at 10Hz, while annotations for test sequences are withheld from the public and used for the leaderboard evaluation. HD map information is available for all sequences.
particular, each sequence contains the positions of all actors in
To normalize the data, we translate and rotate the coordinate system each sequence so that the origin is at current position = of 'agent' actor and x-axis is aligned with its current direction, i.e., orientation from the agent location at -1 to the agent location is the positive x axis. We use all actors and lanes whose distance from the agent is smaller than 100 meters as input. We train the model on TITAN-X GPUs using batch size of with the Adam [43] optimizer with initial learning rate of x 10-3 which is decayed to 1 x 10-4 at 100,000 steps. The training process finishes at 128,000 steps and takes about 10 hours to complete. We provide more implementation details in the supplementary.
Experimental Details:

---

1.8
MuldPati
Lane-GCN
Lane-ICN
1.35
1.35
min- min-FDE Rin--DDB (m))
1.6
min-FDB (m)
1.4
TNT
1.30
min-FDE
1.30
Dense-TNT
mun Transformer
Dense-TNT
WayPoinn
1.2
Lane-RCNN GCN
Ours
1:25
Durs
1.0
Ours
Dense-TNT
Soene-Transormee
Scene-Transfomer
Miss-Rate 2m
Model Purams (M) (b)
Inference time (ms)
(a)
(c)
Figure 3: (a) min- Miss-Rate trade-off Argoverse Validation. Lower-left is better. We optimize both successfully in comparison to other popular approaches. (b) and (c) We plot min-FDE on Argoverse Test against number of model parameters (in millions) and inference time (in milli- seconds). We find that there a trade-off between min-FDE performance, architectural complexity (as measured by number of parameters) and computational efficiency (as measured by inference time). Our work achieves the best trade-off (lower-left).
Results
6.1 Ablation Studies
We first examine the effect of incorporating our proposed pretext tasks (Sec. 4) with the standard data-driven motion forecasting baseline (Sec. 3). While evaluating the importance of our proposed pretext tasks, we wish to underline that motion prediction for autonomous driving is safety-critical task, especially at intersections where most of our data is collected, and most accidents also happen. We thus posit that in this situation, even a small стгоr in predicting final locations (FDE) for a given agent can lead dangerous potential collision scenarios. Results in Tab. 2 show that all proposed pretext tasks improve motion forecasting performance for Argoverse. Specifically, the Lane Mask pretext task improves min-FDE by 8.9% and MR @2 by 20.3%. Distance Intersection improves min-FDE by 7.1% and 19.3%. Maneuver classification improves min-FDE by 6.3% and MR @ @2m by 15.4%. We expect that improving the quality of clustering for maneuvers and thus creating better pseudo-labels will improve this further. Finally, Success/Failure classification improves min-FDE by 9.8% and, perhaps expectedly, MR 2m by 22.4%. Moreover, since pretext tasks are not used for inference and only for training, they also do not add any extra parameters or FLOPs to the baseline, thereby increasing accuracy but at no cost to computational efficiency or architectural complexity.
6.2 Comparison with State-of-the-Art
We compare our approach with top entries Argoverse [1] in Tab. 3. SSL-Lanes improves the metrics for K = convincingly and outperforms existing approaches w.r.t. min- ADE1, min-FDE, and MR, We strongly competitive w.r.t. min-ADEe, Fin- and MRg- with a relatively simple architecture.
Performance:
Trade-off between min-FDE and Miss-Rate: min-FDEs and MRg are both important for au- tonomous robots to optimize. Ideally we wish both of these metrics to low. However, there exists frequent trade-off between them. We compare this trade-off in Fig. 3(a) w.r.t 6 other pop- ular motion forecasting models (in terms of citations and GitHub stars), namely: Lane-GCN [2], Lane-RCNN [2]. MultiPath [3], mm- Transformer [9]. TNT [5] and Dense-TNT [6] the Argov- erse Validation Set. We are on the lowest-left of Fig. 3(a), meaning we optimize both min-FDEg and MR₁ successfully in comparison to other top models.
Trade-off between accuracy, efficiency and complexity: We are the first to point out trade-off that exists for current state-of-the-as art motion forecasting models between forecasting performance, architectural complexity and inference speed, this work. This is illustrated Fig. 3(b, c). In contrast to the popular models, our approach has high accuracy (min-FDEg: 1 1.25m, MRg: 13.3%), while also having low architectural complexity (1 8.44 parameters) and high inference speed (3.3 ms). Thus it provides great balance for application to real-time safety-critical autonomous robots.
7

---

Experimental Setup
Description
Method
minADEg
minFDE,
MR;
Training
Validation
Effects of limited
Baseline
0.82
1.33
14.66
25% of train
All
training data
Ours
0.78
1.22
12.63
Effects of domain
100% PIT 20% MIA
Baseline
0.88
1.46
17.21
MIA val
Ours
0.85
1.34
14.96
Performance on difficult mancuvers
Turning &
Baseline
0.90
1.53
19.90
All
lane changing
Ours
0.84
1.34
14.93
Effects of imbalanced data
2x straight
Turning & lane changing
Baseline
0.94
1.65
21.53
Ix other maneuvers
Ours
0.90
1.49
17.97
Effects of noisy data
Gaussian noise (a 0.2)
Baseline
1.01
1.37
15.59
All
with p= 0.25
Ours
0.96
1,24
11.98
Effects of noisy data
Gaussian noise (o 0.2)
Baseline
1.19
1.56
20.64
All
with p 0.5
Ours
1.13
1.40
15.65
Table 4: Different experimental settings for SSL-based training
6.3
When does SSL help Motion Forecasting?
We design 6 different training and testing setups shown in Tab. 4. We use Success/Failure classifi- cation as the pretext task, and all models are trained for 50,000 steps. We initialize the map-encoder with the parameters from model trained with the lane-masking pretext task.
We hypothesize that training with SSL pretext tasks helps motion forecasting in the following ways: (a) Topology-based context prediction assumes feature similarity smoothness in small neighbor- hoods of maps, and the resulting feature representation may improve prediction performance. This mainly expected to help the first and second settings, which requires generalizing to new topolo- gies. (b) Clustering and classification assumes that feature similarity implies target-label similarity and can group distant nodes with similar features together, leading to better generalization. This is mainly expected to help with dataset imbalance and performance on difficult maneuvers, which re- quires generalizing to hard cases. (c) Supervised learning with imbalanced datasets sees significant degradation in performance. Although most of the data samples in Argoverse are at an intersection, a significantly large number involve driving straight while maintaining speed. Recent studies [44] have shown that SSL tends to learn richer features from more frequent classes, which also allows it to generalize to other classes better. We expect this to help with imbalanced data, limited training data and noisy data.
SSL leads to better generalization compared to pure supervised learning: To provide evidence for our hypotheses, we design 6 different training and testing setups as shown in Tab. 4. We use Success/Failure classification as the pretext task, and all models are trained for 500,000 steps. We initialize the map-encoder with the parameters from model trained with the lane mask pretext task. There is strong evidence from our experiments that SSL--based tasks provide better generalization and can thus be more effective than pure supervised training.
7 Conclusion
We propose SSL-Lanes to leverage supervisory signals generated from data for free in the form of pseudo-labels and integrate with a standard motion forecasting model. We validate our proposed approach by achieving competitive results the challenging large-scale Argoverse benchmark. We further demonstrate that each proposed SSL pretext task improves upon the baseline, especially in difficult cases like left/right turns and acceleration/decelrrttoo We also provide hypotheses and experiments on why SSL.-Lanes can improve motion forecasting.
The different losses for are only used in 1:1 ratio without tuning them. We also use only one pretext task a time and do not explore the combination of these different tasks. For our future work, we plan to incorporate meta-learning [45] to identify effective combination of pretext tasks and automatically balance them. Another limitation is that we report improvements with SSL- pretext tasks in scenarios without specifically considering multiple heavily interacting agents. In the future we would like to explore how the interactions between road agents can influence our SSL losses the interaction split of the Waymo Open Motion dataset (WOMD) [46]. We would also like to study the generalization of our work to other datasets without re- re-training.
Limitations:

---

Acknowledgments
This research was funded by the Mitacs Accelerate Program and Gatik Inc. This article solely reflects the opinions and conclusions of its authors.
References
[1] M. Chang, J. Lambert, P. Sangkloy, Singh, Bak, A. Hartnett, D. Wang, P. Car S. Lucey, D. Ramanan, and J. Hays. Argoverse: 3d tracking and forecasting with rich maps. In IEEE Conference on Computer Vision and Pattern Recogni- tion. CVPR 2019, Long Beach, CA, USA. Jume 16-20, 2019, pages
10.110.109//CPP.2019990 Computer Vision Foundation / IEEE, 2019. httpe220199/mm//ee http:://ppenaccess thecvf. wtt.coooo/////eeeeeee content_CVeR Tracking_ Fow.eaastoo..aiiinnnnno.. With Rich Maps__apP_aa_eeaaaaae22apa__ 2019 _ap.errpprrp... pappr.html.................m...m
URL wew.nnggrrrrrree...om html.. 2, 6,
[2] M. Liang, B. Yang, R. Hu, Y. Chen, Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forecasting. In ECCV, 2020. 1, 7,
[3] Y. Chai, B. Sapp, M. Bansal, and D. Anguelov. Multipath: Multiple probabilistic anchor trajec- tory hypotheses for behavior prediction. In P. Kaelbling, D. Kragic, and K. Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 Novem- ber 1, 2019, Proceedings, volume 100 of Proceedingy of Machine Learning Research, pages mlr, press/v100/ http://v00//chaiiiii chai 20a. html. 20mmlmm...........mmmm...mmmm1.. 2,
86-99. PMLR, 2019. URL 7
[4] J. Mercat, T. Gilles, N. E. Zoghby, G. Sandou, D. Beauvois, and G. P. Gil. Multi-head attention for multi-modal joint vehicle motion forecasting. In 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May 31 August 31, 2020, pages 9638- 9644. IEEE, 2020. doi: 111.0.111//CRRAAAII WAW.://KAAA99952220.099999999 1109/ICR440945, 2020 9197340. 2, 6
URL. https:: //ooi. http://oi.oooooo org/10.
[5] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, C. Li, and D. Anguelov. TNT: target-driven trajectory prediction. In J. Kober, F. Ramos, and C. J. Tomlin, editors, 4th Conference 2020, Virtual Event / Cambridge, MA. USA, volume 155 of Proceedings of Machine Learning Research, pages 895-904. PMLR, 2020. URL https::/proceeeeeeeei zhao21b. html. 1, 2, 6.7
Robot Learning. CoRL 2020, 16- 18 November
mlr, press/v155/
[6] J. Gu, C. Sun, and H. Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 15283-15292. IEEE, 2021. doi: 10. 11.1110.11101 1091CCV48922 2021.01502. URL pettps//doi:. org/ wtt.://gg//...//........ 1109/ICCV48922 2021. 2021. 1, 2, 6,
[7] Khandelwal, W. Qi, Singh, A. Hartnett, and D. Ramanan. What-if motion prediction for autonomous driving. CoRR, abs/2008. 10587, 2020. URI. https:tt:://axtt wwtp://rr///aa.....o// 10587. 1, 6
[8] J. Ngiam, B. Caine, V. Vasudevan, Z. Zhang, H. L. Chiang, J. Ling. R. Roelofs, A. Bewley, C. Liu, A. Venugopal, D. Weiss, B. Sapp, Z. Chen, and J. Shlens. Scene transformer: A unified multi-task model for behavior prediction and planning. CoRR, abs/2106.08417, 2021. URL https: httppp://arxii.or//bbbsete http://bb//......e 08417. 1, 2, 6
[9] Y. Liu, J. Zhang, L. Fang, Q. Jiang, and B. Zhou. Multimodal motion prediction with stacked transformers. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2021, virtual. June 19-25, 2021, pages 7577-7586. Computer Vision Foundation / IEEE, 2021. URL https: / opw./ccssscccsseeeeepp//e/.s...c thecvf. com/ Stacked Braiieeddrraasfffffeeeoee arransoomerss.omm CVWp_________________ 20111_221_ html. 1, 6,
http/////////mmiiuuuuauaa Multimddal.
Oetion_Predictoo 7
[10] S. Casas, W. Luo, and R. Urtasun. Intentnet: Learning to predict intention from raw sensor data. 2nd Annual Conference on Robot Learning, CoRL. 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pages

---

15607/RSS.2019..X 947-956, PMLR, 2018. URL http://procceddigs mlr wttpp//pesss///ssssss///s////// html. 1, 2
[11] W. Zeng, M. Liang, R. Liao, and R. Urtasun. Lanerenn: Distributed representations for graph- centric motion forecasting. In IEEE/RSJ International Conference Intelligent Robots and Systems, IROS 2021, Prague, Czech Republic, September 27 Oct. 1, 2021, pages 532-539. IEEE, 2021. doi: 109/ROS511 68...82.1.... 0021.963 IROS51168 2021. 9636035. 1,2,
URL https: htttps://oi.... http://oooooooto oar.lorgg/// 1109/
[12] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid. Vectornet: En- coding HD maps and agent dynamies from vectorized representation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 11522-11530. Computer Vision Foundation IEEE, 2020. doi: 10.1009/CVPR42600022000 200.244222000220011544 URL bttps::/openaccest
thecvf com/ content_ VectorNet. aantr//eeeeeeeeoooeaaaannnnnnnnn Baaiing__D__MM_ HD_Maps__a___aa and Agent_ Cestorized_Representttee Cttpeetttattttoo_________ 2020__200_______11112222 _aaerr html. 1,
[13] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. von Luxburg, S. Ben- gio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and Garnett, editors, Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017. URL https: /prppe/://ooce neurips wwtpppppe//11/////////// /2a////ah////////////////// Abstract .html.
[14] Z. Huang, X. Mo, and C. Lv. Multi-modal motion prediction with transformer-based neural network for autonomous driving. CoRR, abs/2109.06446, 2021. URL https://arxiv..rg abs/2109. 06446. 1, 2, 6
[15] R. Girgis, F. Golemo, F. Codevilla, M. Weiss, J. A. D'Souza, S. E. Kahou, F. Heide, and C. Pal. Latent variable sequential set transformers for joint multi-agent motion prediction. In 10th International Conference Learning Representations, ICLR 2022, Conference Track Proceedings, 2022. 1, 2,
[16] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties self- supervueee vision transformers. In 2021 IEEE/CVF International Confer- ence on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9630-9640. IEEE, 2021. doi: 10. 109/CCCV89922.2011 ,00951. URL https: /tt:::/dooooooooo org/ 10. II119//CCV9992..... 2021. 00951.
[17] R. E. Kalman and Others. A new approach linear filtering and prediction problems. Journal of basic Engineering, 82(1):35-4, 1960.
[18] G. Xie, H. Gao, L. Qian, B. Huang, K. Li, and J. Wang. Vehicle trajectory prediction by integrating physics- and based approaches using interactive multiple models. IEEE Trans. Ind. Electron., 65(7):5999-6008 2018. 100110..1109/11.2117727788 //doi.org// 10. T...0...191110 1119/TIIIIE... 2017 2017.2788236.
URL https:
[19] A. Houenou, P. Bonnifait, V. Cherfaoui, and W. Yao. Vehicle trajectory prediction based on motion model and maneuver recognition. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, November 3-7, 2013, pages 4363 4369, IEEE, 2013. doi: 111.0..///RO....... RAA.ROSS..01.666699833 URL https://doi: org/10..//0 1109/IRRSS 2013. 6696982. 2
[20] M. Bansal, A. Krizhevsky, and A. S. Ogale. Chauffeurnet: Learning to drive by imitat- ing the best and synthesizing the worst. In A. Bicchi, H. Kress-Gazit, and S. Hutchin- son, editors, Robotics: Science and Systems XV, University Freiburg, Freiburg im Breis- gau, Germany, June 22-26, 2019, 2019.
URL https:
//doi.org/10 10. 10w1155077/1 RSS. 2019 XV.031.
[21] Phan-Minh, F. C. Grigore, A. Boulton, O. Beijbom, and E. M. Wolff. Covernet: Multimodal behavior prediction using trajectory sets. In 2020 IEEE/CVF Conference
10

---

on Computer Vision and Pattern Recognition, CVPR 2020, Seattle. WA, USA, June 13-19, 2020, pages 14062-14077
Computer Vision Foundation IEEE, 2020. doi: URL https: https: /ptp://///eccccssseeeeee thecvf com/ content_ 10. 101100//CVP4422002220000014 Coveerrrreet...t.etttteteee
CVPR_ http2220//h/m//aaaaiiiiaaa CoverNet, Mooeer@et_Multtmmoeee Btw.atimdda_Beaaiororee_e_ wtwetrrreediiooneeeeeeee _tiigcttinnnnnnn.n Setecttory_Ssss_CPere 2020_papeeeeeee html. 2
[22] A. A. Mohamed, K. Qian, M. Elhoseiny, and C. G. Claudel. Social-stgenn: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 14412-14420. Computer Vision Foundation IEEE, URL https://openacceest thecvf. com/ Grapha.ooovouuttttaaaaaaaaaaaear Barp:iooo___ooo_aaaa_aaeeeeee htwprroro__oaaa__cP_2ee_22 Cepppon___cPPR_2__paapee .tmml." 2
2020. 1t0.001000/CVPP426000000000
[23] M. Ye, Cao, and Q Chen. TPCN: temporal point cloud networks for motion forecasting. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 11318-11327. Computer Vision Foundation IEEE, 2021. URL nttps://opps httpa//////paaccsss.sse thecvf cwm/ for www.:ooo.oorraattinnnoo.... CWWp__222______ppp___ .html. 2,
[24] H. Song, D. Luan, W. Ding, M. Y. Wang, and Q. Chen. Learning to predict vehicle trajectories with model-based planning. In A. Faust, D. Hsu, and G. Neumann, editors, Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pages 1035-1045. PMLR, 2021. URL htttss:/pproceeding /ttp///rrceeen...../////// mlr.
press/v164/ songg2a. html. 2, 6
[25] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End- to-end interpretable neural motion planner. In IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20, 2019, pages 8660-8669. Computer Vision Foundation IEEE, 2019. 2019//tmmm/eeeeee 2019.00886. URL http::/ thecvf wtt. conn/oontent_cvpe wwweeeeet_vp____e__eee End-To- Mad-To-End__aae Eed_eInddddneeeeee TarrapprttbbeeeeeuraaaMooooooooo wttpaaaaaeeeeeree11e1reeee-eeeee paper..ppppppp..... html.
101:10.110/CVPP.
[26] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. HOME: heatmap out- put for future motion estimation. In 24th IEEE International Intelligent Transportation Sys- tems Conference, ITSC 2021, Indianapolis, IN. USA, September 19-22, 2021, pages 500-507. IEEE, 2021. 10. L109/ITSCC49988.221..5666 ITSC48978. 2021. .10419496444.. 2,
URL https: /ttp::///oooooo/..///o
[27] B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cornman, K. Chen, B. Douillard, C. Lam, D. Anguelov, and B. Sapp. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. CoRR, abs/2111. 2021. URL https: // //arxiv. oww.oog//aooo att///aa//21111 2111. 14973. 2
[28] M. Ye, J. Xu, X. Xu, T. Cao, and Q. Chen. DCMS: motion forecasting with dual consistency and cullti-pseudo-targel supervision. CoRR, abs/2204.05859, 2022. dbi:10048550//rx1x.2220 05859. URL stttps:///ot org/ 10. 48550/8555/// http//22................2 05859. 2
[29] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 1422-1433 1430. IEEE Computer Society, 2015. doi: ICCV.2015.167. URL https: //tpi///iiiii org/ 10. 1119/199V/VV/ 2015.167.
[30] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors, Computer Vision ECCV 14th European Conference, Amsterdam, The Netherlands, October 1 I- 1-14, 2016, Proceedings, Part VI, volume 9910 of Lecture Notes Computer Science, pages 69 84. Springer, 2016. doi: 111.11101100100100 -00007799778---11-444-------
10. 000077788-3319-466666 3
URL https:///oi.
[31] Gidaris, Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In 6th International Conference on Learning Representations, ICLR 2018,
11

---

Vancouver, BC, Canada. April 30 May 3. 2018, Conference Track Proceedings. OpenRe- view.net, 2018 URL https: httpss://peereeeeee http://eeppeeeeeeeeeeeeeeeeeeeet
[32] M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learn- ing of visual features. In V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, editors, Computer Vision ECCV 2018 15th European Conference, Munich, Germany, Septem- ber 8-14, 2018, Proceedings, Part XIV, volume 11218 of Lecture Notes in Computer Sci- ence, pages 139-156. Springer, 2018. //doi.org/10. 1007 / 10111000000-000000000-----------
URL https:
[33] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pages 2536-2544, IEEE Computer So- ciety, 2016. doi: LL//CC/..2001...22..... URL attpp://doot org 10. 1109/CVPR.2016 278.
[34] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In B. Leibe, J. Matas, N. Sehe, and M. Welling, editors, Computer Vision ECCV 2016 14th European Con- ference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III, volume 9907 of Lecture Notes in Computer Science, pages 649-666. Springer, 2016. 1007/ 978-3-319-46487-9- URL https: portpa:/ddot :/tp:::///oooee htgg///oo.oogg//o/oo/oo 10. -000079978--311-4658888
3
[35] D. Pathak, R. B. Girshick, P. Dollár, T. Darrell, and B. Hariharan. Learning features by watch- ing objects move. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21 21-26, 2017, pages 6024-6033. IEEE Computer Society, 2017. doi: 10.1 10..110//CVPP.220 2017.6 .638. http:: / /tt.///o...ooooo org/ 10. 1119//1P99.0.//
[36] Y. You, T. Chen, Z. Wang, and Y. Shen. When does self-supervision help graph convolutional networks? In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Re- search, pages 10871-10880 PMLR, 2020. URL httpp:/pproeeedinng/ mlr, press/v119/ you20a html. 3
[37] W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang. Z. Liu, and J. Tang. -ulf-vuee learning on graphs: Deep insights and new direction. CoRR, 02s/2006.10144 2020. URL https: //axxiv. hts//abb/200066 10
[38] Y. Liu, S. Pan, M. Jin, C. Zhou, F. Xia, and P. Yu. Graph self-supervised learning: A survey. CoRR, abs/2103.0011 2021. URL https: //axxiv http://axxxxvvoooobe/e org/ wttpo//abb///11000001.1... 3
[39] W. Hu, B. Liu, J. Gomes, M. Zitnik, Liang, V. S. Pande, and J. Leskovec. Strategies for pre- training graph neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: // /pperevview. met/forum? 7 Hd=HHJIWJSFDD.. 3
[40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. von Luxburg, S. Ben- gio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017. URL https: /ptpae////cccecengs.e wttpiins...rrri.cccpppeeeeeeee wwwpcp/ppppp.////////// /hath// Awtiaaa-bbbttt..taaaeeeeaaeaaa.e html. 4
[41] E. A. Abolfathi, M. Rohani, E. Banijamali, J. Luo, and P. Poupart. Self-supervised simultane- ous multi-step prediction of road dynamics and cost map. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 8494-8503. Computer Vision Foundation / IEEE, 2021. URL https: nttps://openaccees thecvf. chm/ http:///c//222//tttm////a/ Seatrroooooeesf--oeee SwwSSeellleeeeeeeeeee Supervised Stw.rraeeedsmmmttteeeeeeeesseese MattaaaeeeouuMMMiiaatSSSSee Multi-Step.
Prediction of of Road_Rodd_dbaae DrrroadddDDmmicc___mmmm
Serd__ost___aie Mappt___aa_CVP__2211111 2wpeeeapeepp_____11_e1___eeeeeea html. 5
[42] K. Wagstaff, C. Cardie, Rogers, and S. Schrödl. Constrained k-means clustering with back- ground knowledge. In C. E. Brodley and A. P. Danyluk, editors, Proceedings of the Eighteenth International Conference on Machine Learning (ICM),2001), Williams College, Williamstown, MA, USA, June 28 July 1, 2001, pages 577-584.
Morgan Kaufmann, 2001. 5
12

---

[43] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
//arxiv. owwooogg//oooeeeooooooooooeeeo wwwgg/ab//ttee /a//aabb///11./ 6980.
[44] H. Liu, J. Z. HaoChen, A. Gaidon, and T. Ma. Self-supervised learing is more robust to dataset imbalance. CoRR, abs/2110.05555 2021. URL https pettps://arxtt wwtp/a//g/////ooooo////e///////a 05025.
[45] D. Hwang, Park, S. Kwon, K. Kim, J. Ha, and H. J. Kim. Self-supervised auxiliary learn- ing with meta-paths for heterogeneous graphs. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httph:tt//// htttt:ttp::/p//ooe p/poee/ddio/oceo/ssaa/oo www.eediiinnnn.uunn....... wwtpcccppppp/eeeeec 2atee///22////aa///
Abstract..ttreeeeeeeeeeeeeaaaeee html. 8
[46] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, Z. Yang, A. Chouard, P. Sun, Ngiam, Vasudevan, A. McCauley, Shlens, and D. Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. 2021 IEEF/CVF International Conference on Computer Vision ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9690-9699. IEEF, 2021. doi:
10. 1..10999CV449922.21...... 1.000577.
URL https: otg////01110//00444422220000
00957. 8
13


Reviews:
[{"review_id": "y0kwc9UFVBU", "reviewer": "Reviewer_dvVA", "summary": "This paper proposes to use self-supervised learning to improve trajectory prediction models. More specifically, it proposes four novel self-supervised tasks for motion forecasting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nTo my knowledge, this is the first approach to exploit self-supervised learning in motion prediction tasks. The presented results clearly show that self-supervised learning enables increasing the motion forecasting models' performance without a significant increase in the model's complexity and inference times. Overall, the paper is well written, it presents a good review of the SoA and strong experimental results.\n\nWeaknesses:\nThere are some minor improvements to improve this paper's readability.\n1 - In Fig.1., The baseline [2] does not seem to miss all the predictions. The second case seems to be a good prediction. I would recommend the authors revise this statement. Additionally, the figures are ver\n2 - Also in Fig.1., please discuss the result for Task 4, the second sub-figure counting from the top. It seems that the proposed method missed predicting a correct mode close to the GT. Why is this the case? Please discuss.\n3 - In Fig.2, the caption does not fully describe the figure. In my opinion, a figure should be self-explainable. I recommend the authors to explain the figure in the caption.\n4 - Adding an accompanying video to present the qualitative results would increase this paper's contribution.\n5 - In Eq. 2 and Eq.1. not all the variables are presented/introduced. Please make sure that all the variables are explained. The same for Eq. 4, what is L_sup?\n6 - In Sec. 4.1.3., which maneuvers are considered? How is the output classification vector defined?\n7 - In Sec. 6.3., line 251, the presented results do not support the statement \"allows it to generalize to other classes better\". Adding evaluation results of the proposed method on new datasets that were not used for training to evaluate how self-supervised learning enables improving generalization would be a significant result. The authors may consider to add such evaluation in the paper or appendix section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "To my knowledge, this is the first approach to exploit self-supervised learning in motion prediction tasks. Although the contribution is incremental, I think the presented results are significant and send a clear message to the community that self-supervised learning can also be employed to improve motion forecasting models. Therefore, I would recommend to accept this paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "No major issues. Some minor improvements are described above.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "bnqhvY5Wvc7", "reviewer": "Reviewer_b8zt", "summary": "The authors develop 4 self-supervision schemes for motion forecasting for autonomous driving. These are: lane masking, distance to intersection, maneuver classification, and success/failure classification. They show that, in general, these pretext tasks improve performance and allow them to achieve a good balance between performance and architectural complexity / inference time. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nOverall I find the work very solid. SSL is a popular approach but has not been successfully applied to the important motion forecasting problem in autonomous driving. The 4 approaches are well-motivated and explained and the results are convincing.  \n\n\n\nWeaknesses / Questions:\n\n - I find Figure 1 very hard to parse / interpret. \n - Perhaps one point to note is that most (all?) data-driven approaches to motion forecasting are self-supervised in the sense that no hand labeling is needed. What is unique about this work is the development of the pretext tasks. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall I find the work to be a nice contribution to the area of motion forecasting that will likely have high impact.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Only small formating and presentations issues listed above. ", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "9PW4FM18Jdj", "reviewer": "Reviewer_QGzU", "summary": "This paper considers motion forecasting for autonomous driving (i.e., predicting the future states of road users). The paper proposes four novel loss terms to be used in combination with a traditional loss (e.g., mean squared error of future agent positions). The four losses are: \"lane masking\" to learn lane structure, \"distance to intersection\", \"maneuver classification\", and \"success/failure\" to predict trajectory endpoints. Where traditional losses are supervised from the labeled/ground-truth data, the proposed losses are self-supervised in the sense that they leverage pseudolabels created by additional processing of the original data. The method assumes access to ground truth data of road user positions and road structure. Training with these losses increases accuracy in future motion forecasts on the Argoverse dataset; the utility of each individual loss is confirmed by an ablation study. Overall, these losses improve performance over the state of the art.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The main strength of this paper is in improving performance by crafting novel losses out of existing data. This is important for two reasons: (1) no additional hand-labeling is required, and (2) inference time does not change because these losses could be used with any model architecture.\n\n- Another key strength of this paper is that the text is fully self-contained, clear, and well-scoped. It outlines a high-level and relevant challenge in robot autonomy, proposes a method to address the challenge, and thoroughly discusses why the proposed method worked.\n\n- A third strength of this paper is in its thorough ablation study. The paper clearly summarizes exactly what performance increase each novel loss provides. Furthermore, the paper presents some well-reasoned hypotheses (and more detailed supplementary discussion) about why these losses increase performance.\n\n- Finally, from a quick look through the literature, I agree that there are no other works that thoroughly investigate self-supervision in motion forecasting. I found the following papers [R1,R2,R3].\n\n\nWeaknesses:\n\n- One weakness (potentially major) of this paper is that the results are limited to a single dataset. While the Argoverse dataset is \"large scale,\" it would be helpful to add some discussion of why this one dataset is sufficient, or why other datasets may not be feasible to use (e.g., NuScenes or Waymo).\n\n- A second weakness (minor) of this paper is that it ignores the case of motion forecasting for interaction simulation. Motion forecasting models are most useful when they can act as simulators for realistic interaction scenarios, which really requires them to generalize (which is, I realize, always a complaint with any neural network model). My understanding of the literature is that models, especially when trained on lane information and road structure, often memorize what they have seen without learning a generic model of how agents actually move around. The current losses explicitly encourage a motion forecasting model to stay near ground truth and potentially memorize the data, so could they inhibit the model's ability to generalize?\n\n- A third (very minor) weakness of the paper is that its self-supervised losses still rely on extensive existing labeled data.\n\n- Finally (major), there is no easy-to-find limitations section within the main document, though some discussion of limitations is spread throughout the results section. The supplementary material also provides a very brief discussion. It would be great if the paper could clearly discuss its limitations, and ideally refute the weaknesses I note above, within the main text.\n\nUpdate: The paper has been revised to clearly address these concerns, so I am happy to upgrade to my recommendation.\n\n\nReferences:\n\n[R1] Janjoš, F., Dolgov, M. and Zöllner, J.M., 2021, July. Self-Supervised Action-Space Prediction for Automated Driving. In 2021 IEEE Intelligent Vehicles Symposium (IV) (pp. 200-207). IEEE.\n\n[R2] Hu, P., Huang, A., Dolan, J., Held, D. and Ramanan, D., 2021. Safe local motion planning with self-supervised freespace forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12732-12741).\n\n[R3] Luo, C., Yang, X. and Yuille, A., 2021. Self-supervised pillar motion learning for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3183-3192).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper basically proposes some task-specific loss functions to improve performance on a benchmark in simulation; in that regard, though the results are nice and the text is well written, the underlying story is not very exciting.\n\nI believe the paper can be greatly improved (i.e., switch from reject to accept) by including a fuller discussion of the importance of this work in a broader context. What are the limitations? Can the proposed method be used for high-quality interaction simulation? Can the proposed method generalize to other datasets, or be demonstrated in real-world use?\n\nUpdate: The authors addressed my concerns well. I think the paper is not revolutionary, but definitely useful and important, hence my \"weak accept.\"", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Please see the \"weaknesses\" section for major issues. Minor issues as follows:\n\n104: \"BEV\" (presumably bird's eye view) should be spelled out for unfamiliar readers\n\nThe lane masking and distance-to-intersection tasks use human-supplied labels of lane structure, so are they really self-supervised? One could certainly argue that labeling lanes is less work or easier than labeling individual objects in video frames, for example. Note, the maneuver classification and success/failure classification tasks do generate pseudolabels, so I would consider them both to be self-supervised."}}, {"review_id": "1BN4tTDA4lX", "reviewer": "Reviewer_fptR", "summary": "This work leverages the potential of self-supervised learning (SSL) to boost representation learning for trajectory prediction. Four pretext tasks are proposed for joint training in trajectory prediction, including lane masking, distance to intersection learning, maneuver classification, and target hitting classification. By incorporating these SSL tasks with typical prediction frameworks, the prediction performance gains without increasing the model complexity and parameter number. The target-hitting classification shows the most prominent performance improvement among all the pretext tasks. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- This work exploits the SSL power to enhance the capability of representation learning from different aspects, which may inspire many more approaches for optimizing trajectory prediction frameworks in this direction. \n- The main structure is built upon Land-GCN. All of the pretext tasks bring some performance improvement, in which the success/failure classification gains most with a simple design. \n\nWeaknesses:\n- The reviewer admits that this is the first work to study the SSL tasks in trajectory prediction systematically, but actually, some of the pretext tasks had been proposed in prior works without being termed as SSL. This is the primary weakness of this paper. Specifically, the lane masking task is used as an auxiliary task in VectorNet[12]. The maneuver classification together with \"classification+regression\" is proposed in CS-LSTM, Multipath[3], etc., to make multimodal predictions. Considering the existing similar tries and the fact that all these SSL tasks contribute to the prediction performance, it is better to make an effective combination of them so that this work would be more solid. \n- The reviewer believes that the tradeoff between performance and model complexity seems to be overclaimed. Boosting the model performance without increasing model complexity and parameter number should be the common advantage of self-supervised learning, but not from any specific design from this work. Secondly, the reviewer wonder if \"the best accuracy-simplicity-efficiency\" is overstated, as the critical ranking metrics in Argoverse are not the best while some methods are absent in the comparison of Model Params and Inference time.  \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Considering the fact that 1) the Lane-masking task and the Maneuver classification task had been proposed in previous works, and 2) the best trade-off between complexity and efficiency are drawn from part of the methods, the reviewer believes this work still needs to improve. An ideal direction to make it solid and convincing is to combine different pretext tasks effectively. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "Some minor issues are listed below:\n- As I know CS-LSTM had used a similar set of maneuver classification for trajectory prediction in highway scenarios, but in the urban scenarios of Argoverse, is that really enough to cover the driving maneuvers in urban driving (overtaking, cut-in, nudging)? Have you tried more clusters and what about the results? \n-  It is suggested to draw Fig3.a with the performance on the Argoverse test set, which would reflect more difference. It's better to supplement more model results on Figure3.b and Figure3.c (for example, TNT results should be added) to make your conclusion convincing.  "}}]

Author rebuttal:

Decision: Accept (poster)