Bootstrap Motion Forecasting With Self-Consistent Constraints
Authors: Maosheng Ye
jiamiao xu
xunnong xu
Tengfei Wang
Tongyi Cao
Qifeng Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Motion Forecasting
Autonomous Driving
Trajectory prediction

Abstract: We present a novel framework to bootstrap Motion forecasting with self-consistent constraints (MISC). The motion forecasting task aims at predicting future trajectories of vehicles by incorporating spatial and temporal information from the past. A key design of MISC is the proposed Dual Consistency Constraints that regularize the predicted trajectories under spatial and temporal perturbation during training. Also, to model the multi-modality in motion forecasting,  we design a novel self-ensembling scheme to obtain accurate teacher targets to enforce the self-constraints with multi-modality supervision. With explicit constraints from multiple teacher targets, we observe a clear improvement in the prediction performance. Extensive experiments on the Argoverse motion forecasting benchmark show that MISC significantly outperforms the state-of-the-art methods. As the proposed strategies are general and can be easily incorporated into other motion forecasting approaches, we also demonstrate that our proposed scheme consistently improves the prediction performance of several existing methods.

Under review as a conference paper at ICLR 2023
BOOTSTRAP MOTION FORECASTING WITH SELF- CONSISTENT CONSTRAINTS
Anonymous authors Paper under double-blind review
ABSTRACT
We present a novel framework to bootstrap Motion forecasting with Self- consistent Constraints (MISC). The motion forecasting task aims predicting future trajectories of vehicles by incorporating spatial and temporal information from the past. A key design of MISC the proposed Dual Consistency Con- straints that regularize the predicted trajectories under spatial and temporal pertur- bation during training. Also, to model the multi-mode molliitodality in motion forecasting, we design novel self-ensembling scheme to obtain accurate teacher targets enforce the self-constraints with multi-modality supervision. With explicit con- straints from multiple teacher targets, we observe a clear improvement in the prediction performance. Extensive experiments on the Argoverse motion fore- casting benchmark show that MISC significantly outperforms the state-of-the- art methods. As the proposed strategies are general and can be easily incorpo- rated into other motion forecasting approaches. we also demonstrate that our pro- posed scheme consistently improves the prediction performance of several exist- ing methods.
INTRODUCTION
Motion forecasting has been crucial task for self-driving vehicles that aims at predicting the future trajectories of agents (e.g., cars, pedestrians) involved in the traffic. The predicted trajectories can further help self-driving vehicles plan their future actions and avoid potential accidents. Since the future is not deterministic, motion forecasting is intrinsically a multi-modal problem with substantial uncertainties. This implies that an ideal motion forecasting method should produce distribution of future trajectories or at least multiple most likely ones.
Due to the inherent uncertainty, motion forecasting remains challenging and unsolved yet. Recently, researchers have proposed different architectures based on various representations to encode the kinematic states and context information from HDMap order to generate feasible multi-modal
trajectories (Bansal et al. 2019 Chai et al. 2019 Gao et al. 2020 Gu et al. 2021 Liang et al.
2020 Liu et al.] 2021: [ giam et al. 2202]] Varadarajan et al. 202 Ye et al. 2021 Zeng et al,
2021 Zhao al. (2020). These methods follow traditional static training pipeline, where frames
of each scenario are split into historical frames (input) and future frames (ground truth) in a fixed pattern. Nevertheless, the prediction task is a streaming task in real-world applications, where the current state will become a historical state as time goes by, and the buffer of the historical state queue structure to make successive predicted trajectories. As result, the temporal consis- tency thus becomes a crucial requirement for the downstream tasks for fault and noise tolerance. To tackle this issue, trajectory stitching widely applied traditional planning algorithms (Fan) et al. 2018) to ensure stability along the temporal horizon. However, as the trajectory stitching op- eration non-differentiable, cannot easily incorporated into learning-band models. Though deep-learning-bact based models show unprecedented motion prediction performance compared with tra- ditional counterparts, they do not explicitly consider the temporal consistency, leading to unstable behaviors in downstream tasks such as planning.
Inspired by these phenomena, we raise question: can we explicitly enforce the consistency when training deep motion prediction model? On the one hand, the predicted trajectories should be consistent given the successive inputs along the temporal horizon, namely temporal consistency. On the other hand, the predicted trajectories should be stable and robust against small spatial noise

---

Under review as a conference paper at ICLR 2023
or disturbance, namely spatial consistency. In this work, we propose self-supervised scheme to enforce consistency constraints in both spatial and temporal domains, namely Dual Consistency Constraints. Our proposed framework, referred as MISC, significantly improves the quality and robustness of motion forecasting, without the need for extra data.
On top of the consistency, multi-modall is another core characteristic of the motion prediction task. Existing datasets (Chang et al.] 2019, Sun et al. 2020) only provide a single ground-truth trajectory for each scenario, which can not satisfy the multi-choice situations such junction scenarios. Most methods adopt the winner-takes-all (WTA) (Lee et al. 2016) or its variants (Breuer et al., [202]] Narayanan al., (02I) alleviate this situation. However, WTA tends to produce confused pre- dictions when two trajectories are very close. In contrast, our method addresses the multi mudli-moddllty issue by introducing more powerful teacher targets from self-ensembling. With self-constraint from multiple soft teacher targets, our model is more likely to be exposed to more high-quality samples, bootstrapping each modality.
Our contributions are summarized as follows,
We propose Dual Consistency Constraints enforce temporal and spatial consistency in our model, which is shown to be a general and effective way to improve the overall perfor- mance in motion forecasting.
We propose self-ensembling constraints training strategy that provides multi- modality supervision explicitly during training to enforce self-consistency with teacher targets.
We conduct extensive experiments on the Argoverse (Chang et al., 2019) motion forecast- ing benchmark and our proposed approach achieves the state-of-the-as results.
RELATED WORK
Motion Forecasting, Traditional methods (Houenou al. 2013 Schulz al.) 2018 Xie et al. 2017 Ziegler et al. 2014) for motion forecasting mainly utilize HDMap information for the prior estimation and Kalman filter (Kalman 1960) for motion states prediction. With the recent progress of deep learning on big data, more and more works have been proposed to exploit the potential of data mining in motion forecasting. Methods (Bansal al. 2019 Chai al. 2019 Duvenaud
let al. 2015; Gao et al. 2020 |Henaff et al. 2015 Liang et al. |020]] Liu et al. | Shuman et al. 2013 Song et al.) 2021: Ye al.] 2021] Zeng et al.) 2021) explore different representations, includ- ing rasterized image, graph representation, point cloud representation and transformer to generate the features for the task and predict the final output trajectories by regression or post-processing sampling. Most of these works focus on finding more effective and compact ways of feature extrac- tion on the surrounding environment (HDMap information) and agent interactions. Based on these representations, other approaches (Casas et al. 2018; Mangalam al., 2020, Song al., 1021 Zeng et al. 2021 2019 Zhao et al., (002)) try to incorporate the prior knowledge with traditional methods, which take the predefined candidate trajectories from sampling or clustering strategies as anchor trajectories. To some extent, these candidate trajectories can provide better guidance and goal coverage for the trajectories regression due to straightforward HDMap encoding. Nevertheless, this extra dependency makes the stability of models highly related to the quality of the trajectory proposals. Goal-guided approaches (Gilles et al., 2021] Gu et al.) 2021 Gilles et al. 2022) are therefore introduced to optimize goals in an end-to-end manner, paired with sampling strategies that generate the final trajectory for better coverage rate.
Consistency Regularization. Consistency Regularization has been fully studied in semi semi-supervised and self- supervsee learning. Temporally related works (Wang al. 2019 Lei et al.) 2020) ZZhou et al. 2017) have widely explored the idea of cyclic consistency. Most of the works apply pairwise matching to minimize the alignment difference through optical flow or correspondence matching to achieve temporal smoothness. Other works (Bachman et al. 2014 Földiák [99] Ouyang et al.) 2021 Sajjadi et al. 2016 Wang et al., 202I) apply consistency constraints to predictions from the same input with different transformations in order to obtain nnturbation-invaiian representations. Our work can be seen as a combination of both types of consistency to fully consider the spatial and temporal continuity in motion forecasting.
Multi-hy hotthess Learning, Motion forecasting task inherently has multi-modality due to the fu- ture uncertainties and difficulties in acquiring accurate ground-truth labels. WTA (Guzman-Rivera
2

---

Under review as a conference paper at ICLR 2023
Goal prediction
Trajectory refisement
Trajectory complction
Figure 1: The overall architecture. We utilize TPCN as feature extraction backbone to model the spatial and temporal relationship among agents and map information. A goal prediction header then used to regress the possible goal candidates; with the goal position, we apply trajectory completion to obtain full trajectories; finally, the trajectories are refined based on the output of the trajectory completion module as anchor trajectories.
et al., 2012) Sriram et al. 2019) in multi-choice learning and its variants (Makansi et al., 2019) Rupprecht et al. 2017) incorporate with better distribution estimation to improve the training con- vergence, thus a allowing more multi-modality. Some machor-based methods (Breuer et al., 2021
Chai et al., 2019 Phan-Minh et al. 2020 Zeng al. 2021) introduce pre-defined anchors based on kinematics or road graph topology to provide guidance. However, these methods only allow one target per training stage. Other methods (Breuer et al., 2021; Gu al. 202T) try to generate multi-target for supervision with heavy handerafted optimizations. We propose Teacher-Target- Constraints approach to provide more precise trajectory teacher labels by leveraging the power of seff-ensembling (Lee et al.) 2013; Zheng et al. 202I). Multiple targets are explicitly provided to each agent to better model the mult- moll-m-odllty..
3 APPROACH
The overall architecture of MISC consists of three parts. We first utilize a joint spatial and temporal learning framework TPCN (Ye et al. (02T) to extract pointwise features. Based on these features, we decouple the trajectory prediction problem as two-stage regression task. The first stage performs goal prediction and completes the trajectory with the goal position guidance. The second stage takes the output of the first stage as anchor trajectories for refinement. 2) To train our MISC, propose Dual Consistency Constraints regularize the predictions both spatially and temporally a streaming task view. 3) We generate more accurate teacher targets by self- ensembling provide self-consistent Teacher Targets Constraints in Sec. 3.3
3.1 ARCHITECTURE
Recently, TPCN (Ye et al. 2021 has gained popularity in this task due to its flexibility for joint spatial-temporal learning and scalability adopt more techniques from point cloud learning. Con- sidering its inferiority in representing future uncertainty, we extend TPCN with a two-stage manner through goal position prediction for more accurate waypoints prediction as our baseline. The whole network is shown in Fig. [
Feature Extraction: TPCN utilizes dual- reprreseeeennnooonn point cloud learning techniques with multi- interval temporal learning to model the spatial and temporal relationship. All the historical trajecto- ries of input agents and map information are based on pointwise representation (P1, P2, P. ). where p, is the i-th point with N points in total, and then go through nulti-representationn learning framework to generate pointwise features P E RN xC, where
the channel number.
Goal Prediction: With the pointwise features from the backbone, we also adopt the popular goal- based ideas Giiles et al. 2021 Gu et al.) 2021 Zhao et al. 2020) to find the optimal planning policy. Specifically,
first gather all corresponding pointwise agent features and then sum over features to get the agent instance feature ó E R xC To generate K goal position prediction = (G* ( o 11 k < K),
simple MLP layer: G MTPP(e). Instead of relying
3

---

Under review as a conference paper at ICLR 2023
Temprrll Consistency
LOSS
history of Frims
history
eillistorical Trajectory
Predddicted
Consinn Lency Loss
The overall idea of the temporal consistency. the training stage, we first generate output prediction trajectory points normal for each given scenario. Then we slide the imput with step in order to introduce the streaming nature to generate the consecutive output trajectory points. The proposed temporal consistency requires the overlap between these two outputs be consistent
Figure 2:
on heavy sampling strategies like previous goal-based methods, our method avoids generating extra proposals, which may lead to large computation overhead.
With the predicted goal positions, we need to complete each trajectory conditioned on these goals. We propose simple trajectory completion module to generate full trajectories (Treyy 1 < k < K} with single MLP layer as follows:
Trajectory Completion:
r { v² (a vz). (+) MLP(ccnnaa(0. G*)).
Trajectory Refinement: Inspired by Faster-RCNN (Ren et al. 2015) and Cascade-RCNN CCi & Vasconcelos) 2018), we use the output trajectories from the Trajectory Completion anchor trajec- tories refine trajectories and predict the corresponding possibility of each trajectory. In particular, the input of the trajectory refinement module will be whole trajectory with agent historical way- points Thietory- With residual block followed by linear layer Reg and Cls respectively, regress the delta offset to the first stage outputs Arr.g = (eg(treesThistorr)) and corresponding scores Tels =
fc S k < K} respectively, where Tale = ClasTregg Thistory). The final output trajectories will be Treg' Arreg Treg-
3.2 DUAL CONSISTENCY CONSTRAINTS
Consistency regularization has been proved as effective self-constraint that helps improve robust- ness against disturbances. Therefore, we propose Dual Consistency Constraints in both spatial and temporal domains align predicted trajectories for continuity and stability.
TEMPORAL CONSISTENCY
In motion forecasting, since each training scenario contains multiple successive frames within fixed temporal chunk, it is reasonable to assume that any two overlapping chunks of input data with small time-shift should produce consistent results. The motion forecasting task aims to predict K possible trajectories with T time steps for one scenario, given M frames historical information. Suppose the information each history frame is J, where S S M and the k-tth output future trajectories are {<x/ yt) [M < S M + T3. We first apply time step shift for the input for tem- poral consistency. Therefore, the input history frames information will be [I,[1 + S i < M + s} and then we apply the same network for the shifted history information with surrounding HDMap information to generate the k-th output trajectories { {aat M + < < M + T). When is small, the driving intentions or behavior keeps stable in a short period. Since both trajectories have T overlapping waypoints, they should be close possible and share consensus. Thus, we can construct self-constraints for a single scenario input due to the streaming property of the input data. Fig. demonstrates the overall idea of the temporal consistency constraint.
Trajectory Matching: Since we predict future trajectories to deal with the multi multi-modality, it is crucial to consider the trajectory matching relationship between original predictions and time-shifted predictions when applying the temporal consistency alignment. For a matching problem, the metric similarity criteria and matching strategies will be two key factors. Several ways be used to
4

---

Under review as a conference paper at ICLR 2023
measure the difference between trajectories, such Average Displacement Error (ADE) and Final Displacement Error (FDE). We utilize FDE as the criteria since the last position error can partially reflect the similarity with less bias from averaging compared with ADE.
Matching Strategy: There are roughly four ways used for matching, namely forward matching, backward matching. bidirectional matching, and Hungarian matching. Forward matching takes one trajectory in the current frame and finds its corresponding trajectory in the next frame with the least cost or maximum similarity. Backward matching is the reverse way compared to forward matching. Furtherly, bidirectional matching consists of both forward and backward matching, which considers the dual relationship. Hungarian matching is linear optimal matching solution based on linear assignment. Forward and backward matching only considers the one-way situation, which sensitive to noise and unstable. Hungarian matching has a high requirement for cost function choice. Based on these observations, we choose bidirectional matching as our strategy. We also show its advantages over the other approaches in Sec. 4.3
After obtaining the optimal matching pairs (m,,, ) < k S KJ, can compute the consistency
constraint by a simple smooth L1 loss (Ren et al. 2015) LHuber:
K
T
Lrcmp
Luueer
(aaberr(a?)) (aa)" th ;9t
mh (x me 1-s1 v-e))
(2)
SPATIAL CONSISTENCY
Since our MISC is two-stage framework, the second stage mainly aims for trajectory refinement. will be more convenient to add spatial permutation the second stage with less computational cost. First, we apply spatial permutation function Z. including flipping and random noise, to the trajectories from the first stage. The refinement module will process these augmented inputs and generate the offset to the ground truth and classification scores. Under the small spatial permutation and disturbance, we assume that the outputs of the network should also be self-consistent, meaning that the outputs have strong stability or tolerance to noise. Compared with data augmentation, is the explicit regularization. Then the spatial consistency constraint Lxpa is as follows:
Lespa L Trey: 7 (Regl (Tregg Thistory))).
Then the total loss for Dual Consistency Constraints module will be Loons Lapa + Lteepp.
3.3 REACHER-TARGEE CONSTRAINTS
Existing datasets (Chang et al. 2019 Sun et al. 2020) only provide a single ground-truth trajectory for the target agent, which is to be predicted in one scenario. In order to encourage the multi- modality of models, the winner-takes-all (WTA) strategy is commonly used to prevent the model from collapsing into a single domain. However, the WTA training strategy suffers from instabil- ity associated with network initialization. Some other approaches (Breuer et al.] 2021; Narayanan al. 2021) introduce robust estimation methods to select better hypotheses. To
extent, these methods can only implicitly model the multi-modality. Some other approaches (Breuer et al. 2021 Zhao al. 2020) generate several possible future trajectories based on the kinematics model and road graph topology. DenseTNT (Gu et al. (021) only uses teacher labels for goal set prediction through hill-climbing algorithm. These optimization methods tend to impose strict constraints and handcrafted prior knowledge, resulting in inaccurate teacher-target targets and inferior performance. In contrast, our approach aims generate more accurate teacher targets to provide explicit multi- modality supervision through self-ensembling to leverage the power of semi- semi-supervised learning.
Seacher-Targe Generation. The key part of our approach lies in generating more accurate teacher labels for each agent. However, it is straightforward to apply model ensembling techniques (He et al.) 2020 Laine & Aila| 2016 Tarvainen Valpola| 2017) to obtain more powerful predictions. Com- pared with previous works (Breuer et al. 2021 Chai et al. 2019 Zhao et al. 2020), we do not rely on handcrafted anchor trajectory sampling, which is based on inaccurate prior knowledge, including better finetune the predictions and reduce the gradient variance for better training convergence. As suggested in works (Dietterich) 2000
motion estimation. Meanwhile, soft targets from ensembling
Opitz & Maclin 1999). the prediction error decreases when the ensemble approach is used
5

---

Under review as a conference paper at ICLR 2023
cluster
6:1" Model's Trajectories 0:24 Hodel's Trafectorios
Bt Trajectories
0:Historical Trafectories
Cluster results
Cluster2
Figure 3: The overall procedure for the teacher-target generation. We obtain multiple predictions from outputs of different models for the target agents in each scenario; then we apply K-means clustering algorithm ensemble the trajectories
the model is diverse enough. Therefore, apply k-means algorithm MacQueen et al. 1967) to the predicted trajectories that are collected within different training procedures (for example, launched with different seeds of random number generators, optimized with different learning rates, etc.) of MISC without Teacher-Target Constraints to generate trajectories with corresponding scores for each scenario. Fig. 3 shows the overall process of our approach. Then with the original ground-trum label, we will formulate J + target trajectories as follows:
Tconf = {co: CI,...,ch).
(4)
Agt; tats) Tgt (1 1111)) (x2 .12 )
(#T
tgty
Egt
(5)
where Tége the j-th trajectory with score el, among J target trajectories. To simplify the notation, Tit is the ground-truth trajectory with co set to 1.
3.4 LEARNING
The total supervision of our MISC can be decoupled into several parts, as described in previous sections. For the regression and classification parts, loop over all the possible J + 1 targets Tigt- For each target Ti with confidence = apply WTA strategy as described in Sec. 3.3
Suppose -th trajectory from trajectory refinement output Treg' is the best trajectory which has the maximum similarity with target Tigt, the classification loss and regression loss are defined as
K
Cdse
Foo T CHuber ch"
(6)
K
km
Glog
Poon (la) Mi (x (2 W1: (.
tgts
T
For classification loss design, we adopt the displacement prediction idea from TPCN YYe et al. 2021 alleviate the hard assignment phenomenon. As for converting the displacement into probability, we use the standard softmin function to distribute the scores. Since we have trajectory completion and refinement modules, the regression loss will be Lreg Σi=o (Cteg + L'sreg), where La is the regression loss for the refinement module. The final loss is L. freg + Lets Lrons
4 EXPERIMENTS
We conduct experiments the Argoverse dataset (Chang et al. 2019 of the largest publicly available motion forecasting datasets. We compare our MISC with other state-of-the methods. Furthermore, provide ablation studies to evaluate the effectiveness and generalization ability of each proposed module and design experiments for some hyperparameter choices.

---

Under review as a conference paper at ICLR 2023
Table 1:
The detailed results of MISC and other top-performing approaches the Argoverse test set. And b-FDEş is the abbreviation of brier-minFDE,
Models
minADE;
minFDF;
MR;
minADFa
minFDFg
MRg
b-FDFe
Jean Chang et al
2019
Mereat et al.
2020
1.74
4.24
0.68
0.98
1.42
0.13
2.12
LaneConv
[Liang ctall
2020
1.71
3.78
0.59
0.87
1,36
0.16
2.05
LaneRCNN
Zeny
1.45
man mmn:Transorrmerr [Liu el al.
2021
1.77
4.00
0.62
0.87
1.34
0.15
2.03
SeemeTransoomer Ngiam
2021
1.81
4.06
0.59
0.80
1.23
0.126
1.88
TNT Zhao al.
2020
1.77
3,91
0.59
0.94
1.54
0.13
2.14
DeTscTNT
2021
1.68
3.63
0.58
1.28
0.125
1.97
PRIME
Song ot al
2021
1.91
3.82
0.59
1.22
1.55
0.12
2.09
TPCN
Ye et
2021)
1.58
3.49
0.56
0.88
1.24
0.13
1.92
HOME Gilles et al.
2021)
1.70
3.68
0.57
0.89
1.29
0.08
1.86
MuliiPath++
Varadarajan
2021
1.623
3.614
0.564
0.790
1,214
0.13
1,793
Ours
1.135
1.756
4.1 EXPERIMENTAL SETUP
Dataset. Argoverse (Chang et al. 2019) is currently one of the most popular motion forecasting datasets. It provides more than 300K scenarios with rich HDMap information. For each scenario, objects are divided into three types: agent, AV and others, where "agent" is the object to be predicted. Moreover, each scenario contains 50 frames sampled at 10 Hz, meaning that the time interval be- tween successive frames is 0. The whole dataset is split into training, validation, and test sets, with 205942, 39472, and 78143 sequences, respectively.
Metrics. We use the standard evaluation metrics, including ADE and FDE. ADE is defined as the average displacement error between ground-truth trajectories and predicted trajectories over all time steps. FDE is defined displacement error between ground-truth trajectories and predicted trajectories at the last time step. We predict K candidate trajectories for each scenario and calculate the metrics with the ground truth labels. Accordingly, minADE and minFDE minimum ADE and FDE over the top K predictions. Moreover, miss rate (MR) is also considered, defined as the percentage of the hest-predicted trajectories whose FDE is within threshold (2m). Brier-minFDD the minFDE plus (1 1--p) p)² where is the corresponding trajectory probability. Metrics for K K= and K 6 used in our experiments. Note that Brier-minFDEs the ranking metric.
Experimental Details. We apply some data augmentation, including random flipping with a prob- ability 0.5 and global random scaling with the scaling ratio between [0.8, 1.25) during the training stage. As for model settings, the time shift for the temporal consistency constraint is set to 1. We adopt K to generate trajectories and use J teacher targets for each scenario. Furthermore, we choose bidirectional-matee ensembling due to computation resource limits. For more training details, we have included them in the supplementary materials.
for temporal consistency constraint. We finally use 10 models for
4.2 EXPERIMENTAL RESULTS
Argoverse Leaderboard Results. We provide detailed quantitative results of our MISC on the Argoverse test set well as public state-of-the-art methods in Tab. [ Compared with previous methods, our MISC improves all the evaluation metrics except MR, by large margin. Furtherly, since the proposed modules all general training components, other existing motion forecasting models can also benefit greatly from these strategies.
Qualitative Results. We also present some qualitative results on the Argoverse validation set in Fig. 4 Compared with results without consistency, the Dual Consistency Constraints improve both the quality and smoothness of the predicted trajectories significantly, resulting in more feasible and stable results despite the input noise.
4.3 ABLATION STUDIES
Component Study. As shown in Tab. 2 we conduct an ablation study for our MISC on the Argov- erse validation set to evaluate the effectiveness of each proposed component. We adopt TPCN (Ye
7

---

Under review as a conference paper at ICLR 2023
W/o Consistency
W/ Consistency
Figure 4: The past trajectory is in yellow, the predicted trajectory in green, and the ground truth in red. The top row of the figure shows the results without consistency, while the bottom row shows the results with consistency
Table 2:
Ablation study results of modules. Goal refers to Trajectory completion with goal predic- tion. "Ref." is the trajectory refinement module, and the "Temp." temporal consistency. TTC refers to Teacher-Targel Constraints during training
Architecture
Consistency
K=l
K=6
Temp. Spatial
TTC
minFDE
minFDE
1.34
2.95
0.73
1,15
1.33
2.91
0.725
1.10
2.89
0.71
1.07
1.24
2,70
0.662
0.981
1.22
2.67
0.653
0.954
1.26
2.77
0.69
1.01
2.60
0.929
al. 2021) the baseline shown in the first row of Tab. and add the proposed components pro- gressively. The architecture modifications from the goal set prediction and trajectory refinement module show their promising improvements of about 2%. Dual consistency Constraints have the largest improvements more than 5% among all the evaluation metrics. Especially for minFDE, temporal consistency can optimize 20 em, indicating the temporal constraints can improve both final position and trajectory probability prediction. Compared with temporal consistency, spatial consis- tency has less effect on models since we only enforce this constraint in the trajectory refinement stage. Finally, the Teacher-Taare -aahher-Target Constraints significantly increases performance, manifesting its effectiveness in helping training convergence.
Temporal Consistency Factors. We study the factors in the matching problems, including similar- ity and matching strategies. As shown Tab, 3 both Hungarian and Bidirectional matching show their advantages over the single direction matching. Although Hungarian matching can ensure the
Table Ablation study on matching factor for temporal consistency, In this experiment, we remove the Teacher- Target Constraints to fairly study the effect
Matching Strategy
Similarity
K=1
K=6
minFDE
minADE
Forward
ADE
1.25
2.70
0.46
0.670
0.982
0.089
FDE
1.24
2.69
0.46
0.668
0.980
0.088
ADE
0.982
Backward
FDE
1.24
2.68
0.46
0.667
0.958
0.085
Bidirectional
ADE
1.22
2.67
0.446
0.666
0.972
0.087
FDE
0.445
0.653
0.954
0.084
Hungarian
ADE
1.24
2.69
0.46
0.668
0.975
0.088
FDE
2.69
0.45
0.660
0.968
0.088
8

---

Under review as a conference paper at ICLR 2023
Table 4: Ablation study results the teacher target number
Teacher Target Num
K=1
K=6
minADE
minFDE
MR
minADE
minFDE
MR
0:70
103
0.104
1.28
2,80
0.48
0.69
102
0.10
1.26
2.77
0.47
0.69
1.01
0.09
Table 5: Ablation study of consistency constraints and Teacher Target Constraints on different state- of-the-art methods on Argoverse validation set. Performance for methods without constraints is obtained from corresponding papers or our reproduction
Mathod
Consistency
TTC
K=1
K=6
minFDE
minF"DE
1.35
2.37
0,71
L,88
LaneOCN Liang et al.
2020
1.29
2.80
0.68
1,00
1.30
2.88
0.69
1.04
x
x
1.34
2.95
0.73
1.13
TPCN
Ye
1.30
2.86
0.69
1.09
x
1.38
3.03
0.71
1.15
mnTrrasffrmer Liu
2021
2.83
0.68
1.02
x
1.29
2.80
0.68
1.04
1.36
2.94
0.73
1.03
DenseTNT Gu al.
2021
1.25
2.81
0.68
0.98
X
1.30
2.82
0.69
1.00
one-to-one matching relationship, it is sensitive to the similarity metric and numerical precision, stable in the early training stage. In contrast, bidirectional matching with the
both of which FDE similarity metric nearly achieves the best results across all the evaluation metrics. Meanwhile, we also conduct experiments to find the best time-shift value s in the temporal consistency, The details can be found in appendix 6
Number of Teacher Targets. As shown in Tab. more teacher targets could bring better perfor- mance. Compared with J = 1, 6 teacher targets bring an extra nearly 1% improvements. However, the marginal improvement decreases significantly we finally choose J 6.
4.4 GENERALIZATION CAPABILITY
To verify the generalization capability of Dual Consistency Constraints and Teacher Targets Con- straints, we also apply them to different models with state-of-the-ar performance to show that they can be plugin-in training schemes.
Consistency Component. As shown in Tab. 5 our dual consistency constraints can effectively improve the performance of models regardless of their representations through the training phase. There is noticeable improvement of over 5% on every metric, especially for minFDE
Teacher Target. Teacher- Target Constraints is another general training trick that can be widely used in other frameworks. In Tab. (5) we also verify its effectiveness on other public methods. Methods with Teacher-Target Constraints have nearly
3% improvement in all metries. For the origi- nal DenseTNT (Gu et al., 202I), we replace its original handcrafted optimization for teacher goal targets with our self-ensembling teacher targets. This strategy brings over 5% increase in per- formance, demonstrating the better quality of the self-ensembling teacher targets than handcrafted optimizations and estimation.
CONCLUSION
In this work, propose MISC, effective architecture for the motion forecasting task that explic- itly models the multi-modality. We also impose dual consistency regularization on both spatial and temporal domains to leverage the potential of self-supervision, which has been ignored by previous efforts. Besides, we explicitly model the multi-modality by providing supervision with powerful self-ensembling techniques. Experimental results on the Argoverse motion forecasting dataset show the effectiveness of our approach and generalization capability to other methods.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We use the publicly available Argoverse Dataset (Chang et 2019) available at /tttts:////tt argoverse org/avi .. html #forecastinn link Dataset preprocessing is shown in 4.1
Training process is in Appendix A.2 And the model architecture is illustrated in the Sec. 3. Appendix A.1
and
REFERENCES
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensenie geeudo-nseemble.. Advances in neural information processing systems, 27:3365-3372 2014.
Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet. Systems XV, 2019.
In Robotics: Science and
Antonia Breuer, Quy Le Xuan, Jan-Aike Termöhlen, Silviu Homoceanu, and Tim Fingscheidt. Quo vadis? meaningful multiple trajectory hypotheses prediction in autonomous driving. 202/ IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 637-644. IEEE, 2021.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6154-6162, 2018.
Delving into high quality object detection. In
Sergio Casas, Wenjie Luo, and Raquel Urtasun. Intentnet: Learning to predict intention from raw sensor data. Conference on Robot Learning, pp. 947-956, 2018.
Yuning Chai, Benjamin Sapp. Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple prob- abilistic anchor trajectory hypotheses for behavior prediction. arXiv preprint urXiv:/910.05: 2019.
Ming-Fam Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hart- nett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 8748-8757 2019.
Thomas G Dietterich. Ensemble methods in machine learning, In International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Gure and Ryan Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances in neural information processing systems, pp. 2224-2232, 2015.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 9710-9719, 2021.
Haoyang Fan, Fan Zhu, Changchun Liu, Liangliang Zhang, Li Zhuang, Dong Li, Weicheng Zhu, Jiangtao Hu, Hongye Li, and Qi Kong. Baidu apollo em motion planner. arXiv preprint arXiv: 2018.
Liangji Fang, Qinhong Jiang, Jianping Shi, and Bolei Zhou. Tpnet: Trajectory proposal network for motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6797-6806, 2020.
Peter Földiák. Learning invariance from transformation sequences. Neural computation, 3(2) 200, 1991.
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectorne:: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11525-11553 11533, 2020.
10

---

Under review as a conference paper at ICLR 2023
Thomas Gilles, Stefano Sabatini, Dzmitry Tsiskkou, Bogdan Stanciulescu, and Fabien Moutarde. Home: Heatmap output for future motion estimation. arXiv preprint arXiv:2 10968, 2021.
Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Gohome: Graph-oriented heatmap output for future motion estimation. In 2022 International Conference on Robotics and Automation (ICRA), pp. 9107-9114. IEEE, 2022.
Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15303- 15312, 2021.
Abner Guzman RiveannRivera, Dhruv Batra, and Pushmeet Kohli. Multiple choice learning: Learning to produce multiple structured outputs. Advances in neural information processing systems, 25, 2012.
Chenhang He, Hui Zeng. Jianqiang Huang, Kian-Sheng Hua, and Lei Zhang. Structure aware single- stage 3d object detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11873-11882, 2020.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arXiv preprint 150.v:5506.05100 163,, 2015.
Adam Houenou, Philippe Bonnifait, Véronique Cherfaoui, and Wen Yao. Vehicle trajectory predic- tion based on motion model and maneuver recognition. In 2013 IEEE/RSJ international confer- ence on intelligent robots and systems, pp. 4363-4369. IEEE, 2013.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. J. Basic Eng, 82(1):35-45, 1960.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242, 2016.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges representation learning, ICML, volume 3, pp. 896, 2013.
Stefan Lee, Senthil Purushwalkam Shiva Prakash, Michael Cogswell, Viresh Ranjan, David Cran- dall, and Dhruv Batra. Stochastic multiple choice learning for training diverse deep ensembles. In Advances in Neural Information Processing Systems, pp. 2119-2127, 2016.
Chenyang Lei, Yazhou Xing, and Qifeng Chen. Blind video temporal consistency via deep video prior. Advances in Neural Information Processing Systems, 33, 2020.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In Proceedings the European Conference on Computer Vision (ECCV), pp. 541-556, 2020.
Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction with stacked transformers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 7577-7586, 2021.
Qiujing Lu, Weiqiao Han, Jeffrey Ling, Minfa Wang, Haoyu Chen, Balakrishnan Varadarajan, and Paul Covington. Kemp: Keyframe-based hicrarchical end-to-end deep model for long-term tra- jectory prediction. arXiv preprint 2xXi:22055.06244 2022.
James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statisties and probability, volume 1, pp. 281-297. Oakland, CA, USA, 1967.
Osama Makansi, Eddy Ilg, Ozgun Cicek, and Thomas Brox. Overcoming limitations of mixture den- sity networks: A sampling and fitting framework for multimodal future prediction. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 7144-7153, 2019.
11

---

Under review as a conference paper at ICLR 2023
Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It not the journey but the destination: Endpoint conditioned trajec- tory prediction. arXiv preprint arXhv:2004.02025, 2020.
Jean Mercat, Thomas Gilles, Nicole El Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Pita Gil. Multi-head attention for multi-modal joint vehicle motion forecasting. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9638-9644 IEEE, 2020.
Sriram Narayanan, Ramin Moslemi, Francesco Pittaluga, Buyu Liu, and Manmohan Chandraker. Divide-and-conquer for lane- diverse trajectory prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 55799-15805, 2021.
Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified multi-task model for behavior prediction and planning. arXiv preprint arXiv:2106.08847. 2021.
David Opitz and Richard Maclin. Popular ensemble methods: An empirical study. Journal of artificial intelligence reseurch, 11:169-198, 1999.
Hao Ouyang, Tengfei Wang, and Qifeng Chen. Internal video inpainting by implicit long-range propagation. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pp. 14579-14588, October 2021.
Stefano Pellegrini, Andreas Ess, and Luc Van Gool. Improving data association by joint modeling of pedestrian trajectories and groupings. European conference on computer vision, pp. 452 465. Springer, 2010.
Tung Phan-Minh, Elena Corina Grigore, Freddy A Boulton, Oscar Beijbom, and Eric M Wolff. Covemet: Multimodal behavior prediction using trajectory sets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14074-14083, 2020.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical
feature learning on point sets in metric space. In Advances in neural information processing systems, pp. 5099-5108, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: towards real-time object detection with region proposal networks. In International Conference on Neural Information Processing Systems, 2015.
Christian Rupprecht, Iro Laina, Robert DiPietro, Maximilian Baust, Federico Tombari, Nassir Navab, and Gregory D Hager. Learning in uncertain world: Representing ambiguity through multiple hypotheses. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3591-3600, 2017.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. Advances in neural information processing systems, 29: 19:116 1613-117,, 2016.
Jens Schulz, Constantin Hubmann, Julian Löchner, and Darius Burschka. Interaction-aware proba- bilistic behavior prediction in urban environments. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3999-4006. IEEE, 2018
David Shuman, Sunil K Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. The emerging field of signal processing graphs: Extending high-dimensional data analysis to net- works and other irregular domains. IEEE signal processing magazine, 30(3):83-98, 2013.
Haoran Song, Di Luan, Wenchao Ding, Michael Y Wang, and Qifeng Chen. Learning to predict vehicle trajectories with model-basee based planning. In Conference on Robot Learning, pp. 1035- 1045. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
NN Sriram, Gourav Kumar, Abhay Singh, M Siva Karthik, Saket Saurav, Brojeshwar Bhowmickk and K Madhava Krishna. A hierarchical network for diverse trajectory proposals. In 2019 JEEE Intelligent Vehicles Symposium (IV), pp. 689. IEEE, 2019.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp.
2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Mul- tipath++: Efficient information fusion and trajectory aggregation for behavior prediction. arXiv preprint arXiv:2111.14911 2021.
Tengfei Wang, Jiaxin Xie, Wenxiu Sun, Qiong Yan, and Qifeng Chen. Dual-camera super-resolution with aligned attention modules. Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 2001-2010, October 2021.
Xiaolong Wang. Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2566-2576, 2019.
Guotao Xie, Hongbo Gao, Lijun Qian, Bin Huang. Keqiang Li, and Jianqiang Wang. Vehicle trajec- tory prediction by integrating physics-and maneuver-based approaches using interactive multiple models. IEEE Transactions on Industrial Electronics, 65(7):5999 6000999990000---00011011098888 2017.
Maosheng Yc, Tongyi Cao, and Qifeng Chen. Tpen: Temporal point cloud networks for motion forecasting, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 1318-11327, 2021.
Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urta- sun. End-to-end interpretable neural motion planner. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8660-8669, 2019.
Wenyuan Zeng, Ming Liang, Renjie Liao, and Raquel Urtasun. Lanerenn: Distributed representa- tions for graph-centric motion forecasting. arXiv preprint arXiv: 101.06653, 2021.
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp. Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, al. Tnt: Target-driven trajectory prediction. arXiv preprint arXiv:2008.082 2020.
Wu Zheng, Weiliang Tang, Li Jiang, and Chi-Wing Fu. Se-ssd: Self-ensembling single-stage object detector from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14494-15503 2021.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1851-1858, 2017.
Julius Ziegler, Philipp Bender, Markus Schreiber, Henning Lategahn, Tobias Strauss, Christoph Stiller, Thao Dang, Uwe Franke, Nils Appenrodt, Christoph G Keller, et al. Making bertha drive-an an autonomous journey on historic route. IEEE Intelligent transportation systems mag- azine, 6(2):8-20, 2014.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 MODEL DETAILS
We provide the detailed network architecture of our MISC in Fig. 5 We use TPCN (Ye et al. as our backbone. The feature extraction consists of 4 spatial modules and 4 dynamic temporal learn- ing layers same as TPCN. Before the prediction header, we calculate the mean features and remove map instances features. For the spatial module, the point representation utilizes PointNet++ (Qi al., 2017) with neighborhood radius of [0.2m, 0.4m, 0.8mJ, while the voxel representation uses Sparse BottleNeck. We use all the points in this process without any sampling. More details about backbone can be found in TPCN (Ye et al. 2021
2021
A.2 TRAINING DETAILS
We train MISC for 50 epochs using batch size of 32 with Adam (Kingma & Ba 2014) optimizer with an initial learning rate of 0.001, which is decayed every 15 epochs in a ratio of 0.1.
History 1x20x2
Goal Kx2
Trajectory Refinement (MLP) Kx30<2
Goal Prediction Header
Map Instances
Agent Instances 128
Trajectory completion (MIPP Kx30x2
Nx 128
Mean over Instance
Mx 128
Ourput Kx30x2
Instance Pooing
Dynamic Temporal Learning
Multi-interval Learinng (2.4, 8,16)
Spatial Module
Dynamic Temporal Learning
concatenate
Voxel
Spatial Module
PointNet++ r0.iux:0.2, 0.4, 0.8)
Sparse BottleNock
Input: M x
Poinz To Vaxel
TPCN feature extraction
Figure 5: Detailed illustration of our MISC.
A.3 ABLATION STUDY
A.3.1 TEMPORAL CONSISTENCY
Meanwhile, also conduct experiments to find the best time-shift value in the temporal consis- tency. As shown Tab. 6, choosing time shift s = 1 has already achieved decent performance, with five out of six metrics ranking the first. Further increasing the will not bring much performance gain since the driving behavior could change lot with large s.
We use the average L2 distance among all predicted trajectory waypoints to measure the temporal consistency. As shown in Fig. D our model without temporal consistency will have large inconsis- tency even though the time shift small, which may lead to unstable behavior for the downstream
14

---

Under review as a conference paper at ICLR 2023
Table 6: Ablation study results of time-shift used by temporal consistency
Time shift
K=1
K=6
minADE
minFDE
MR
minADE
minFDE
MR
1.22
2.67
0.444
0.653
0.954
0.084
1.23
2.67
0.444
0.654
0.958
0.082
1.25
2.69
0.445
0.662
0.964
0.085
1.25
2.70
0.446
0.667
0.969
0.086
Figure 6: The L. distance in our model varies with the time shift 8.
consistency
w/ consistencyy
0.9
(mn)
0.5
0.3
Time shill
task such as planning. With temporal consistency constraints, there a significant improvement for the 1.2 distance divergence, demonstrating the effectiveness of our method.
A.3.2 SPATIAL CONSISTENCY
Furthermore, also measure the spatial inconsistency against flipping and Gaussian noise with zero mean and standard deviation of 15m. The average spatial inconsistency will be 19.3cm, while the number decreases to 10.2cm with our spatial consistency constraint.
A.3.3 COMPONENT STUDY
We provide controlled experiment to verify the effectiveness of the proposed method when turning both Dual Consistency Constraints and Teacher-Targe Constraints on the same time shown in Tab. 7 With both modules on, the performance of all the methods benefits lot, about nearly 7%, demonstrating the generalization capability and effectiveness of our approach. It also shows that these two modules can be independently helpful.
A.3.4 RESULTS ON WAYMO DATASET
Table 8: Quantitative results on the validation set the Waymo Open dataset motion prediction task.
Method
minADEL
minFDF]
Miss Rate.
mAP
Baseline (Etinger et al. 2021)
0.675
1.349
0.183
0.268
KEMP (Lu et al. 2022)
0.5691
1.1993
0.1458
0,394
SceneTransformer (Ngiam al.
2021)
0.613
1.22
0.157
0.284
Ours
0.54
1.11
0.128
0.41
15

---

Under review as a conference paper at ICLR 2023
Table 7: Results of consistency constraints and Teacher-Target Constraints (TTC) supervision
different state-of-the-art methods on Argoverse validation set. Performance for methods without consistency constraints is obtained from corresponding papers or our reproduction.
K=1
K=6
Method
Consistency TTC
minADE
minFDE
minADE minFDE
LaneGCN (Liang et
2020
x
1.35
2.97
0.71
1.08
1.25
2.71
0.66
0.98
1.34
2.95
0.73
1.15
TPCN (Ye et al.
2021
1.23
2.70
0.67
1.00
1.38
3.03
0.71
1.15
mmTransformer (Liu al.
2021
1.25
2.77
0.67
0.99
1.36
2.94
0.73
1.05
Den Guu et al.
2021
1.23
2.71
0.66
0.95
We provide some quantitative results on the validation set of the Waymo Open dataset motion pre- diction task (Ettinger al.) 202T), b. shown in Tab, g Compared with KEMP (Lu et al., 2022) and Scene Transformmr (Ngiam et al. (021). we also achieve very promising results and show compara- ble improvement, demonstrating the effectiveness of our approach.
A.3.5 ABLATION STUDY ON WAYMO DATASET
Since the scale and object types in waymo dataset and argoverse dataset are different, we conduct experiments to find the best time shift for each class on Waymo Dataset. As shown Tab. best time shift for vehicle and cyclist will be 1, while the value will be 2 for pedestrian class. To achieve best performance for the overall metrics, we finally choose = in our setting.
Table 9: Ablation study results of time-shift used by temporal consistency Waymo Open Motion Dataset motion prediction
minADE
minFDE |
MR J
mAP
Time
veh
ped
eye
veh
ped
eyc
veh
ped
cyc
veh
ped
cyc
0.622
0.34
0.654
1.262
0.663
1.294
0.135
0.085
0.197
0.285
0.252
0.214
0.625
0.33
0.660
1.263
0.662
1.296
0.135
0.084
0.200
0.283
0.252
0.215
3
0.632
0.34
0.667
1.274
0.666
1.302
0.136
0.086
0.198
0.290
0.254
0.217
0.634
0.33
0.672
1.278
0.670
1.303
0.137
0.086
0.199
0.288
0.253
0.217
A.4 RESULTS ON ETH DATASET
To verify the temporal consistency on the low framerate dataset, we conduct experiments the ETH Pellegrini et al. 2010) dataset. We report the ADE and FDE metrics for lpred = and 12 respectively. Following the common settings used by previous methods Fang et al. 2020). we use K = and K = 20. As shown in Tab. 10 our temporal consistency significantly improves the performance. Choosing s works well most of the evaluation metries.
predd
A.5 MODEL COMPLEXITY
We provide detailed runtime speed evaluated a single RTX2080TI with the model parameters shown in Tab. [] Compared with other state-of-the-an models, we achieve decent performance without introducing more computation cost.
A.6 QUALITATIVE ANALYSIS
We provide visual results of MISC the the Argoverse (Chang et al. 2019) validation set in Fig. as well as the Argoverse test set in Fig. These qualitative results demonstrate the effective- ness and the high-quality predicted trajectories of our method.
16

---

Under review as a conference paper at ICLR 2023
Table 10: Ablation study results of time-shift used by temporal consistency ETH Dataset
K=1
K=20
Time shift
Dataset
ADE
FDE
ADE
FDE
ETH
0.69 / 00.8
1.30 1.98
0.51/0.79
1.05 / 1.66
HOTEL
0.27 / 0.33
0.46 0.55
0.20 0.25
0.36 / 0.44
ETH
0.65 / 00.3
1.22 1.86
0.47 0.73
0.97 / 1.55
HOTEL
0.23 / 0.29
0.42 / 0.50
0.18 / 0.23
0.33 / 0.42
ETH
0.65 / 0.92
1,23 1.88
0.48 / 0.73
1.00 / 1.56
HOTEL
0.24 / 0.27
0.43 / 0.49
0.18 / 0.25
0.34 / 0.42
ETH
1.24 1.89
0.48 / 0.73
0,98 / 1.57
HOTEL
0.24
0.43 0.52
0.19 0.24
0.34 / 0.44
ETH
0.66 0.94
1.23 1.89
1.4970.73
0,99 / 1.58
HOTEL
0.25 / 0.31
0.44 / 0.51
0.20 / 0.25
0.33 / 0.44
Table 11: The number of parameters and running time.
Method
Param (M)
Speed (ms)
LaneGCN
55
DenseTNT
1.1
40
mm mmmrransformer
2.6
34
Ours
3.6
36
A.7
FAILURE CASES
We also present some failure cases on the validation set in Fig. Some possible reasons are:
The ground-truth labels contain some noises. Since the ground-truth labels are obtained from tracking, there may be some id switches, leading to the sudden perturbation of the agents' location (e.g., the first and third example the second row of Fig. 77. Under these scenarios, the predicted trajectories from MISC are reasonable and stable without large jerks.
The multi-modality problem. some situations, MISC can not predict the intention per- fectly without enough motion and map information. The first and third example in the first row of Fig. demonstrate this phenomenon. The agent makes a lane change decision without many hints in the historical information. Thus, this can be furtherly improved by introducing more map constraints.
17

---

Under review as a conference paper at ICLR 2023
Figure 7: Failure cases on the Argoverse validation set. The target agent's past trajectory in yellow, predicted trajectory in green, and ground truth in red.
18

---

Under review as a conference paper at ICLR 2023
Figure 8: The motion forecasting results on the Argoverse validation set. The target agent's past trajectory in yellow, predicted trajectory is in green, and ground truth is in red.
19

---

Under review as a conference paper at ICLR 2023
Figure 9: The motion forecasting results the Argoverse test set. The target agent's past trajectory in yellow and predicted trajectory in green.
20


Reviews:
[{"review_id": "_45zMCRmhvT", "reviewer": "Reviewer_5FaF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an approach for motion forecasting. Specifically, the paper proposes Motion forecasting with self-consistent constraints (MISC). It introduces temporal and spatial consistency priors to regularize predictions generated by the learning model. Furthermore, the paper also introduces a self-ensembling technique to model multi-modality. The proposed approach is evaluated on the Argoverse dataset.", "summary_of_the_review": "The work presents an interesting formulation of temporal consistency and applies it to the motion forecasting problem on the Argoverse dataset. The paper achieves strong performance and shows extensive ablation studies to show the effectiveness of each of the proposed components. However, my main concern with this work is that the evaluations are conducted on a single dataset which puts a question mark on the generalization capabilities of the proposed work. Therefore, my initial rating is **reject**. However, I am on the fence and would happily reconsider my recommendation if the authors are able to address the limitations discussed above. In particular, performing evaluations on either more datasets or other motion forecasting tasks could raise the impact of this work significantly. \n\nUpdate: I appreciate the results on the Waymo Open dataset. I have raised by score to 6. I think the temporal consistency idea is novel and seems to work well. However, I still believe the paper lacks a clear emphasis on a single contribution and seems to combine several independent ideas to improve performance on a single task which reduces the general applicability of the work.", "strength_and_weaknesses": "**Strengths**\n1. The temporal and spatial consistency constraints are well-motivated, novel and clearly described.\n2. Strong performance on the Argoverse dataset show the effectiveness of the proposed approach.\n3. Extensive ablation studies are conducted to validate the design choices taken in the paper.\n4. The paper is well-written and technically correct.\n\n**Weaknesses** \n1. **Single Dataset:** The evaluations are conducted on a single dataset which makes it difficult to judge whether the proposed approach would work in general.\n2.  **No experiments for online prediction:** One of the motivations of this work is that temporal consistency could be useful in the real world where the predictions are being made online or in a \"stream\" as stated in the paper. However, there are no explicit experiments where the paper is evaluated under this setting. \n3. **Lack of clear emphasis on main contribution:** The paper introduces several ideas including dual consistency, trajectory refinement and teacher-target constraints. While, I understand the motivation of the individual components and they do help in achieving strong performance on Argoverse, having these individual components makes it difficult to appreciate the main motivation of the work i.e. the dual consistency constraint. Further, it's not clear how the consistency constraint helps for the Argoverse dataset in particular and whether the proposed approach would generalize to other motion forecasting datasets/tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SNmyhnvpsg", "reviewer": "Reviewer_mCiK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces several consistency losses that help improve the performance of behavior prediction models. \n* The main innovation is the temporal consistency constraint, which enforces that predictions from adjacent time steps are similar. This is simple, but effective and -- to the best of my knowledge -- novel. \n* The spatial constraint is in effect a data augmentation technique -- enforcing robustness to small Gaussian noise to the input trajectories. I am not aware of this being commonly used, however, this is fairly straightforward, and the effect is pretty minor, compared to the temporal consistency constraint. As related work, ChauffeurNet (Bansal et al, 2019) also apply spatial perturbations, when they train a somewhat different type of model. \n* A model distillation technique is applied as an additional loss in the model, to improve the results. Model distillation for behavior prediction models is not novel per se. I am aware of at least one instance that is not cited:  \"Narrowing the Coordinate-frame Gap in Behavior Prediction Models: Distillation for Efficient and Accurate Scene-centric Motion Forecasting\" by DiJia Su and Bertrand Douillard and Rami Al-Rfou and Cheolho Park and Benjamin Sapp, ICRA 2022. \n\nThe losses above are applying to a TPCN architecture, augmented with a three-stage predictor head (goals -> trajectories -> trajectory refinement). The above is a fairly straightforward combination of existing architecture (TPCN) and model head ideas. ", "summary_of_the_review": "Strong experimental results. Good ablations. Several other ideas with marginal novelty like distillation. The main novelty -- temporal consistency loss -- is fairly straightforward but effective. Main concern is limited overall novelty in this paper, which seems low for ICLR. \n\nI am on the borderline overall, as I appreciate simple techniques that work. My final rating depends on clarifying details around model ensembles: whether leaderboard submissions were one model, or many. If there were many models, how were those obtained -- were they all trained using the same teacher ensemble? More details and experimental results there would help. Also, doing ablation of the main idea -- temporal and spatial consistency loss -- on the Waymo Open Dataset, which has less input noise, would help as well. \n\n", "strength_and_weaknesses": "Model strengths: \n- The temporal consistency constraint is simple, effective and seems novel. \n- Distillation (called Teacher - Target Constraints) in the paper), even if it does not appear novel per se, is shown to lead to SOTA (state of the art) results. The ICRA 22 paper by Su et al does not come close experimentally to push a model to SOTA. \n\nExperiment strengths: \n- SOTA or close to SOTA results on both Argoverse and Waymo Open Motion Dataset. \n- Relevant ablation results on the effect of introduced losses, also showing the constraints benefit models other than TPCN. \n\nWeaknesses: \n- Main novelty seems to be the temporal consistency constraint. Given how straightforward it is (predictions from two adjacent steps should be similar), unclear if this is sufficient for acceptance. \n- Some relevant work or context is not cited, especially for teacher-target constraints. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LyMIk0p8_Q", "reviewer": "Reviewer_xHGP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Motion forecastIng with Self-consistent Constraints (MISC), a framework that allows bootstrapping the performance of motion prediction without additional data. There are three different components proposed in the paper, namely, temporal consistency, which aims to enforce the prediction model to output consistent results given two overlapping observations. Spatial consistency aims to enforce the refinement module to generate the same results given perturbation on the input. The teacher-target constraints allow us to distill a model from multiple teacher models with different initializations and the same architecture. \n\n", "summary_of_the_review": "Given the limited insight provided in the paper, I think the technical contribution of the work is not significant enough. I lean towards rejection for now. ", "strength_and_weaknesses": "\nStrength:\n\n1. The proposed method would only increase the training cost by a reasonable amount of training resources and does not make deployment more computationally expensive.\n\n2. The authors do a great job of demonstrating the framework's efficacy through an extensive comparison with the current state of the art.\n\n3. The framework is tested with existing approaches, making the result even more convincing. \n\nWeaknesses:\n\n1. I am not sure whether it is a good idea to put different modules under the same umbrella, namely, self-consistency since the connection between components presented in the paper is loose. Each of them seems ad-hoc, thus, providing limited insights into the overall framework.\n\n2. The paper emphasizes too much on the result comparison, yet, misses important analysis. While there is a ‘standard’ component analysis tailored to understand ‘how much' each component contributes to the final improvement, not much analysis is given to ‘how’ each component helps. For example, in section 3.2, the paper states ‘Compared with data augmentation, it is the explicit regularization”. It is easy for the reader to identify the difference here, but much harder for the reader to understand which one would be better and why it is better. I think the experiment part suffers from the issue most. The experiment results do not reveal much information about the design choice of the framework. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-a-NEFkAyp", "reviewer": "Reviewer_MgTW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new method for trajectory forecasting. The proposed method consists of two ideas. (1) It introduces a new constraint such that (1a) predicted trajectories from consecutive inputs should not be different (temporal consistency) and (1b) the predicted trajectories should also be consistent regarding spatial transformations due to flipping and random noise (spatial consistency). (2) It further introduces a teacher-target constraint by enforcing multiple predictions from different training procedures to be consistent. Experimental results on the Argoverse dataset demonstrates the effectiveness of the proposed method over state-of-the-art methods.", "summary_of_the_review": "Currently I'm very neutral about this paper. It proposes a simple learning strategy involving several ideas, which performs really nice on the Argoverse leaderboard. However, it is not clear how applicable the proposed method is to a wide variety of situations, such as lower/higher framerate videos, or videos with other moving agents such as pedestrians. Limitations and failure cases are not discussed in detail, leaving it difficult to judge the significance of the proposed approach with the current manuscript.\n\n---\nPost-discussion comment: As my initial concerns above were resolved by the active discussion and paper revision with the authors, I would like to upgrade my score. Nevertheless, ", "strength_and_weaknesses": "## Strong points\n- Overall, the paper is well written and easy to follow.\n- The motivation to introduce the proposed spatiotemporal constraints is valid, and the specific approach to realize that key idea seems straightforward and technically sound. As far as I have checked, introducing such spatiotemporal constraints for trajectory forecasting seems new.\n- The proposed method outperforms many recent methods on the Argoverse leaderboard. The ablation study further shows that each of the technical components in the proposed method contributes to the final forecasting performance.\n\n## Weak points\n- The paper could have become stronger if the method is evaluated on various datasets other than Argoverse, which involve not only vehicles but also pedestrians, together with more detailed analysis and discussions on how s time shift s should be determined. I think so because the appropriate setting of s can depend on video framerates and the frequency of sudden changes in motion. As s becomes larger or for a lower framerate videos, the temporal constraint could enforce inappropriate matching and may result in performance degradation as implied in Table 6. On the other hand, when s is too small or video framerates are too high, the temporal constraint will do little. It also remains unclear how well the proposed method works when trajectories involve many sudden changes of motion speeds and directions, like those of pedestrians. Currently we can learn from the paper only that s=1 worked the best for Argoverse, which is not very much informative considering the potential applications of the proposed method to other datasets.\n- The limitation and failure cases of the proposed method are not discussed, which I believe are crucial to judge the significance of the method. If the proposed method is applicable just for smooth vehicle trajectories recorded at a framerate within a certain narrow range, the proposed work could have a limited applicability and significance. If not, I think that this paper presents a nice contribution with a simple idea that works effectively for various situations. \n\n## Additional feedback\n- I would expect qualitative results to visualize not only the proposed method but also other baselines.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject