Towards Capturing the Temporal Dynamics for Trajectory Prediction: a Coarse-to-Fine Approach
Authors: Xiaosong Jia
Li Chen
Penghao Wu
Jia Zeng
Junchi Yan
Hongyang Li
Yu Qiao
Venue: CoRL 2022 Poster (2022)
Keywords: autonomous prediction
trajectory prediction

Abstract:  Trajectory prediction is one of the basic tasks in the autonomous driving field, which aims to predict the future position of other agents around the ego vehicle so that a safe yet efficient driving plan could be generated in the downstream module. Recently, deep learning based methods dominate the field. State-of-the-art (SOTA) methods usually follow an encoder-decoder paradigm. Specifically, the encoder is responsible for extracting information from agents' history states and HD-Map and providing a representation vector for each agent. Taking these vectors as input, the decoder predicts multi-step future positions for each agent, which is usually accomplished by a single multi-layer perceptron (MLP) to directly output a Tx2 tensor. Though models with adoptation of MLP decoder have dominated the leaderboard of multiple datasets, `the elephant in the room is that the temporal correlation among future time-steps is ignored since there is no direct relation among output neurons of a MLP. In this work, we examine this design choice and investigate several ways to apply the temporal inductive bias into the generation of future trajectories on top of a SOTA encoder. We find that simply using autoregressive RNN to generate future positions would lead to significant performance drop even with techniques such as history highway and teacher forcing. Instead, taking scratch trajectories generated by MLP as input, an additional refinement module based on structures with temporal prior such as RNN or 1D-CNN could remarkably boost the accuracy. Furthermore, we examine several objective functions to  emphasize the temporal priors. By the combination of aforementioned techniques to introduce the temporal prior, we improve the top-ranked method's performance by a large margin and achieve SOTA result on the Waymo Open Motion Challenge.

Towards Capturing the Temporal Dynamics for Trajectory Prediction: a Coarse-to-Fine Approach
Xiaosong Jia Li Chen", Penghao Wu!!2, Jía Zeng' Junchi Y2n Hongyang Li!:2, Yu Qiao²
1.2x
*corresponding author
Shanghai Jiao Tong University Shanghai AI Laboratory (jiaxiaosong, wupenghaocraig, (lichen, lihongyang, (taoyu]@pjaan org. en
yanjunchi @s edu. cn
Trajectory prediction is one of the basic tasks in the autonomous driv- ing field, which aims predict the future position of other agents around the ego vehicle so that a safe yet efficient driving plan could be generated in the down- stream module. Recently, deep learning based methods dominate the field. State- of-the-art (SOTA) methods usually follow an encoder-decoder paradigm. Specif- ically, the encoder is responsible for extracting information from agents' history states and HD-Map and providing a representation vector for each agent. Taking these vectors as input, the decoder predicts multi-step future positions for each agent, which is usually accomplished by single multi-layer perceptron (MLP) to directly output a Tx2 tensor. Though models with adoptation of MI.P decoder have dominated the leaderboard of multiple datasets, 'the elephant in the room is that the temporal correlation among future time-steps is ignored since there is no direct relation among output neurons of MLP. In this work, we examine this design choice and investigate several ways to apply the temporal inductive bias into the generation of future trajectories on top of a SOTA encoder. We find that simply using autoregressive RNN to generate future positions would lead to significant performance drop even with techniques such history highway and teacher forcing. Instead, taking scratch trajectories generated by MLP as input, additional refinement module based on structures with temporal prior such RNN or ID-CNN could remarkably boost the accuracy. Furthermore, we examine several objective functions emphasize the temporal priors. By the combination of aforementioned techniques to introduce the temporal prior, we improve the top- ranked method's is performance by large margin and achieve SOTA result on the Waymo Open Motion Challenge.
Abstract:
Keywords: Autonomous Driving, Trajectory Prediction, Temporal Correlation
Introduction
Trajectory prediction aims to predict positions in the next several seconds, which is one of the essential tasks for the autonomous driving. Recently, deep learning based methods have dominated the field and they usually follow an encoder-decoder paradigm [1, 2]. The encoder takes all agents' history states (position, velocity, heading, etc.) and HD-Map (lanes, road lines, traffic lights, etc.) as inputs and outputs representation vector for each agent. The decoder then generates future trajectories for the agents based on these vectors. We observe that current top-ranked methods |3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14] on multiple datasets' all adopt an MLP as the decoder output T x tensor and then reshape it into the final prediction.
'The investigation is based their corresponding public leadrboard of the date of the submission. DCMS [3] and HiVT [4] Argoverse [15], MultiPath+ [5], SceneTransfommer [6], DNT [7]. and HDGT [8] on Waymo Open Dataset [9], Multimodel Tranformer [10], GOHOME [11] and THOMAS [12] on INTERACTION [16), PGP [13] and LaPred++ [14] on nuScenes [17].
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Though the MLP design achieves satisfying performance empirically, we point out that it ignores the temporal correlation among the future time-steps. Mathematically, we denote the hidden repre- sentation vector of an agent as H E IR# H and its predicted future trajectory as E R where T indicates the length of prediction horizon and 2 indicates the 2D coordinate. The MLP accomplishes the mapping:
W2 W2 (W HH) reshape(T 2),
(I)
where Wi E H and W E (2 H' are weight matrices, and 8(.) is a non-linear function.
We can find that every two rows of W2 contains the weights generate the position for one specific time-step. Since aforementioned SOTA works adopt either per-step negative log likelihood (NLL) loss or per-step mean squared error (MSE) loss, the back-propagation updating for every two rows of W2 is independent. Such design is based on the assumption of conditional independence among future time-steps. However, the assumption is untenable in real-world scenarios and thus could be harmful for the prediction.
In this work, aim discuss about 'the elephant in the room' and explore several approaches to model the temporal correlation for future time-steps on top of a SOTA encoder. We find that simply replacing the MLP with autoregressive RNN leads to significant performance drop, even with techniques such as teacher forcing [18] or history highway [19] to alleviate its drawbacks of long-term dependencies and cumulative error. Inspired by the success of refine modules in the deep learning based methods [20, 21, 22, 23] which ease the optimization difficulty of a single decoder head, we find that when first using an MLP to generate a scratch trajectory and then using a structure with temporal inductive bias (RNN/ID-CNN) to refine it, the SOTA model's per- formance could be boosted. We also explore different training objectives apply temporal priors on the output besides modifications of the neural network structure. We find that directly fitting the velocity degenerates the performance, while accumulating the velocity into the displacement (i.e. calculating the integral of velocity over time) and still using the coordinates as the target would lead to better prediction. By combining these discovered effective techniques, we significantly boost the performance of a SOTA model on Waymo Open Motion Dataset, which verifies the benefit of utilizing the temporal prior.
In summary, the contributions of the paper are:
We explore ways to incorporate the temporal structure in the decoder and propose ef- paradigm. By using an MLP and a temporal module in two steps respectively, it explicitly models the temporal correlation among future time steps without the burden of training an autoregressive RNN.
fective srratch-then--eepprall----fffff
We examine several objective functions to add temporal priors to the output. We report the notable improvements by first generating velocity and then accumulating temporally into coordinates to calculate per-step loss.
We conduct thorough ablation study on the Waymo Open Motion Dataset to test different ways of introducing the temporal prior into the decoding process. With the combination of our effective attempts, the performance of SOTA encoder boosted by a large margin on Waymo Public Leaderboard.
We consider the correlation among future time steps as a non-negligible factor, which is not exploited by current SOTA methods. We hope the attempts and successful parts illustrated in this study could provide useful information for this line of study in the trajectory prediction community.
2 Related Works
Pioneering works of deep learning based trajectory prediction models usually adopt an RNN decoder to generate future trajectories in an autoregressive way. Social LSTM [24] applies an LSTM decoder after the LSTM encoder, and adopts the NLL. loss for each time-step. Its extension. Social GAN [25], uses GAN [26] to encode different modalities and adopts per-step MSE loss. DESIRE [27] employs a conditional variational autoencoder to sample diverse set of hypothetical modalities and adopts a GRU to generate future predictions. MFP [28] introduces a latent distribution to model the joint modality of multiple agents and utilizes an RNN for decoding as well. Mulitpath [29] leverages fixed set of future state-sequence anchors that correspond to modes of the trajectory distribution and regresses offsets along with uncertainties with respect to anchor waypoints by RNN. IDE-Net [30]

---

applies additional loss to infer the interaction types in an unsupervised fashion. To capture the temporal dynamics, Trajecton++ [31] utilizes GRU as the decoder to predict a bivariate Gaussian distribution over control actions. By applying dynamically-extene
unicycle models over the pre- dicted control signals, able to guarantee that its trajectory samples are dynamically feasible. DKM [32] designs an autoregressive differentiable dynamics layer to generate the control signal per-step.
Though it is intuitive to use an autoregressive RNN to generate the sequence of future positions due to its recurrent nature, modern methods with a concise MLP decoder outperform previous RNN- based ones and dominate multiple public motion prediction leaderboards, including Waymo [9], nuScenes [17]. Argoverse [15] and INTERACTION [16]. [33] utilizes CNN to encode the scene and generates trajectories directly by an MLP, while LaneGCN [2] encodes the scene by four distinct graph convolutional network (GCN) layers. PGP [13] traverses the lane graph to incorporate the lane topology while LaPred [14] applies attention mechanism on lanes. HiVT [4] designs hierarchical way to aggregate the local and global information efficiently. Scene' Transforme [6] encodes the spatial and temporal information in factorized way. HDGT [8] models the driving scene heterogeneous graph. TPCN [34] and its extension DCMS [3] encodes the scene in the form of point cloud and apply an MLP on the pooled instance vector to generate future trajectories. VectorNet [1] encodes both the agents and lanes as vectors on a global graph and decodes the future trajectories based on the corresponding agents' vectors with an MLP. Its extension TNT [35] and Dense [7] generate trajectories in a two-stage way: the stage one aims to predict the target/goal of agents and the stage two aims to complete the entire trajectories conditioned on the predicted end point by an MIP. GOHOME [11] and its extension THOMAS [12] output heatmaps for goals and then sample trajectories based the selected goals. LaneRCNN [36] follows the goal-based prediction framework as well but they complete the trajectories with a polynomial prior and use an MLP to refine the polyline. In PLOP [37], means of the distribution of future coordinates are generated in polynomials of degree 4 of time. In the recent study Multipath++ [5], they find that both polynomial representations and control signal representations lead to degenerated performance compared to directly regressing the coordinates.
Revisiting the existing literature, can find that decoders and training objectives with different types of temporal structure are proposed. However, they are usually entangled with the encoder module and other settings, which makes it hard to draw clear conclusion about the advantages and disadvantages of these designs. In this work, we examine different choice of applying temporal structure in the generation process of future trajectories while keep other settings such as the dataset, the encoder, the training schedule, and the hyperparameter the same for a fair comparison. To the best of our knowledge, we are the first to explore this design choice with the modern vector-based encoder on large-scale autonomous driving datasets.
3 Problem Formulation
Given history states [S N) of N agents such as their coordinate, velocity, and heading and their surrounding information Senv including lanes, traffic lights, etc., the goal is to predict their future positions in the next T time-steps {T' E 2 = I,...,NN N). Current SOTA methods usu- ally follow the encoder-decoder paradigm. The encoder takes all agents' histories and environment information as inputs and outputs a representation vector H' IR for each agent
[ j 1,..., N) = Encoder((S li = 1,...,N), Sear).
(2)
Conditioned on agents' hidden feature H, the decoder aims to predict the distributions of their future trajectories respectively:
p(T: H.) p P(Y Y H.),, i= N.
Y is agent i's distribution of states at future time-step
As mentioned in Sec. 1, most SOTA works generate coordinates of different future time-steps with an MLP and adopt a step-wise MSE or NLL loss, which assumes that the distributions of states on different future time-steps are conditionally independent:
T
P PY Y! = II PYY H.), i= 1,.,.,N N.
(4)
3

---

We argue that this assumption ignores the temporal correlation among future time-steps. which could result less consistent and physically infeasible trajectories. In this work, we explore two parts to apply the temporal priors: the mapping function (H,) = f; = ( = 1,.. and the objective function [[T,, (H,)).
We examine different decoders and objective functions on the validation set of Waymo Open Motion [9]. For all models, we adopt the same encoder module HDGT [8]. top-ranked encoder on Waymo Leaderboard. All experiments are conducted with the same setting and hyperparameters. Please refer to Sec. 9 for more details. The following metrics are used: a) mi- nADE (Minimum Average Distance Error): the minimum value of the Fuclidean distance between the prediction and ground truth averaged by the prediction length T. for K required predictions. b) Fin (Minimum Final Distance Error): similar minADE, despite that it only calculates the error at the final time T. c) MR (Missing Rate): the ratio of whether the Euclidean distance between the prediction and ground truth at the final time-step for all K predictions larger than 2 meters.
Experiments Setting:
4 Temporal Enhanced Decoder
4.1 Autoregressive RNN
Decoder
Decoder
RNN Decoder (llighway)
(Teacher Forcing)
Mix
RNNN
RNN
RNN
RNN
RNN
RNN
RNN
RNN
RNN
RNN
RNN
Figure 1: Illustration of the MLP, autoregressive RNN decoder, and their variants.
The autoregressive RNN predicts future trajectories in iterative way with the assumption that the coordinate distribution each time-step depends on that from previous time-steps (for notation brevity, drop the subscript i of agents in the following sections):
T
P(Y' H = P Y H).
(5)
The generation process is implemented Two commonly used RNN structures LSTM [38] and GRU [39] and we adopt an LSTM here for its high capacity?. Mean- while, there are two widely known draw- backs of RNN: the difficulty to capture long-term dependency due to the implicit past information retention mechanism [40] and the cumulative error [18]. As for the first problem, inspired by highway RNN [41, 42), we implement a variant of LSTM which takes the vector H as an ad- ditional input at each time-step so that the history information is always accessible. To alleviate the cumulative error of autoregressive model, we adopt the teacher forcing with scheduled sampling strategy [18]. Specifically, each time-step, the L.STM would have a possibility par to take the ground-truth (GT) state as input instead of the predicted one, and the possibility PGT decays as the training goes by. In this way, the model could updated with less noise at the start of training while at the end could learn to robustly do the prediction based on its previous output [18]. To futher explore the differences between the RNN and MLP decoder, we design a 'mix' them. We
by maintaining an internal state vector in RNN.
Table Performance comparison of different de- coder structure. H indicates the hightway technique and TF indicates the teacher forcing technique. Note that in Mix decoder, adopt 20 which means autoregressing for 4 steps.
Decoder
minADE
minFDE
MR.
MIP
0.6056
1.2328
0.1723
RNN
0.7259
1.4903
0.2116
RNN (H)
0.7454
1.5438
0.2272
RNN (TF)
0.7179
1.4575
0.2053
Mix
0.7002
1,4620
0.2033
2 Pleasse refer to [38] for details about LSTM.

---

use an LSTM to autoregressively generates / steps' representation vectors and then use an MLP to predict positions of T/l time-steps with each step's vector as input.
In Figure 1, we give illustrations of the aforementioned decoders. Table is the performance of the different decoder structures Waymo Motion Dataset (validation set). We conclude that:
Autoregressive RNN performs worse than the MLP by a large margin, which explains why most SOTA methods adopt the latter one.
The cumulative error is one cause of autoregressive RNN's bad performance. The teacher forcing technique could alleviate this problem, but its performance is still worse than MLP.
Concatenating the history feature H with each time-step's state as input to RNN could not solve the long-term dependency problem. Indeed, it leads to worse performance than vanilla RNN. It might come from the difficulty of updating H for all T time-steps as the input RNN.
The mix of the two structure has the performance between the MLP and RNN decoder. By reducing the autoregressive steps, the optimization becomes easier. However, it is still much worse than concise MLP decoder.
In summary, we find that the inherent issues of autoregressive RNN make it hard to compete with the MLP decoder in the trajectory prediction task. We should explore other ways to apply the temporal inductive bias the network structure.
4.2 Scratch-then-Temporally
Y offset
Refine Loss
Refine Module
seraach
Scratch Loss
Seratch MI.P
Ground-Truth
Figure 2: The proposed seautch-teen-temporallll---e paradigm. An MLP is first trained to generate the scratch trajectory Y scratchh Taking Y scratch, the input, the refinement module predicts the offset for each time-step. could be a structure with temporal inductive bias (RNN/ID-CNN).
Through Section 4.1, we find that directly utilizing RNN to predict trajectories is hard because has to generate in an autoregressive fashion, which troublesome. Thus, to utilize the RNN while avoiding the autoregressive part, need generate proper input states to RNN for each time-step. We notice the fact that a simple MLP is proven to be able to do the prediction accurately in the existing literature, which inspires to train MLP to generate scratch trajectories and then using structures with temporal prior (RNN or ID-CNN) to refine the scratch.
Formally, given the hidden feature H from the encoder, we first train an MLP with the output shape of T 2, denoted as @scratch'
scratch It 1. scratch (H).
Based on the scratch trajectories { which encodes agents interactions and map information, we train a refinement module érefine to refine them:
scratch | 1, I..? as well as the feature H from the encoder
offset! 1,...T) = refine (f scratch 11....T)
where the objective for Y 2 gives an illustration of the proposed paradigm.
offset is the difference between Ý y
scratch and the ground-truth Y!. Fig-
5

---

Note that Pscratch and crffine could be trained in an end-to-end fashion. In experiments. we have different choices for refine a) another MLP, similar to [3], which takes the flattened scratch trajectory the input and output a 2 tensor. b) an LSTM which takes the scratch tra- jectory the input of the th time-step.. c) ID- CNN, which serves as the local temporal infor- mation filter. The results Additionally, for the variant with best perfor- mance, ID-CNN, also conduct experiments on cascading multiple refinement modules to explore the effects of the multi-granularity refinement.
Table 2: Performance comparison of different re- finement modules. 5 * ID-CNN means 5 cascaded ID-CNN refinement modules.
Peefine
minADE4
minPDEL
MR4
None
0.6056
1.2328
0.1723
MLP
0.6004
1.2300
0.1680
RNN
0.5931
1.2195
0.1637
ID-CNN
0.5908
1.2084
0.1625
shown in Table 2.
5 ID-CNN
0.5871
1.1893
0.1554
We can conclude that: An additional refinement module could improve the performance notablely, even if is simply another MIP. b) Refinement modules with temporal inductive bias (RNN and ID- CNN) perform better than the one without such bias. c) By cascading multiple refinement modules, the performance could be further boosted.
In summary, we find that the proposed scratch-then-teee frauch-then--eemorrlly----eeeff paradigm could combine the best of two worlds: the easy optimization of the MLP and the temporal inductive bias RNN/ID- CNN.
Temporally Correlated Objective Function
Coordinate Loss
Velocity Loss (Vezi Vy,?
Cumulative Coordinate Loss (Ours) (x,,1) (x2. (x).
(1) (x).
(x
(Veg Vy
(Vigr Vyg
(x F
(2 Ks.ys)
( Gi
Vag. Vyg)
Vy La V 53
Decoder
Decoder
Figure 3: Three objective functions for regression. Cumulative coordinate loss achieves the best performance.
In addition to designing the neural network with the temporal structure, the temporal priors among future time-steps could be applied on the objective function part as well. Existing works [1, 6, 30, 5] usually calculate the regression loss per time-step and then take the mean of all time-steps as the overall loss:
T Σ
C((Ý' 1,.., 7J, (Y'lt
TT)
T Stepp Y", Y'),
(8)
where Y and Y are the predicted and ground-truth coordinate at time-step / respectively, and Cstep is the loss function applied on each time-step, which could either be the MSE loss [1, 7] or NLL loss [6. 5J. This objective function assumes that agents' future positions are independent with each other, which ignores the physical constraint and could result in kinematically infeasible prediction. Additionally, for the long-term trajectory prediction, agents might move hundreds of meters in the predicted interval. As result, the magnitude of the late time-steps' coordinates might be much larger than the early ones, which brings difficulty for the learning process.
One intuitive solution for the aforementiond issues to predict the control signal instead [32, 31, 43] and the coordinate could be obtained by physical models during inference. In this way, the network explicitly learns the dynamics which provides strong prior knowledge to the model while keeping the output magnitude of different time-steps similar. However, in empirical experiments, predicting velocity leads to performance drop, which is consistent with the conclusion in [5]. One burden for predicting the control signal the cumulative error which makes the model sensitive noise. With a prediction error occurring at the first time-step, the entire generated trajectory would shift from the ground fround-truth, even if predictions at all other time-steps are correct.

---

To alleviate the problem, we propose let the model output velocity while calculating the regression loss between the accumulation of the models' out- put and the ground-truth coordinate. Formally, given the predicted velocity
Table Performance comparison of different objective
functions. The third is the proposed cumulative coordi- nate loss.
Loss Func
minADEy
minFDE4
MR.
Coordinate Loss
0.6056
1.2328
0.1723
Velocity Loss
0.6346
1.2590
0,1822
[Y = [,,..,'1 T and the GT coordi- nate (Y' I,... T), the objective is:
Cumu. Coord. Loss
0,5921
1.1851
0.1542
1
cumuuattvve
Σ 1=1 m/=1
step
Ym Y
(9)
T
In this way, we combine the best of the two worlds: the output is the form of velocity which introduces temporal correlation among time-steps while the early time-step's influence on its later coordinates is considered by the accumulation of the loss. Figure. 3 illustrates the three objective functions mentioned above and Table. 3 gives their performance. The proposed cumulative coordi- nate loss function achieves the best performance.
Results on the Public Leaderboard
Table 4: Performance comparison on Waymo Open Motion leaderboard, test set.
Here provide results the test set of Waymo Open Motion to ver- ify that conclusions not over- fitting the validation set, as shown in Table 4. We find that by combining the proposed techniques, the SOTA encoder HDGT''s perfor- mance could be further boosted and achieves competitive performance on the learderboard".
Model
minADE
minFDE]
MRJ
ReCoAt [44]
0.7703
1.6668
0.2437
DenseTNT [7]
1,0387
1.5514
0.1573
SceneTransfornee [6]
0.6117
1.2116
0.1564
MultiPath++ [5]
0.5557
1.1577
0.1340
golfer
0,5533
1.1608
0.1354
MTRA
0.5640
1.1344
0.1
HDGT [8]
0.5933
1.2055
0,1511
HDGT + Ours
0.5703
1.1434
0.1440
7 Ablation Study
Table 5: Ablation study of the proposed techniques. Note that for the temporal refinement module, we use the best setting, 5* ID-CNN.
Temporal Refine
Cumu.
minADE
minFDE]
MR
TRI(%)
UR(S))
Time(s)
Coord. Loss
0.6056
1.2328
0.1723
10.95
2.37
1146 + 16.5
0.5871
1.1893
0.1554
5.27
0.38
1181 1: 14.49
X
V
0.5921
1.1851
0. 0.1548
12.05
4.87
1153 + 15.33
0.5835
1.1833
0.1532
6.89
0.39
1206 + 14.22
In this section, we study the effects of the combination of the proposed
techniques. In addition to the commonly used metrics minADE//inFDE/MRR, to evaluate the physical feasibility of predicted trajectories under different settings, calculate the Turning Ra- dius Infeasibility (TRI) [5]. If the turning radius (the circumradius constituting three consecutive waypoints) along the predicted trajectories is less than certain threshold, it is treated as violation. We set this threshold as 3.5m the approximate minimum turning radius threshold for a midsize sedan, following [5]. We also evaluate the smoothness of the trajectory by calculating the accelera- tion and jerk, i.e., the second and third time derivative of the position. According to [45, 46], human drivers' maximum acceleration around 5.0m and that of jerk is 2.0m/s*. Thus, we define the
Phttpa: /wymo. www/open//eeee/ec/cccoocc hemm/ppen/chhlll http/challlllea Sataallennee///0 "Note that some methods with ensemble while ours is single model.
wet.eeeee22/mmmttaeeeee 200022/motion -rrdiiiiioreiiiioonn/
7

---

Unsmooth Ratio (UR) as: if at any time-step of a trajectory, its corresponding acceleration is larger than 5.0m/s² or jerk is larger than 2.0m/s", it considered an unsmooth step.
The ablation study results are shown in Table 5. We can conclude that: both techniques could im- prove the distance-based metrics of the baseline model and the combination of them could further boost the performance. As for the Turning Radius Infeasibility (TRI) and Unsmooth Ratio (UR), the temporal refinement module could make generated trajectories much more smooth and physically feasible while the cumulative coordinate loss is harmful for them. We think it is because predic- tions of different time-steps have more information interaction in the temporal refinement module, which is helpful for their consistency. However, the cumulative coordinate loss entangles all the information by the accumulation process which might be unstable compared to direct regression.
Also, to evaluate the impact of this modification is on the inference time of the origin model, we cal- culcate the mean time and std of 30 runs while keeping the environment, devices, hyperparameters like batch-size the same. The computational burden of Cumulative Coordinate Loss is neglectable while the 5 cascaded refine modules takes 3-5% more inference time.
8 Conclusion
In this paper, we explore ways of applying the temporal priors on the generation process of trajecto- ries. We find that intuitively using autoregressive RNN would lead degenerated performance compared to the commonly used MLP decoder. To combine the best of two methods, we propose seallci-then-temporllll-----
paradigm, which first generates a scratch trajectory by an MLP and then incorporates the temporal structure to refine the trajectory. We also find that directly fitting the velocity degenerates the performance, while accumulating the velocity and then still using the coordinate as the target would lead to better performance. Finally, by combining these discovered effective techniques, we boost the SOTA encoder HDGT's performance by a large margin in the Waymo Motion Leaderboard, which verifies the benefits of utilizing our proposed temporal prior.
Implementation Details
We adopt HDGT [8] as the encoder for all models with the same hyperparameters. We use all agents in the scene as a sample for the wholeness of the information while [8] uses every target agent and its surrounding
agents as a sample. We use max- max-pooling as the aggregation function for the lane node for less computational cost, which we find has few degeneration of the performance. We train all ablation models with 30 epochs for quick verification of the idea and train the final submission to the leaderboard with 120 epochs which has similar training steps with [8].
10 Limitation and Future Work
Experiments are only conducted on one SOTA encoder on the Waymo Open Motion Dataset (WOMP). Though it is the largest public dataset for motion prediction, there still could be poten- tial biased conclusion. To address this issue, in the future works, we might implement more SOTA encoders and conduct experiments on more datasets to further verify the conclusion.
Autoregressive RNN, as an structure naturally with temporal prior, did not work well under the simple attempts tried in this paper. More sophisticated and well-designed streture could be explored such as [47, 48] for better convergence and performance.
Acknowledgments
This work was partly supported by NSFC (62206172, 62222607). Shanghai Municipal Science and Technology Major Project (2021SHZDZX0100), and Shanghai Committee of Science and Technol- ogy (21021100100).
References
[1] Gao, C. Sun, H. Zhao, Y. Shen, Anguelov, C. Li, and C. Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In CVPR, 2020.
8

---

[2] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forecasting. In ECCV, 2020.
[3] M. Ye, J. Xu, X. Xu, T. Cao, and Q Chen. Dems: Motion forecasting with dual consistency and multi-pseudo-target supervision, 2022.
[4] Z. Zhou, L. Ye, J. Wang, K. Wu, and L. Kejie. Hivt: Hierarchical vector transformer for multi- agent motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[5] B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cormman, K. Chen, B. Douillard, C. P. Lam, D. Anguelov, et al. Multipath+:: Efficient information fusion and trajectory aggregation for behavior prediction. arXiv preprint arXiv:2111.
2021.
[6] Ngiam, B. Caine, V. Vasudevan, 7. Zhang, H.-T. L.. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, et al. Scene transformer: A unified multi-task model for behavior prediction and planning. arXiv preprint (rXiv:2106.08417 2021.
[7] J. Gu, C. Sun, and H. Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In ICCV, 2021.
[8] X. Jia, P. Wu, L. Chen, H. Li, Y. Liu, and Yan. Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. arXiv preprint arXiv:2205.09753 2022.
[9] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, Sapp. C. Qi, Y. Zhou, et al. Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset. arXiv preprint wrXiv:2104. 10133, 2021.
[10] Z. Huang, X. Mo, and C. Lv. Multi-modal motion prediction with transformer-based neural network for autonomous driving. arXiv preprint arXiv:2109.0644 2021.
[11] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. GOHOME: graph- oriented heatmap output for future motion estimation. arXiv preprint arXiv:2109.0180 01887, 2021.
[12] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. Thomas: Trajectory heatmap output with learned multi-agent sampling. arXiv preprint arXiv:21 10.06607, 2021.
[13] N. Deo, E. Wolff, and O. Beijbom. Multimodal trajectory prediction conditioned on lane-graph traversals. In 5th Annual Conference on Robot Learning, 2021.
[14] B. Kim, S. H. Park, S. Lee, E. Khoshimjonov, D. Kum, J. Kim, J. S. Kim, and J. W. Choi. Lapred: Lane-aware prediction of multi-modal future trajectories of dynamic agents. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 16636-16443 June 2021.
[15] M.-F. Chang, J. W. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, and J. Hays. Argoverse: 3d tracking and forecasting with rich maps. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[16] W. Zhan, L. Sun, D. Wang, H. Shi, A. Clausse, M. Naumann, J. Kummerle, H. Konigshof, C. Stiller, A. de La Fortelle, et al. Interaction dataset: An international, adversarial and co- operative motion dataset in interactive driving scenarios with semantic maps. arXiv preprint 1rXiv:1910.03088,, 2019.
[17] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[18] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
[19] J. G. Zilly, R. K. Srivastava, J. Koutník, and J. Schmidhuber. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' page 4189-4198. Jog,, 2017.

---

[20] Z. Cai and N. Vasconcelos. Cascade r-cmm: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154- 6162, 2018.
[21] Zhang, Wen, X. Bian, Z. Lei, and Li. Single-shot refinement neural network for object detection. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 2203-4212, 2018.
[22] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213- 229. Springer, 2020.
[23] H. Shao, L. Wang. R. Chen, H. Li, and Y. Liu. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. arXiv preprint arXiv:2207.14022 2022.
[24] A. Alahi, K. Goel, Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 961-971, 2016.
[25] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2255-2264, 2018.
[26] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014.
[27] N. Lee, W. Choi, P. Vernaza, C. B. Choy, P. H. Torr, and M. Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 336-345, 2017.
[28] Y. C. Tang and R. Salakhutdinov. Multiple futures prediction. Advances in neural information processing systems, 2019.
[29] Y. Chai, B. Sapp, M. Bansal, and D. Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. arXiv preprint arXivv1910.05444, 2019.
[30] X. Jia, L. Sun, H. Zhao, M. Tomizuka, and W. Zhan. Multi-agent trajectory prediction by combining egocentric and allocentric views. In Conference on Robot Learning, pages 1434- 1443. PMLR, 2022.
[31] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++:: Dynamically-fcasible trajectory forecasting with heterogeneous data. In ECCV, 2020.
[32] H. Cui, T. Nguyen, F-C. Chou, T.-H. Lin, J. Schneider, D. Bradley, and N. Djuric. Deep kinematic models for kinematically feasible vehicle trajectory predictions. In 2020 IEEE In- ternational Conference on Robotics and Automation (ICRA), pages 10563-10569, IEEE, 2020.
[33] H. Cui, Radosavljevic, F.-C Chou, T.-H. Lin, Nguyen, T.-K. Huang. J. Schneider, and N. Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 International Conference on Roboties and Automation (ICRA), pages 2090- 2096. IEEE. 2019.
[34] M. Ye, T. Cao, and Q. Chen. Tpen: Temporal point cloud networks for motion forecasting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 1131--1132,, June 2021.
[35] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, C. Li, and Anguelov. Tnt: Target-driven trajectory prediction. In CoRL., 2020.
[36] W. Zeng. M. Liang, R. Liao, and R. Urtasun. Lanerenn: Distributed representations for graph- centric motion forecasting. In 2021 IEEF/RSI International Conference on Intelligent Robots and Systems (IROS), pages 532-539, 2021. 1oi:10.1109/RROSS 168.202...666033.
10

---

[37] T. Bubet, É. Wirbel, and X. Perrotton. Plop: Probabilistic polynomial objects trajectory plan- ning for autonomous driving. ArXiv, abs/2003.08744, 2020.
[38] Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 9()) 1735- 1780, 1997.
[39] K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1259, 2014.
[40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I., Jones, A. N. Gomez, t.. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[41] J. G. Zilly, R. K. Srivastava, J. Koutník, and J. Schmidhuber. Recurrent highway networks. ArXiv, abs/1607.0344 607.03474, 2017.
[42] Y. Zhang, G. Chen, Yu, Yao, S. Khudanpur, and J. R. Glass. Highway long short- term memory mns for distant speech recognition. 2016 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5755-5759, 2016.
[43] Wu, X. Jia, L. Chen, J. Yan, H. Li, and Y. Qiao. Trajectory-guided control prediction for end- to-end autonomous driving: A simple yet strong baseline. arXiv preprint arXiv:2206.08129, 2022.
[44] Z. Huang, X. Mo, and C. Lv. Recoat: A deep learning framework with attention mechanism for multi-modal motion prediction. Workshop on Autonomous Driving, CVPR, 2021.
[45] I. Svensson and C. Eriksson. Tuning for ride quality in autonomous vehicle Application to linear quadratic path planning algorithm. 2015.
[46] I. Bac, J. Moon, and J. Sco. Toward comfortable driving experience for self-driving shuttle bus. Electronics, 2019.
[47] A. G. ALIAS PARTH GOYAL, A. Sordoni, M.-A. Côté, N. R. Ke, and Y. Bengio. Z-forcing: Training stochastic recurrent networks. Advances in neural information processing systems, 30, 2017.
[48] T. Trinh, A. Dai, T. Luong, and Q. Le. Learning longer-term dependencies in rmns with aux- iliary losses. In International Conference on Machine Learning, pages 4965-4974 PMLR, 2018.


Reviews:
[{"review_id": "fMkE31T87xKF", "reviewer": "Reviewer_z8DX", "summary": "Most of the current trajectory prediction methods use MLP as the final decoder, which ignores the inherent temporal correlation constraint in the future trajectory. To solve this problem, the authors propose Scratch-then-Temporally-Refine for the decoder and Temporally Correlated Objective Function for the loss to impose the temporal inductive bias during the trajectory generation. Through the experiment on the Waymo Open Motion Challenge, the authors show the improvement of the current top-ranked method by using the above methods. The ablation suggests that not only the distance-based metrics are boosted but also the feasibility e.g. Turning Radius Infeasibility (TRI) and Unsmooth Ratio (UR). ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\n* I enjoy the idea “Scratch-then-Temporally-Refine”. Since the Autoregressive RNN and MLP-based trajectory refinement are commonly used in the decoder independently, the authors combine them together to further improve prediction results. But it seems to be just a trick in the decoder. \n\n* The ablation study is comprehensive and convincing. Detailed experiments have been conducted to indicate the superiority of each module.  \n\n* The paper is clearly written and easy to read. \n\nWeaknesses \n\n* The proposed loss function “Temporally Correlated Objective Function” is not novel. Concretely, the prediction header generates control signal velocity at each time step, then the predicted location is computed by velocity accumulation. This process is identical with Producing Dynamically-Feasible Trajectories in the paper [“Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data”](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123630664.pdf). \n\n* As mentioned by authors in paper, more SOTA encoder should be considered to verify the performance of the proposed methods. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the “Scratch-then-Temporally-Refine” module shows promising performance for the decoder, the “Temporally Correlated Objective Function” loss has been used in the previous work. Thus, the overall contribution is not strong enough to support paper publishing at CoRL. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "* I am curious why the additional refinement module (MLP, RNN, 1D-CNN) could improve the prediction performance. Is there any intuitive explanation? Could you please add more discussions?", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "MOWAw_r-UsN", "reviewer": "Reviewer_G9vs", "summary": "The paper analyses why current SotA motion forecasting methods use MLP output heads and not recurrent/auto-regressive ones. The paper then proposes several improvements to an MLP-output method that brings in benefits from RNN-based approaches. This leads to a noticeable score improvement on the public leaderboard for the WOMD challenge.\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper was such a fun read. The structure is more like a student report than a classic method-experiments-results paper but that works in its favor. The authors make a very convincing case that there is a problem with using RNNs to decode waypoints from the latent. After that, the authors carry out meaningful experiments that lead to a clear improvement of a method on the official leaderboard.\n\n### Strengths\n\n- **S.1** Clarity. The writing is clear and concise. The structure is great. All method parts are very easy to follow. The figures are excellent and help greatly in making the point. I can't stress this point enough. Among my 5 review assignments this CoRL, this was the best read.\n- **S.2** Results. The authors submitted their results to the official WOMD leaderboard and I verified that the results match the ones reported in the paper. The improvement of 0.02 points is very notable, as this has increased the leaderboard rank of the method by one and reduced the margin to the next-best method down to 0.006.\n- **S.3** I appreciate that the negative results of using RNNs on the HDGT method (sec. 4.1) were included. It serves as a great jumping-off point for the rest of the paper.\n- **S.4** Limitation section is honest and spot-on.\n\n### Weaknesses\n\n- **W.1** Reproducibility. I wished you would've included code. A standalone Jupyter notebook on a toy example would be great for making this method accessible to others.\n- **W.2** Generality. This is already discussed in your Limitations section and I agree - the paper would be stronger with more different backends and applied to more different datasets.\n- **W.3** Robotics focus. Since this is CoRL, we reviewers are asked to mark how relevant to real-world robot experiments this method is, and right now I don't see any mention of deployment. I'd be curious what the impact of this modification is on the inference time of HDGT.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I vouch for this paper. The method is well-justified and achieves good results on a famous public benchmark.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "I already think the paper is in a very good spot and could definitely be accepted but I'd appreciate it if the authors could respond to points **W.1** and **W.3** above. In particular, a standalone Jupyter notebook would really make me excited."}}, {"review_id": "IEqVVRZLA3", "reviewer": "Reviewer_QD7m", "summary": "To better process the temporal relationship in decoding sequential locations in trajectory prediction, this work proposes to produce the future trajectory using a scratch-then-temporally-refine strategy and optimize the NN by calculating loss from accumulating velocity into coordinates. The method is applied to an existing trajectory framework [9]. The experimental validation is conducted by comparing the common prediction performance with SOTA methods' on the public leaderboard and some feasibility-related metrics with the baseline method [9].", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- this work focuses on the temporal relationship in future prediction, which is highly related to the feasibility of predicted trajectories and greatly affects the downstream PnC in autonomous driving. This is an essential but not well-addressed topic in the area.\n\nWeakness: \n- The paradigm design of so-called scratch-then-temporally-refine looks unreasonable. The refined layers' input comes from the MLP decoder, in which the paper argues the temporal relation would be lost. However, as the only information source in equation (7), how could the refine layers recover the temporal relation? How could the refine layers capture the effect from other aspects (interaction, map, etc.)? In this way, the design of refinement parts looks like a simple smoother at the end of the decoder.\n- The loss function termed temporally correlated objective loss is nothing new. Although it claims that the NN output with the form of velocity and then the loss is calculated by accumulating the velocity output as coordinates, this is exactly the same as coordinate offset prediction which is widely used in many prediction works.\n- The effect has not been fully validated. All the performance improvements (including common metrics and feasibility-related metrics) come from comparing [9] with [9]+ours, but there still exists some noticeable performance gap compared with SOTA methods.\n- What's most important is that if the effectiveness of the proposed paradigm could improve different prediction frameworks, it's good as a general design. If the proposed method can only improve a specific framework (as only validated on [9] in the paper), then this could only be regarded as incremental work. Therefore, it is necessary to validate the effectiveness in different prediction frameworks. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would not recommend this paper as there still exists critical issues around its main contributions. \n- The proposed design of scratch-then-temporally-refine discards most encoded information and thus only acts in smoothing the MLP results. \n- The loss function makes no difference with commonly used offset output. \n- The experiment cannot fully validate its effectiveness and universality. All the performance gain comes from the comparison with [9], which makes it more like incremental work.\n- Lastly, this paper's core point of temporal constraints looks overclaimed. Even with the proposed predict-and-refine paradigm and velocity output, it can hardly claim any temporal constraints are imposed.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "In addition to the main weakness, the following issues need to be addressed. \n- How about the performance of directly using scratch loss with the \"MLP + 5x1D-CNN\" structure? It could validate the necessity of \"scratch loss + refine loss.\"\n- Please claim if there is any difference between velocity output and offset output.\n- How about the TRI and UR performance of other SOTA methods? By comparing with those SOTA using MLP output, it could convince if the proposed method owns superiority in temporal-related aspects.\n- How does the acceleration and turning radius of discrete points are calculated? Are the effects of lateral and longitudinal acceleration both counted and separately considered?   "}}, {"review_id": "8T1DfrPAR5", "reviewer": "Reviewer_YWfE", "summary": "This work examines the problem of trajectory prediction.  Since most SOTA works utilize an encoder-decoder architecture with a single MLP, temporal correlation among future time-steps is ignored and therefore infeasible trajectories may be generated.  Thus far, auto-regressive decoders have not worked well empirically for trajectory prediction, but may offer some opportunity to mitigate this temporal correlation problem.\n\nThe method proposed here learns scratch trajectories using an MLP and then refines these using an RNN.  A temporal constraint loss is also used to enforce a temporal prior on the generated trajectories.  They find that the best empirical performance is achieved using a velocity prediction and coordinate prediction loss for the final objective.\n\nThe results are quite impressive when evaluated on Waymo Open Motion and the public leader-board results are compared with.  The ablations involving temporal refinement and the coordinate losses are useful and clear.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This method is very well motivated in terms of temporal correlation and the results of it are impressive.  I think the idea of using the RNN for refinement is quite novel in this problem space, but do not consider myself an expert in trajectory prediction.  The results are impressive on the public leader board. The number of baselines compared against appears to be extremely extensive, but it is hard for me to comment on the quality here since I am not familiar with most of them.  I am surprised that performance is so poor when using an auto-regressive decoder for these tasks, and this paper appears to alleviate the problem well.\n\nThe ablations are also extensive and appear to demonstrate the importance of the additional losses and refinement using the RNN.\n\nThe paper is clearly written and well positioned with literature.  The method is concisely summarized.  I could understand most of the motivation and method without much knowledge of trajectory prediction problems.\n\nI was wondering if Z-forcing was ever utilized for these auto-regressive methods for trajectory prediction (Goyal et al., 2017)? It should show some improvements over the original teacher forcing work for the auto-regressive architecture comparisons.  This may be a baseline to be compared with. I also assume the choice of auxiliary loss could introduce temporal constraint.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am not an expert in trajectory prediction, but the experimental results seem impressive and the set of baselines compared with is extensive.  The experimental procedure appears sound and the method is extensively ablated.\n\nI was able to clearly understand the motivation for this work, it identifies a clear and unsolved problem (the difficulty of training an auto-regressive architecture for this type of trajectory prediction and temporal correlation among these trajectories) and introduces a working, well defined solution.  I cannot comment as to the significance much.\n\nThis work was also easy to read and the method is concisely presented.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "-- See above: Question on Z-forcing baseline for auto-regressive architecture.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)