Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals
Authors: Nachiket Deo
Eric Wolff
Oscar Beijbom
Venue: CoRL2021 Poster (2021)
Keywords: Motion prediction
autonomous vehicles
graph neural networks

Abstract: Accurately predicting the future motion of surrounding vehicles requires reasoning about the inherent uncertainty in driving behavior. This uncertainty can be loosely decoupled into lateral (e.g., keeping lane, turning) and longitudinal (e.g., accelerating, braking). We present a novel method that combines learned discrete policy rollouts with a focused decoder on subsets of the lane graph. The policy rollouts explore different goals given current observations, ensuring that the model captures lateral variability. Longitudinal variability is captured by our latent variable model decoder that is conditioned on various subsets of the lane graph. Our model achieves state-of-the-art performance on the nuScenes motion prediction dataset, and qualitatively demonstrates excellent scene compliance. Detailed ablations highlight the importance of the policy rollouts and the decoder architecture.

Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals
Nachiket Deo
Eric M. Wolff
Oscar Beijhom²
UC San Diego
Motional
ndeo@ucsd.edi
oscar bei @motionoal com
A Abstract:
Accurately predicting the future motion of surrounding vehicles requires reason- ing about the inherent uncertainty in driving behavior. This uncertainty can be loosely decoupled into lateral (e.g., keeping lane, turning) and longitudinal (e.g.. accelerating, braking). We present a novel method that combines learned discrete policy rollouts with focused decoder on subsets of the lane graph. The policy rollouts explore different goals given current observations, ensuring that the model captures lateral variability. Longitudinal variability is captured by our latent vari- able model decoder that is conditioned on various subsets of the lane graph. Our model achieves state-of-the oath--of-hhe-att performance on the nuScenes motion prediction dataset, and qualitatively demonstrates excellent scene compliance. Detailed abla- tions highlight the importance of the policy rollouts and the decoder architecture.
Keywords: Motion prediction, autonomous vehicles, graph neural networks
Introduction
To safely and efficiently navigate through complex traffic scenes, autonomous vehicles need the abil- ity to predict the intent and future trajectories of surrounding vehicles. There is inherent uncertainty in predicting the future, making trajectory prediction a challenging problem. However, there's struc- ture to vehicle motion that can be exploited. Drivers usually tend to follow traffic rules and follow the direction ascribed to their lanes. High definition (HD) maps of driving scenes provide succinct representation of the road topology and traffic rules, and have thus been a critical component of recent trajectory prediction models well as public autonomous driving datasets.
Early work [1] encodes HD maps using a rasterized bird's 's eye view image and convolutional layers. While this approach exploits the expressive power of modern CNN architectures, rasterization of the map can be computationally inefficient, erase information due to occlusions, and require large receptive fields aggregate context. The recently proposed VectorNet [2] and LaneGCN [3] mod- els directly encode structured HD maps, representing lane polylines as nodes of graph. VectorNet aggregates context using attention [4], while LaneGCN proposes a dilated variant of graph convo- lution [5] to aggregate context along lanes. These approaches achieve state-of-the-an performance using fewer parameters than rasterization-base approaches.
The above methods represent the HD map as a graph and encode the input context into a single context vector shown in 1i The context vector is then used by multimodal prediction header [1, 6] to output multiple plausible future trajectories. The prediction header thus needs to learn complex mapping, from the entire scene context multiple future trajectories, often leading to predictions that go off the road or violate traffic rules. In particular, the prediction header needs to account for both lateral moul variability (e.g. will the driver change lane, will they turn right etc.) as well as longitudinal variability (c.g. will the driver accelerate, brake, maintain speed), This decoupling of routes and motion profiles for trajectories has been used in path planning [7, 8J, and more recently in prediction [9].
WWok done during an internship Motional.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Multimodal regression
Proposed approach
Graph representation of HD maps
Multimodal regression header
Predictions conditioned on sampled paths
Aggregating function
Context vector
Path traversed by behavior cloning pollcy
Selective aggregation
O
Figure I: Overview of our approach. We encode HD maps and agent tracks using a graph rep- resentation of the scene. However, instead of aggregating the entire scene context into a single vector and learning one-to one-to-many mapping to multiple trajectories, we condition our predictions on selectively aggregated context based on paths traversed in the graph by a discrete policy.
Our core insight is that the graph structure of the scene can additionally be leveraged to explicitly model the lateral or route variability in trajectories. We propose a novel approach for trajectory prediction termed Prediction via Graph-based Policy (PGP). Our approach relies on two key ideas.
Predictions conditioned on traversals: We selectively aggregate part of the scene context for each prediction, by sampling path traversals from learned behavior cloning policy as shown in Fig. 1. By more directly selecting the subset of the graph that is used for each prediction, we lessen the rep- resentational demands the output decoder. Additionally, the probabílistic policy leads to diverse set of sampled paths and captures the lateral variability of the multimodal trajectory distribution.
Latent variable for longitudinal variability: To account for longitudinal variability of trajectories, we additionally condition our predictions with a sampled latent variable. This allows our model to predict distinct trajectories even for identical path traversals. We show through our experiments that this translates to greater longitudinal variability predictions.
We summarize our main contributions on multimodal motion prediction using HD maps:
A novel method which combines discrete policy roll-outs with lane-graph subset decoder.
State-of- the-art performance on the nuScenes motion prediction challenge.
Extensive ablations demonstrating ability to capture lateral and longitudinal motion variations.
2 Related Work
Graph representation of HD maps: Most self-driving cars have access to HD vector maps, which include detailed geometric information about objects such as lanes, crosswalks, stop signs, and more. VectorNet [2] encodes the scene context using hierarchical representation of map objects and agent trajectories. Each component is represented as a sequence of vectors, which are then processed by local graph network. The resulting features are aggregated globally via a fully-connected graph network. LaneGCN [3] extracts a lane graph from the HD map, and uses a graph convolutional network compute lane features. These features combined with both agent and other lane features in a fusion network. Both methods utilize the entire graph for making predictions, relying on the header to identify the most relevant features.
Multimodal trajectory prediction: Researchers have proposed a variety of ways to model multiple possible future trajectories that vehicles may take. One approach is to model the output as a probability distribution over trajectories, using either regression [1], ordinal regression [6], or classification [10]. Another approach models the output Sampling methods use stochastic policy roll outs [12, 13] or latent variable models that map a latent variable sampled from simple distribution to predicted trajectory. Latent variable models are trained as GANs [14, 15], CVAEs [16, 17], or directly using the winner-takes: all regression loss [18]. These models must learn one-to-mant mo-m mapping from the entire input context (except the random variable) to multiple trajectories, and can lead to predictions that are not scene compliant.
spatial-temporal occupancy grid [11].
Coal-conditioned trajectory prediction: Rather than learning one to many mapping from the entire context to multiple future trajectories, methods such Tn' [19], LaneRCNN [20], and PEC- Net [21] condition each prediction on goals of the driver. Conditioning predictions future goals

---

makes intuitive sense and helps leverage the HD map by restricting goals to be near the lanes. How- ever, one limitation is that over moderate time horizons. there can be multiple paths that reach a given goal location. Additionally, certain plausible goal locations might be unreachable due to con- straints in the scene that are not local to the goal location, e.g., barrier that blocks turn lane. In contrast, our method conditions on paths traversed in a lane graph, which ensures that the inferred goal is reachable. Furthermore, the traversed path provides a stronger inductive bias than just the goal location. A similar stream of work conditions on candidate lane centerlines
goals (e.g., WIMP [22], GoalNet [9], CXX [23]). While the lane centerline provides more local context than just the goal, accounting for lane changes can be difficult. Additionally, routes need to be determin- istically chosen, with multiple trajectories predicted along the selected route. Our approach allows for probabilistic sampling of both routes and motion profiles. In scenes with just single plausible route, our model can use its prediction budget of K trajectories purely for different plausible motion profiles. Closest
work is P2T [24]. They predict trajectories conditioned paths explored by an IRL policy over a grid defined over the scene. However, they use a rasterized BEV image for the scene, which leads inefficient encoders and loss of connectivity information due to occlusions. Additionally, their model cannot generate different motion profiles along a sampled path.
3 Formulation
We predict the future trajectories of vehicles of interest, conditioned on their past trajectory, the past trajectories of nearby vehicles and pedestrians, and the HD map of the scene. We represent the scene and predict trajectories in the bird's eye view and use an agent-centric frame of reference aligned along the agent's 's instantancous direction of motion.
3.1 Trajectory representation
We assume access to past trajectories of agents in the scene obtained from on-board detectors and multi-object trackers. We represent the past trajectory of agent i as a sequence of motion state vectors s' s'=,,:0 = |s tt, ..., S',, over the past ty time steps. Each s = [x vl. ui, wt, I'), where x/ vi are the BEV location co-ordinates, vt. at and wi are the speed, acceleration and yaw- rate of the agent at time t, and I' is an indicator with value I for pedestrians and 0 for a vehicles. We nominally assign the index 0 to the target vehicle, and timestamp 0 to the time of prediction.
3.2 Representing HD maps as lane graphs
We represent the HD map directed graph G(V, E). The network of lane centerlines captures both, the direction of traffic flow, and the legal routes that each driver can follow. We seek to use both as inductive biases for our model. We thus use lane centerlines nodes (V) in our graph. We consider all lane centerlines within a fixed area around the target vehicle. To ensure that each divide longer lane centerlines into smaller
Nodes:
node represents lane segment of similar length, snippets of fixed length, and discretize them to a set of N poses. Each snippet corresponds to a node in our graph, with node v represented by sequence of feature vectors fi. [,, s&]- Here each f = [n yur 0% IW]. where =/ Vi and 0% are the location and yaw of the n1 pose of and I is a 2-D binary vector indicating whether the pose lies on a stop line or crosswalk. Thus, our node features capture both the geometry well as traffic control elements along lane centerlines.
Edges: We constrain edges (E) in the lane graph such that any traversed path through the graph corresponds to legal route that vehicle can take in the scene. We consider two types of edges. Successor edges (Exue) connect nodes to the next node along a lane. A given node can have multiple successors if lane branches out at intersection. Similarly, multiple nodes can have the same successor if two or more lanes merge. To account for lane changes, we additionally define proximal edges (Eprow) between neighboring lane nodes if they are within distance threshold of each other and their directions of motion are within a yaw threshold. The yaw threshold ensures that proximal edges are not erroneously assigned in intersections where multiple lanes cross each other.
3.3 Output representation
To account for multimodality of the distribution of future trajectories, we output set of K trajec- tories [Tle,, -,,,, .... Tk, for the target vehicle consisting of future x-y locations over a prediction horizon of t, time steps. Each of the K trajectories represents a mode of the predictive distribution, ideally corresponding different plausible routes different motion profiles along the same route.
3

---

Graph Encoder
Node and agent features
GRU Encodere
Agent-Node attention
CNN layors
Node
gooa
GRU
Agont ercaddngg
GRU
Motion ensoding
Policy header
Trajectory decoder
mode ensodinge ggs
Motion encuding
Scoring layers
Artion probatsities
Final mode encodings
Context
trajectory
Multfress antenton
MUP
Mition encoding
MLM
MLP
Samp path policy
vanable
Node
Edge lane graph
Incutoouppt module
Surnonnding vehicte
Figure 2: Proposed model. PGP consists of three modules trained end-to-end. The graph encoder (top) encodes agent and map context as node encodings a directed lane-graph. The policy header (bottom-left) learns discrete policy for sampled graph traversals. The trajectory decoder (bottom- right) predicts trajectories by selectively attending to node encodings along paths traversed by the policy and sampled latent variable.
4 Proposed Model
Fig, 2 provides overview of our model. It consists of three interacting modules trained end-to-end. The graph encoder (Sec. 4.1) forms the backbone of our model. outputs learned representations for each node of the lane graph, incorporating the HD map as well as surrounding agent context. The policy header (Sec. 4.2) outputs discrete probability distribution over outgoing edges each node, allowing to sample paths in the graph. Finally, attention based trajectory decoder (Sec. 4.3) outputs trajectories conditioned on paths traversed by the policy and a sampled latent variable.
4.1 Encoding scene and agent context
Inspired by the simplicity and effectiveness graph based encoders for trajectory prediction [2, 3], we seek to encode all agent features and map features as node encodings of our lane graph G(V, E).
GRU encoders. Both, agent trajectories and lane polylines form sequences of features with well defined order. We first independently encode both sets of features using gated recurrent unit (GRU) encoders. We use three GRU encoders for encoding the target vehicle trajectory s" = surrounding vehicle trajectories s' 10 and node features fT.N- These output the motion encoding hmotions agent encodings hagent and initial node encodings hijode respectively.
Agent-node attention. Drivers co-operate with other drivers and pedestrians to navigate through traffic scenes. Thus, surrounding agents serve as a useful cue for trajectory prediction. Of particular interest are agents that might interact with the target vehicle's route. We thus update node encodings with nearby agent encodings using scaled dot product attention [4]. We only consider agents within distance threshold of each lane node to update the node encoding. This allows our trajectory decoder (Sec 4.3) to selectively focus on agents that might interact with specific routes that the target vehicle might take. We obtain keys and values by linearly projecting encodings h aggent of nearby agents, and the query by linearly projecting Wg Finally, the updated node encoding obtained by concatenating the output of the attention layer with the original node encoding.

---

GNN layers. With the node encodings updated with nearby agent features, we exploit the graph structure to aggregate local context from neighboring nodes using graph neural network (GNN) layers. We experiment with graph convolution (GCN) [5] and graph attention (GAT) [25] layers. For the GNN layers, we treat both successor and proximal edges as equivalent and bidirectional. This allows us to aggregate context along all directions around each node. The outputs of the GNN layers serve as the final node encodings learned by the graph encoder.
4.2 Discrete policy for graph traversal
Every path in our directed lane graph corresponds to a plausible route for the target vehicle. How- ever, not every route is equally likely. For example, the past motion of the target vehicle approaching an intersection might indicate that the driver is preparing to make a turn rather than go straight. A slow moving lane make it likelier for the target vehicle to change lane rather than maintain lane.
We seek to learn a policy Troute for graph traversal such that sampled roll-outs of the policy cor- respond to likely routes that the target vehicle would take in the future. We represent our policy as a discrete probability distribution over outgoing edges at each node. We additionally include edges from every node to end state to allow Troute to terminate ities are output by the policy header shown in Fig. 2. The policy header uses an MLP with shared weights output scalar score for each edge (u, v) given by.
goal location. The edge probabil-
score(u, v) = MI.P (conca((hmmttoon, Brode: moode 1 (u.v)EE..))
(1)
The scoring function thus takes into account the motion of the target vehicle as well as local scene and agent context at the specific edge. We then normalize the scores using a softmax layer for all outgoing edges at each node to output the policy for graph traversal,
Troute (v/u) = softmax((score(( (), u) E E.).
(2)
We train the policy header using behavior cloning. For each prediction instance, we use the ground truth future trajectory to determine which nodes were visited by the vehicle. We can naively assign each pose in the future trajectory to the closest node in the graph. However, this can lead to erroneous assignment of nodes in intersections, where multiple lanes intersect. We thus only consider lane nodes whose direction of motion is within a yaw threshold of the target agent's pose. An edge (u,v) is treated as visited if both nodes u and are visited. We use negative log likelihood of the edge probabilities for all edges visited by the ground truth trajectory (Egt), as the loss function for training the graph traversal policy, given by
LBC
Σ (N,
-log Trouurrettee (v) ())
(3)
4.3 Decoding trajectories conditioned on traversals
Sampling roll-outs of Troute yields plausible future routes for the target vehicle. We posit that the most relevant context for predicting future trajectories is along these routes and propose a trajectory decoder that selectively aggregates context along the sampled routes.
Given a sequence of nodes [11, U2, UM] corresponding to a sampled policy roll-out, our trajectory decoder uses multi-head scaled dot product attention [4] to aggregate map and agent context over the node sequence as shown in Fig. 2. We linearly project the target vehicle's motion encoding to obtain the query, while we linearly project the node features [h hooder ..., Wn to obtain keys and values for computing attention. The multi-head attention layer outputs a context vector C encoding the route. Each distinct policy roll-out yields distinct context vector, allowing to predict trajectories along a diverse set of routes.
Diversity in routes alone does not account for the multimodality of future trajectories. Drivers can brake, accelerate and follow different motion profiles along planned route. To allow the model to output distinct motion profiles, we additionally condition our predictions with a sampled latent vector z. Unlike routes, vehicle velocities and accelerations vary on a continuum. We thus sample from a continuous distribution. We use the multivariate standard normal distribution for simplicity.
5

---

Table 1: Comparison to the state of the art on nuScenes
Model
MinADE;
MinADE1o
MissRates,2
MissRate10.2
Offrrod rate
CoverNet [10]
1.96
1,48
0.67
Trajectron++ [17]
1.88
1.51
0.70
0.57
0.25
SG-Net [28]
1.86
1.40
0.67
0.52
0.04
MHA-JAM [29]
1.81
1.24
0.59
0.46
0.07
CXX [23]
1.63
1.29
0.69
0.60
0.08
P2T [24]
1.45
1.16
0.64
0.46
0.03
PGP (Ours)
1.30
1.00
0.61
0.37
0.03
Finally, sample trajectory TLe, from model, sample roll-out of Troute and obtain Ck. sample zạ from the latent distribution and concatenate both with Amotion and pass them through an MLP to output Tit, the future locations over ty timesteps,
(LLP((Conat(/mootioo Ck. 2x)).
(4)
F1:ty
The sampling process can often be redundant, yielding similar or repeated trajectories. However our light-weight encoder and decoder heads allows us sample large number of trajectories in parallel. To obtain a final set of K modes of the trajectory distribution, we use K-means clustering and output the cluster centers our final set of K predictions [Ti,, Tay T I We train our decoder using the winner takes all average displacement error with respect to the ground truth trajectory (r91) in order penalize the diverse plausible trajectories output by model,
1
Greg mink
T2
(5)
We train model end-to-end using multi-task loss combining losses from Eq. 3 and Eq. 5,
= LBC + Creg-
(6)
5 Experiments
We evaluate our method on nuScenes [26], a self-driving car dataset collected in Boston and Singapore. nuScenes contains 1000 scenes, each 20 seconds, with ground truth annotations and HD maps. Vehicles have manually- annottted 3D bounding boxes, which are published at 2 Hz. The prediction task to use the past 2 seconds of object history and the map predict the next seconds. We use the standard split from the nuScenes software kit [27].
Dataset:
the standard metrics the nuScenes leaderboard [27]- The minimum average displacement error (ADE) over the top predictions (MinADEx). The miss rate (MissRatex;2) only penalizes predictions that are further than 2 m from the ground truth. The offroad rate measures the fraction of predictions that are off the road. Since all examples in nuScenes on the road, this should be zero. Additionally, we report metries measuring sample diversity of a set of predictions. To measure lateral diversity, we report the average number of distinct final lanes reached, and the variance of final heading angle of the target vehicle yow for the set of K trajectories. To measure longitudinal diversity, we report the variance of average speeds (a² @ppped)) and accelerations (o2.e) for the set of K trajectories.
Metrics: To evaluate our model,
Comparison to the state of the art: We report our results on the standard benchmark split of the nuScenes prediction dataset table 1, comparing with the top performing entries on the nuScenes leaderboard. We achieve state of the art results on almost all metrics, significantly outperforming the previous best entry P2T [24] on the MinADE and MissRate metrics, while achieving comparable off-road rate. This suggests that our model achieves better coverage of the modes of the trajectory distribution, while still predicting trajectories that are scene-compliant.
Encoder ablations:
We analyze the effects of our graph structure and components of the graph encoder by performing ablations on the graph encoder reported in table 2. particular we analyze the effect of including proximal edges, modeling surrounding agents with agent-node attention and

---

Table 2: Encoder ablations
Graph structure
Agent-node atlention
GNN layers
MinADEx
MissRatex.2
Offroad
Esue
E Exprax
K=5
K=10
K=5
K=10
1.35
1.03
0.64
0.41
0.04
1.33
1.01
0.63
0.38
0.03
1.30
1.00
0.61
0.37
0.03
GCN X
1.31
1.01
0.62
0.39
GCN X 2
1.31
1.01
0.61
0.39
0.04
GAT X
1.30
1.00
0,62
0.38
0.03
GAT x
1.31
1.01
0.61
0.37
0.03
Table 3: Decoder ablations
Decoder
MinADE;
MinADEjo
MissRate5,2
MissRate
Offroad
MTP [1]
1.59
1.12
0.57
0.48
0.08
Latent (LV) only
1.38
1.08
0.65
0.43
0.05
Traversal only
1.37
1.10
0.65
0.44
0.04
Goals + LV
1.33
1.02
0.60
0.42
0.06
Traversals + L.V
1.31
1.01
0.61
0.37
0.03
Table 4: Lateral diversity metrics (K=10)
Table 5: Longitudinal diversity metrics (K=10)
Decoder
# distinct final lanes
3
Decoder
doc
speed
LV only
1.22 1.41
0.11
Traversal only
2.33 4.07
5.28
Traversals LV
0.13
Traversals +
6.65
finally aggregating local context using GCN [5] or GAT [25] layers. We get improvement across all metrics by adding proximal edges, and agent-node attention, suggesting the importance of modeling lane changes and agent context. Somewhat surprisingly, adding GNN layers gives ambiguous results with GCN layers achieving slightly worse results and GAT layers performing par with the encoder without GNN layers. This could be because the multi-head attention layer aggregates context across the entire traversed path, making the GNNs redundant.
We next analyze the effect of our traversal and latent variable based decoder. We compare several decoders, all built on top of our proposed encoder with both types of edges, agent-node attention and GAT layers. First, we consider the multimodal regression header from MTP [1]. Next we consider ablations of our decoder without the graph traversals and without the latent variable conditioning. Finally, we consider a model that conditions predictions on sampled goals at different node locations, instead of traversals. Table 3 reports quantitative results while Fig. shows qualitative examples comparing the decoders. We make the following observations.
Decoder ablations:
MTP generally fares worse compared to the other decoders, particularly in terms of offroad rate. We trajectories, several veer off-road.
note from Fig. that while it generates diverse
The decoders conditioned purely on the latent variable or purely on traversals both fare worse in terms of MinADE and MissRate compared to decoder conditioned both. From the sample diversity metrics (Tables 4 and 5) and qualitative examples (Fig.3) we observe that this is for dif- ferent reasons. The 'LV only' decoder generates diverse motion profiles, but almost always predicts trajectories along single route, leading to poor lateral diversity of trajectories. On the other hand, the 'Traversal only' decoder predicts trajectories over variety of routes, but lacks diversity in terms of motion profiles.
Finally, the 'Goals + LV' decoder also fares worse compared to 'Traversals LV" decoder, again, especially in terms of off-road rate. Qualitatively, we observe that this due to two types of errors. First, it tends to predict spurious goals which aren't 't reachable for the target vehicle (Fig.3 O +), and second, while predicts correct goals, generates trajectories that don't follow accurate paths to those goals (Fig.3 2.))

---

and tracks
MTP
only
Traversal
Goals LV
Traversal LV
UMI
-
Predictions
Ground truth
Figure 3: Qualitative comparison of decoders: MTP (column 2) predicts trajectories that often veer off-road (I)-).G). The decoder purely conditioned on latent variables (column 3) lacks lateral diversity and predicts trajectories along a single route, even missing the correct route in ④ The decoder conditioned purely on traversals (column 4) predicts diverse routes, but lacks longitudinal diversity (0.2.5). Finally, the decoder conditioned on goals rather than path traversals (column 5) predicts spurious goals that may not be reachable (3) ④.. Our model (column 6) predicts scene- compliant trajectories over diverse of routes. In cases with few plausible routes (e.g.5), it uses its prediction budget of K trajectories to generate more longitudinal diversity.
6 Conclusions
We presented novel method for multimodal trajectory prediction conditioned on paths traversed in a lane graph of the HD map by discrete policy, and sampled latent variable. Through experimental analysis and ablation studies using the publicly available nuScenes dataset, we showed that
Selectively conditioning predictions on lane-graph traversals leads to trajectories that are (i) di- verse in terms of routes, and (ii) precise and scene compliant with the lowest offroad-rates. Additionally conditioning predictions on sampled latent variables leads to trajectories that are diverse in of motion profiles.
Both put together lead to state of the art results in terms of MinADE and MissRate metrics.

---

References
[1] H. Cui, V. Radosavljevic, F. Chou, T. Lin, T. Nguyen, Huang, J. Schneider, and N. Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In International Conference on Robotics and Automation (ICRA), 2019.
[2] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid. Vectornet: Encoding hd maps and agent dynamies from vectorized representation. In IEEE/CVF Conference on Computer Vision and Pattern Recogmition (CVPR), 2020.
[3] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forecasting. In European Conference on Computer Vision (ECCV), 2020.
[4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, E. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems (NeurIPS), 2017.
[5] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
[6] Y. Chai, B. Sapp. M. Bansal, and D. Anguelov. MultiPath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In Conference on Robot Learning (CoRL), 2019.
[7] B. Paden, M. Čáp. S. Z. Yong. D. Yershov, and E. Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on intelligent vehicles, 1(1):33-55, 2016.
[8] S. M. Valale. Planning algorithms. Cambridge university press, 2006.
[9] I.. Zhang, P.-H. Su, J. Hoang, G. C. Haynes, and M. Marchetti-Bowic Map-adaptive goal- based trajectory prediction. In Conference on Robot Learning (CoRL., 2020.
[10] T. Phan-MMnh, E. C. Grigore, F. A. Boulton, O. Beiijoom, and E. M. Wolff. Covernet: Multi- modal behavior prediction using trajectory sets. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[11] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[12] N. Rhinehart, K. M. Kitani, and P. Vernaza. R2P2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In European Conference on Computer Vision (ECCV), 2018.
[13] N. Rhinehart, McAllister, K. Kitani, and Levine. PRECOG: Prediction conditioned on goals in visual multi-agent settings. In IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[14] A. Gupta, J. Johnson, L.. Fei-Fei, S. Savarese, and Alahi. Social gan: Socially acceptable tra- jectories with generative adversarial networks. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2018
[15] T. Zhao, Y. Xu, M. Monfort, W. Choi, C. Baker, Y. Zhao, Y. Wang, and Y. N. Wu. Multi- agent tensor fusion for contextual trajectory prediction. In IEEE/CV Conference Computer Vision and Pattern Recognition, 2019.
[16] N. Lee, W. Choi, P. Vernaza, C. Choy, P. Torr, and M. Chandraker. DESIRE: prediction in dynamic scenes with interacting agents. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Distant future
[17] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically trajectory forecasting with heterogeneous data. In European Conference on Computer Vision (ECCV), 2020.

---

[18] O. Makansi, E. 1lg, O. Cicek, and T. Brox. Overcoming limitations of mixture density net- works: A sampling and fitting framework for multimodal future prediction. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[19] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, C. Li, and D. Anguelov. Tnt: Target-driven trajectory prediction. In Conference on Robot Learning (CoRL), 2020.
[20] W. Zeng, M. Liang, R. Liao, and R. Urtasun. Lanerenn: Distributed representations for graph- centric motion forecasting. arXiv preprint arXiv:2101.06633 2021.
[21] K. Mangalam, H. Girase, S. Agarwal, K.-H. Lec, E. Adeli, J. Malik, and A. Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. European Conference Computer Vision (ECCV), 2020.
[22] S. Khandelwal, W. Qi, J. Singh, A. Hartnett, and D. Ramanan. What-if motion prediction for autonomous driving. arXiv preprint arXiv:2008. /0587, 2020.
[23] C. Luo, L. Sun, D. Dabiri, and A. Yuille. Probabilistic multi modal trajectory prediction with lane attention for autonomous vehicles. arXiv preprint 2xXiv:0077.2574, 2020.
[24] N. Deo and M. M. Trivedi. Trajectory forecasts in unknown environments conditioned grid-based plans. arXiv preprint XiXi::000000073.. 2020.
[25] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks. International Conference on Learning Representations (ICLR), 2018.
[26] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, E. Liong, Q. Xu, Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuScenes: A multimodal dataset for autonomous driving. arXiv preprint 19Xiv: 1027, 2019.
[27] nuScenes Contributors. nuScenes.
org/ 2020.
[28] C. Wang, Y. Wang, M. Xu, and D. J. Crandall. Stepwise goal-driven networks for trajectory prediction. arXiv preprini arXiv:2103. 14107, 2021.
[29] K. Messaoud, N. Deo, M. M. Trivedi, and Nashashibi. Trajectory prediction for autonomous driving based on meati-head attention with joint agent-map map representation. In IEEE Intelligent Vehicles Symposium (IV), 2021.
10


Reviews:
[{"review_id": "bY-oiK5bP3B", "reviewer": "Reviewer_NjKJ", "summary": "The paper tackles the problem of agent prediction, an extremely important problem for autonomous driving. The authors leverage the fact that the graph structure of the underlying road scene can be exploited for making trajectory predictions, and they demonstrate how this can be done efficiently by decomposing the trajectory into the lateral and longitudinal components together with clever sampling of candidate trajectories and some other tricks. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n[S1] The results seem solid and the ablations are extensive\n\n[S2] From what I can tell, the approach seems novel and the extensions over prior art are well-explained and make intuitive sense\n\nWeaknesses:\n\n[W1] I find some statements hard to understand, or that they lack precision and clarity. E.g.: \"Since the entire scene is aggregated together and used to compute all predictions, it places a strong requirement on the output header\" what requirement?\n\n[W2] There seem to be a lot of parameters and hyperparameters in the approach. E.g. \n - \"We consider all lane centerlines within an area of [-50, 50] m laterally and [-20, 80] m longitudinally around the vehicle of interest. We divide the lane centerlines into snippets of length ≤ 20m, and discretize them to a set of N poses with 1m resolution\" \n - \"For each node v, we only consider nearby agents within 10m of the lane centerline\" \n - \"with direction of motion within a yaw threshold of the pose\"\nThe prevalence of all of these heuristics calls into question the robustness of the proposed approach. For each of these choices it would strengthen the work to have some more context or analysis of the sensitivity of the method to the parameters.  \n\n[W3] The method seems quite complicated and the architecture used is quite complex with many learned compenents. It is fine that it takes a complex solution to solve a complex problem, but I feel the manuscript could do a better job of elevating the discussion so that we may learn some more generally applicable lessons rather than the step by step recipe for solving the exact problem that the authors set out to solve. \n\n\n[W4] Minor issues:\nprobabilitic -> probabilistic\n\"encodes the scene context a hierarchical representation of\" a word is missing somewhere\n\"fig 3\" vs. \"figure 3\" be consistent. \n\"the model tend\" -> the model tends\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In summary, I think that the approach seems novel and the results demonstrate its effectiveness. However, I think that the work could be elevated and that the clarity of the exposition could be improved. At present, it comes across as clever set of heuristics and a particularly good job of engineering the components to work together. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": " - Sensitivity analysis with respect to parameter choices\n - Improved clarity of description of method that separates the core generalizable ideas from the heuristics and implementation details. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "OEIKqWu-U4", "reviewer": "Reviewer_hmkE", "summary": "This work focuses on the task of trajectory prediction in a driving context in the presence of multiple road uses. Their approach relies on decomposing the scene into a spatial graph which encodes legal and plausible transitions from one location to another. The key to their approach is that they train a probabilistic prediction policy to generate candidate trajectories in the graph, rather than in coordinate space, and are later transformed to coordinate sequences with another sampling step. By generating probabilistic predictions in graph space and final predictions conditioned on the graph sequences, they more cleanly separate spatial modes.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper demonstrates state of the art prediction results on the nuScenes benchmark.\nThey perform an ablation study to demonstrate the effectiveness of various aspects of their approach\nWeaknesses:\nThe paper is selling the premise that their approach of conditioning the final prediction on an encoded graph traversal is the secret sauce. However, in their ablation study they show that conditioning instead on a goal is only slightly worse (1-2cm avg displacement error). If anything their own results show that conditioning the final prediction on a latent variable contributes just as much to their final performance. In fact, all variants in their ablation study outperform the state of the art listed in their baselines.\nWould be nice to see performance on other baselines", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Authors propose novel trajectory prediction approach conditioned on probabilistic graph traversals and achieve state of the art on the nuScenes benchmark. However, from the ablition study it's unclear how much impact the graph traversal truly has. The impact could perhaps be better highlighted by implementing a more diverse set of benchmarks.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "I think readers would benefit from a more careful analysis of what makes their approach perform so well. It could be that the success depends less on the novel aspects proposed in the current paper and more on other aspects (i.e. use of transformer, etc...)"}}, {"review_id": "FW9oJaiU8k", "reviewer": "Reviewer_qYAi", "summary": "This paper focus on the vehicle trajectory prediction, which is one of the necessary parts of autonomous driving technology. In this condition, a future trajectory of a vehicle is predicted based on the environment (e.g., a map of road lane) and past trajectories of other vehicles or pedestrians nearby. While this predicted trajectory is expected to meet the certain constraints such as traffic rules and road topology, it is also involved in uncertainties from other vehicles and pedestrians or drivers’ own behaviors. In this paper, instead of analysing the interaction of these uncertainties with a vehicle, authors decouple them into two parts at the trajectory stage: the lateral changes (e.g., keeping lane, turning) and the longitudinal changes (e.g., accelerating, braking). A model named PGP is therefore raised in this paper, where the lateral variability is captured by a selectively aggregate part with behavior cloning policy based on the features from a graph encoder, and the longitudinal variability is presented by a latent variable model. The prediction on nuScenes dataset shows a SOTA performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "An encoder-decoder structured model with a policy header included is proposed. The model performs a SOTA results on nuScenes dataset. However, the hypothesis “decoupling uncertainty into lateral and longitudinal” (mentioned in line 3-5) , as the precondition of the method in this paper, has not been discussed by authors. Another main issue is that authors stress two key ideas, predictions conditioned on traversals by behavior cloning and latent variable for longitudinal variability, while these two ideas have not been fully discussed and analysed in the paper, which makes the paper lose focus on its main cocepts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper fails to fully discuss its key ideas and show the relationship between the SOTA performance and its ideas. It is recommended to reorganize the paper and focus more on these ideas than other details of medel.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1.\tThe caption of figure 3 is incomplete. Lacking of explanation of figures.\n2.\tAs a precondition of the method proposed in this paper, decoupling uncertainty into lateral and longitudinal (mentioned in line 3-5) is not fully discussed and analysed, or supported by other literature.\n3.\tOne of the two key ideas, predictions conditioned on traversals, is implemented by sampling path traversals from a learned behavior cloning policy (mentioned in line42-47). However, this key idea is insufficiently discussed, only few details is shown in section 4.2 from line 163 to line 178. It could be more convincing if the structure of SVM used in section 4.2 (mentioned in supplementary material), or the formula of loss function used in line 178, and any other necessary details are included in this paper.\n4.\tOne of the two key ideas, latent variable for longitudinal variability, is seemed to be only discussed in section 4.3 from line 190 to line 196. It could be more convincing if the literature of latent variable, or the reason why vector z is sampled from multivariate standard normal distribution, or the comparison with latent variable part in other SOTA models, and any other necessary details are included in this paper. \n5.\tThe multi-agent behavior is a main topic in vehicle trajectory prediction. It is implicitly or explicitly considered in SOTA models (e.g., the model Trajectron++ mentioned in comparison). The model PGP proposed by this paper achieved SOTA results. Therefore, it is expected to indicated how these two key ideas or other features of PGP help to describe and analyse the multi-agent system in trajectory prediction.\n6.\tIn figure 3, as it is described from line 258 to line 260, the model performance seems to be different in a straight road and different types of road junctions. However, this difference is not discussed."}}, {"review_id": "393F1rSM8dV", "reviewer": "Reviewer_HKDg", "summary": "The authors propose a method for multimodal trajectory prediction using a graph neural network. They decouple the predicted trajectories components as lateral and longitudinal behaviors and achieve the lateral variability that aggregates part of the scene context by sampling path traversals while ensuring longitudinal diversity via an added latent variable. The experiments are well-presented, and the benchmark evaluations are convincing.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method utilizes the road content as lane graphs. Therefore, the generated trajectories are refined within the road ranges with lower off-road rates. The methodology is concise and with good performance compared with the state-of-the-art on a public dataset and explains the observations of the results. Overall, the paper is sound and well-structured. However, it is unclear about the lateral and longitudinal behavior division on these two decoders, and the implementations about decoder comparison are confusing. There are also some issues to clarify or revise.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper has done a good job but there are still some weaknesses and issues to solve. But I still recommend this paper to be published.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- Abstract:\nThe statement \"Accurately predicting ... inherent uncertainty in goals and driving behavior. This uncertainty can be loosely decoupled into lateral (e.g., keeping lane, turning) and longitudinal (e.g., accelerating, braking). \"It mentions the inherent uncertainty in goals and driving behavior, but the second sentence only explains the uncertainty of driving behaviors.\n\n-Introduction, line 46, \"Probabilitic\" Is it a typo?\n\n- \"Fig. 1.\", \"Figure 2\", \"figure 3\", \"fig. 3\" Please be consistent with the notations.\n\n- 4.3 Decoding trajectories conditioned on traversals\n\nThe proposed method directly linearly propagates vehicles' motion with latent variable z to achieve longitudinal resilience instead of using the states of the predicted trajectories. But, does it is accurate to qualitatively represent the longitudinal movements?\n\n- Figure 3 is confusing and too small to distinguish the difference. It is recommended to add some notations.\n\n- It is recommended to integrate some details of the supplementary materials into the main content.\n\n- The gif is good to illustrate the information of the proposed method, and it is recommended to add different scenarios with other lateral behaviors."}}]

Author rebuttal:

Meta-review: This paper proposes a novel method for multimodal trajectory prediction. While the reviewers agree that this paper contains some interesting ideas and the proposed method is effective, they also raise a number of important concerns. The authors should carefully address the reviewers' comments in their rebuttal. 

UPDATE POST DISCUSSION PHASE: I would like to thank the authors for their comments and clarifications during the discussion phase. The reviewers generally agree that this paper provides a valuable contribution, and I concur with this assessment. In the final version of this paper, the authors should carefully address all reviewers' comments and suggestions, in particular regarding hyperparameter sensitivity and exposition clarity. 

Decision: Accept (Poster)