Leveraging Future Relationship Reasoning for Vehicle Trajectory Prediction
Authors: Daehee Park
Hobin Ryu
Yunseo Yang
Jegyeong Cho
Jiwon Kim
Kuk-Jin Yoon
Venue: ICLR 2023 poster (2023)
Keywords: Trajectory prediction
Autonomous driving
Neural relation inference
Stochasticity modeling
Multimodal prediction

Abstract: Understanding the interaction between multiple agents is crucial for realistic vehicle trajectory prediction. 
Existing methods have attempted to infer the interaction from the observed past trajectories of agents using pooling, attention, or graph-based methods, which rely on a deterministic approach. 
However, these methods can fail under complex road structures, as they cannot predict various interactions that may occur in the future. 
In this paper, we propose a novel approach that uses lane information to predict a stochastic future relationship among agents. 
To obtain a coarse future motion of agents, our method first predicts the probability of lane-level waypoint occupancy of vehicles. 
We then utilize the temporal probability of passing adjacent lanes for each agent pair, assuming that agents passing adjacent lanes will highly interact. 
We also model the interaction using a probabilistic distribution, which allows for multiple possible future interactions. 
The distribution is learned from the posterior distribution of interaction obtained from ground truth future trajectories. 
We validate our method on popular trajectory prediction datasets: nuScenes and Argoverse. 
The results show that the proposed method brings remarkable performance gain in prediction accuracy, and achieves state-of-the-art performance in long-term prediction benchmark dataset.

Published as a conference paper at ICLR 2023
LEVERAGING FUTURE RELATIONSHIP REASONING FOR VEHICLE TRAJECTORY PREDICTION
Dachee Park', Hobin Ryu², Yunseo Yang', Jegyeong Cho', Jiwon Kim', Kuk-Jin Yoon Koorea Advanced Institute of Science and Technology "NAVER LABS (bag2824, acorn, j2k0618, k kjyoon) J @xaist. ac. (hobin. ryu, gl kim)
com
ABSTRACT
Understanding the interaction between multiple agents is crucial for realistic vehicle trajectory prediction. Existing methods have attempted to infer the interaction from the observed past trajectories of agents using pooling, attention, graph-based methods, which rely on a deterministic approach. However, these methods can fail under complex road structures, they cannot predict various interactions that may occur the future. In this paper, we propose a novel approach that uses lane information predict stochastic future relationship among agents. To obtain coarse future motion of agents, our method first predicts the probability of lane- level waypoint occupancy of vehicles. We then utilize the temporal probability of passing adjacent lanes for each agent pair, assuming that agents passing adjacent lanes will highly interact. We also model the interaction using probabilistic distribution, which allows for multiple possible future interactions. The distribution is learned from the posterior distribution of interaction obtained from ground truth future trajectories. We validate our method on popular trajectory prediction datasets: nuScenes and Argoverse. The results show that the proposed method brings remarkable performance gain in prediction accuracy, and achieves state-of- the-art performance long-term prediction benchmark dataset.
INTRODUCTION
For safe autonomous driving, predicting vehicle's future trajectory is crucial. Early heuristic prediction models utilized only the past trajectory of the target vehicle [Lin et al. (2000); Bartth & Franke (2088). However, with the advent of deep learning, more accurate predictions can be made by also considering the vehicle's s relationship with the High-Definition (HD) map [Liang et al. 20200) Zeng al. 2)2T)) surrounding agents (Lee et al. (2017); Chandra al. (2020). Since surrounding vehicles are not stationary, predicting relationships with them is much more complicated and has become essential for realistic trajectory prediction. Furthermore, since individual drivers control each vehicle, their interaction has a stochastic nature.
Previous works modeled interaction from past trajectories of the surrounding vehicles by employing pooling, multi-head attention, or spatio-tempon graph methods. However, we observed that these methods easily fail under complex road structures. For example, Fig. shows the past trajectories of agents (left) and the attention weights among agents (right) obtained by previous method (Mercat) al. (2020) that learned the interaction among agents using multi-head attention (MHA). Since agents O and 4 are expected to join in the future, the attention weight between them should be high. However, the model predicts low attention weight between them, highlighting the difficulty of reasoning future relationships between agents based solely on past trajectories. Incorporating the road structure should make the reasoning process much easier.
The eecisionnmaaingg process of human drivers can provide insights on how to model interaction. They first set their goal where they are trying to reach on the map. Next, to infer the interaction with surrounding agents, they roughly infer how the others will behave in the future. After that, they infer the interaction with others by inferring how likely the future path of other vehicles will overlap the path set by themselves. The drivers consider interaction more significant the more the future paths of other vehicles overlap with their own. We define the interaction from this process as a "Future Relationship". We use the following approaches model Future Relationship, shown in Fig. 3

---

Published as a conference paper at ICLR 2023
0
Figure 1: Past trajectories and corresponding attention map between agents from previous work (Mercat al.) (2020)). A weak relationship is inferred between agents that will highly interact in the future: agents and
Figure 2: Lane segments represented in different colors.
Pas Tojectslis
Waypeler Clocpanncy (Agyropadd thed]
Tranraction odgo
apoes
Figure 3: Key concept of the proposed method: From past observed trajectories, we predict the lane that a vehicle will pass in the future. The Interaction between agents is represented by an edge connecting their nodes, and determined by the probability that two agents will pass adjacent lanes. The greater the probability, the higher the expected interaction.
First, obtain the rough future motion of all vehicles in the scene. Since vehicles mainly move along lanes, we utilize lane information as strong prior for representing the rough future motion of vehicles. Because lane centerlines contain both positional and directional information, rough future motion can be represented as waypoint occupancy. The waypoint occupancy is defined as the probability of a vehicle passing specific lane segment at every intermediate timestep. In the middle of Fig. 3 each agent's ss waypoint occupancy is shown. Here, we aggregated the temporal axis for simplification. The probability that the vehicle passes that lane during the prediction horizon is drawn using the tone of red color.
Second, based on waypoint occupancy, we infer the Future Relationship in probabilistic distribu- tion. In most vehicle trajectory prediction methods, the interaction between agents is still made in deterministic manner. However, we take note that interaction between vehicles is highly stochastic, and there can be multiple possible interactions. The deterministic relation inference averages out diverse interactions, interrupting socially-aware trajectory prediction. Therefore, we define Future Relationship in Gaussian Mixture (GM) distribution. Motivated by Neural Relational Inference (NRT) (Kipf et al. 2018)), we propose method to train the diverse interaction distribution explicitly.
In summary, contributions
1) We propose a new approach for modeling the interaction between vehicles by incorporating the road structure and defining it Future Relationship.
2) We propose to infer the Future Relationship in probabilistic distribution using Gaussian Mixture (GM) distribution capture diverse interaction.
3) The proposed method is validated on popular real- wolld vehicle trajectory datasets: nuScenes and Argoverse. In both datasets, there remarkable improvement in prediction performance and state-of-the-art performance achieved in the long-range prediction dataset, nuScenes.
RELATED WORK
2.1 GOAL-CONDITIONED TRAJECTORY PREDICTION
Predicting future trajectories at once is a challenging task. Instead, goal-conditional prediction, which samples goal candidates and then predicts trajectory conditioned on them, is helpful and has shown
2

---

Published as a conference paper at ICLR 2023
state-of oftthe-rtttttttt performance, especially in long- Long-range prediction tasks Zhao et al. (2021); ); Gu et al. (2021) Phan-Minh et al. (2020); Chai et al. (2020): Zhang et al. (202T)) CoverNet Phan-Minhh et al. (2000) and MultiPath [Cha et al. (2020g), which quantize the trajectory space set of anchors, often generate map-agnostic trajectories that cross non-drivable areas because the surrounding map is not considered. Recently, several studies have exploited the map information to obtain more performant goal candidates based on the assumption that vehicles follow lanes, TNT (Zhao et al. (202T) uses goal points sampled from a lane centerline, and GoalNet ( (Zhang et al. (202T) lane segments as trajectory anchors. However, while previous methods assume that the likelihood of arriving at a final destination random, they assume that trajectories are unimodal in order to reach a
)) uses
specific goal area. In this paper, we assume inherent uncertainty in which trajectories can vary due to the interactions with surrounding vehicles order to reach a specifie goal area.
2.2 INTERACTION MODELING
Considering the interaction between agents helps to predict socially aware trajectory. In the very early stage, interaction is obtained by pooling interaction features the local region (Deo & Trivedi
(2018); Gupta et al. (2018)) In other works, researchers attempted to obtain interaction through attention-based ( (N Ngiam et al. (2022); Mercat et al. 2020); Vemula et al. (2018) or GNN-basee
method ( CCarrasco et al. (2021); Cao al. 2021); Zeng et al. (2021) Casas et al. (2020): : Liang
itt al. 2020); Gao et al. (2020)) ). However, in most previous methods, interactions between agents are learned only with regression loss, which is insufficient to represent dynamic and rapidly changing situations. There exists a line of works that employs Neural Relational Inference (NRI) (Kipf et al.
(2018) that explicitly predicts and learns interaction using a latent interaction graph. EvolveGraph (Li et al. (2020)) utilizes two interaction graphs, static and dynamic, and NRI-MPM (Chen et al. (202I) uses a relation interaction mechanism and ppatio-temporal message passing mechanism. Similarly, we apply the NRI-based method to predict and train the interaction explicitly.
2.3 MULTI-MODAL TRAJECTORY PREDICTION
Trajectory prediction is stochastic problem, which means that there are multiple possible futures instead of unique answer. Recently, deep generative models like GAN (Goodíellow et al. (2014)) or
VAE (Kingma Welling ) have been employed to address this issue. GAN-based (Gupta et al.)
(2018); Kosaraju et al. 22019] Li et al. (2021b)) and VAE-based models (Ivanovie & Pavone (2019);
Salzmann et al. (2020;; Tang& & Salakhutdinov (019)) predict multiple futures by sampling multiple latent vectors. A orgl-onga latent space is necessary to sample meaningful latent vectors for predicting diverse, yet plausible future trajectories. This has become a natural choice in recent works Ma et al. (2021): Bae et al. (2022)). The work most closely related to ours is GRIN (Li al.) (2021a)) which argues that muuti--modllity in trajectory prediction comes from two sources: personal intention and social relations with other agents. However, GRIN only considers past interaction, while we propose to consider future interaction by taking into account the characteristics of vehicle motion. Since vehicle motion mainly follows lanes, we utilize lane information infer future interactions.
FORMULATION
In each scene, the past and future trajectories of N vehicles are observed. The past trajectory x, consists of positions for -t, : timesteps before the current timestep, and the future trajectory x consists of positions for 1 ty timesteps after the current timestep (f 0). Lane information obtained from the HD map, which consists of M segmented lane polylines. The lane information is represented as a graph: g = e), where the nodes (é) correspond to the different lane segments, and the edges (e) represent the relationships between the segments. There are five relationship between segments: predecessor, successor, left/right neighbor and in-same-inte -nteeeiitersection The input to the model is denoted as X, which consists of the past and future trajectories of the vehicles and the lane information. Here, the future trajectories is only used in training. The output of the model is denoted as Y, which consists of F predicted future trajectories for each agent. The model also predicts the future lane occupancy (i.c., which vehicles are occupying which lanes) as a medium using a probability distribution T for each vehicle and lane segment at each future timestep: 1 ty. The predicted future lane occupancy is denoted as T and the ground truth future lane occupancy is denoted as T.-
3

---

Published as a conference paper at ICLR 2023
Trajectory (
Waypoint Occupancy (Kiep)
Trajectory (C")
x-1
x
Waspoint Profiction
agenE
agient
1gg
sample
Inieation Feature
Lane Graph &
Future Relation (FRM)
Staupe
massage passing
Internction Feature (hx)
Decoder
Only
F Trajectory Samples (V)
Inferaction edge
x,
Figure 4: Overall structure of the proposed method. Given past/future motion inputs, the waypoint occupancy (T (Tayt) ) is obtained. The goal features are then sampled following Tzy, Intention feature is derived from the goal features and the past motion (h The Future Relationship Module (FRM) utilizes the intermediate waypoint occupancy (7 (1tt to sample the interaction edges among agents. Message passing is then performed to obtain the interaction feature. Finally, the decoder predicts F future trajectories from concatenation of intention and interaction features.
METHOD
Our focus is modeling the "Future Relationship" between agents. A naive method to infer the Future Relationship is to predict all future vehicle trajectories and then calculate similarity among them. However, this method is inefficient and redundant, as it requires performing prediction twice. Moreover, the criteria for calculating similarity between trajectories may not be clear. In this paper, we utilize lane information for modeling the Future Relationship. inspired by the idea that the vehicles mainly follow lanes. Our key idea is that if two vehicles are expected to pass on adjacent lanes, they will have a high chance of interacting in the future.
We present the overall structure of our method in Fig. First, we predict the waypoint occupancy, which represents the probability a vehicle passing a specific lane segment during future time steps 1 f (Sec. 4.1. Using this information, our Future Relationship Module (FRM) infers interaction as an edge feature connecting agent node pairs (Sec. 4.2). These interaction edges are used transfer information between agent nodes to form the interaction feature through message passing. Finally, in the decoding stage (Sec. 4.3 the decoder predicts future trajectories from the aggregation of the interaction feature and intention feature, which is derived from the concatenation of past motion and goal features. Following AgentFormer (Yuan et al. (202T), our method is based on CVAES where the condition corresponds the intention, and the latent code corresponds to the interaction feature. Then, we compute prior and posterior distribution of the interaction feature, as described in Sec. 4.2.2 4.2.3
4.1 WAYPOINT OCCUPANCY
In this section, we describe how to obtain the waypoint occupancy. We need two waypoint occu- pancies: predicted from past trajectory (T,) and the ground truth (r for obtaining prior and posterior distributions of interaction, respectively.
To predict the waypoint occupancy from past trajectory, first encode the past trajectories and the lane graph 9 into past motion and lane features: h hg. Then, following TNT Zhao et al. (2021)), we predict the waypoint occupancy Eq. (1). Here, [ denotes concatenation, and apply softmax to ensure that the waypoint occupancy sum up to one: > =1
1:ty softmax(MLP((H) hx))) E NxMXI
For GT waypoint occupancy, we can directly obtain it from GT future trajectory since we know the position and heading of vehicles. More details can be found in the supplementary material.
4

---

Published as a conference paper at ICLR 2023
Only
Successor Professsaor
Lame Connactivity
AZ
Transuoe
Naglèvr intosscction
F
Doc Product
GCN
Fg
occupancy
Waypoint
oscepenccy
waypoire
Inter-agea: PR-/*
prosimity
Preritust
Figure 5: Future Relationship Module. During inference, predicted waypoint occupancy (T is fed to GCN, dot-producted by itself to obtain inter-agent proximity (PR~). Prior of interaction (TK, μ.. ak) then obtained as Gaussign Mixture. During training. GT waypoint occupancy (r fed to obtain posterior of interaction (H, Gaussian distribution.
4.2 FUTURE RELATIONSHIP MODULE (FRM)
Fig. 5 shows the FRM, which consists of three parts: computing inter-agent proximity and obtaining posterior and prior distribution. From intermediate waypoint occupancy of vehicle (Ti:t,-1), compute how each pair of vehicle pass adjacent lanes adjacent to each other at each timestep (inter-agent proximity). Based on that information and agents' past motion features, we obtain two distribution of interaction. In the following sections, we describe the details of each part.
4.2. IGENN-AGENT PROXIMITY
To compute the inter-agent proximity (PR), we first smooth the waypoint occupancy using Graph Convolutional Network (GCN) (Welling & Kipf (2016)). The reason for doing so is that when a vehicle passes specific lane, it affects other vehicles that pass the adjacent lane, not necessarily the same lane. Therefore, we apply different smoothing for each lane connectivity (predecessor, successor, neighbor, hi-the-same-intereeetiie layer aggregated information from neighboring lanes and applies a non-linear transformation. This allows the model capture spatial dependencies among agents and improve the accuracy of the inter-agent proximity computation. Each layer is expressed as (2) where a, D.- A, and W, are softmax followed by ReL.U, degree, adjacency and weight matrix for each edge type, respectively.
by employing 2-hop GCN layers. Specifically, each
-1
Σ eE suce.pred, fitht.
о (D -
NXM(((()
With this smoothed waypoint occupancy, we can compute the inter-agent proximity using the dot product -1:t,-1 across the lane axis.
PR F1a-1 (f14y-1)
NxNx(1-
(3)
4.2.2 PRIOR OF THE INTERACTION
To obtain the prior distribution, we use the past motion features (h₂) and inter-agent proximity. In this subsection, omit superscript for simplification. There are two design factors for our interaction modeling: (i) interaction should reflect diverse and stochastic properties, and (ii) it occurs in every pair of vehicles. Consequently, the prior distribution defined as Gaussian Mixture (GM) agent pair. Then, we define interaction edge e between agent and as a aassaaddiimmmm d-dimensional feature (po(e)X) ~ I (uR, lo following GMVAE (Dilokthanakul et al. 2016)). The distribution I:N,1:N are obtained from the neural network Fg: [uk. K. k
parameters (μK. o K. HK
1 k. 2 (pr 14. bg))
TR Kdd TR p
5

---

Published as a conference paper at ICLR 2023
F6 is composed of MLP layers and 1-d conv layer (Deo & Trivedi (2018)). We then perform two sampling steps, one for the interaction mode & (from T and one for e (from Gaussian noise). This allows for K distinct interactions modes:
Mk & argmax, (TK + +g), g Gumbel(0,1)
(5)
Mk + OFE E &BNNNNx e~ (O.I))
(6)
Next, we compute the interaction feature (h = { I:N) via message passing from sampled interaction edge, as follows:
hiR 0'( N-
d @ (2)))) jti
Σ
g
4.2.3 CVAE POSTERIOR
To obtain the posterior distribution, use GT waypoint occupancy (T") and the future motion feature (ht), which is obtained from GT future trajectory and same motion encoder with past trajectory. Similarly, omit superseript in this subsection. Inter-agent proximity is obtained with same procedure in Eqs. (3) The difference from the prior is that the posterior is modeled in a single Gaussian (Ht (1") 11:::N,::N), ). Thus, Fg is replaced with Pe:
1 00,0 = Fo h h(l) E RRR Rd
(8)
Then we sample I), and interaction edge is obtained: 2 = ac. Finally, following Eq. 7). interaction features (h is obtained.
4.3 DECODER
The decoder predicts future trajectories from the aggregation of the interaction feature (hn) and intention feature (h,). Here, the intention feature is obtained from past motion feature and goal feature following TNT. During training. the unique GT intention feature repeated F times, and we sample the interaction feature (hzp) F times from the posterior distribution. During inference, the intention feature is obtained from the past motion feature (h₂ and goal feature (h,), which is sampled F times from predicted waypoint occupancy at the final timestep (Ti,). The interaction feature ( is sampled F times from the prior distribution. The decoder composed of 2-layer MIP and predicts sequence of x.y coordinates. More details can be found in the supplementary material.
4.4 TRAINING
Because the GT waypoint occupancy (I+) available, we can train the model to predict waypoint occupancy (T using negative likelioel (NLL): Luul - =ogloggT
However, since the interaction edge ze is unobservable, we optimize the evidence lower bound (ELBO) to train the interaction distribution using the CVAE scheme.
ELBO = Bgg (Pog(po(Y X, T] + (Loga X,T) || pa(ze X,T)]
Here, ge is the approximate posterior, and pe is the prior. Since our model only allows the posterior to be Gaussian distribution, we can simplify the Kullback-Libler (KL) divergence term as follow:
LKL = - ll Po] ze log Σ Teexp(-Kii (I || po.:))
(10)
The detailed derivation with the reparameterization trick can be found in the supplementary material. However, a common drawback with the NRI-based method is the "degenerate" " issue, where the decoder tends to ignore the relation edge during training. To address this issue, we train the network to give different roles to the intention and interaction features. Since the GT trajectory is conditioned on the GT goal feature, training strategy restricts the role of interaction edge to momentary motion, resulting in the following reconstruction loss: Lyecon ming, [E]log(p> (Y 2x2,:*
use the GT goal feature compute the reconstruction term. This
Finally, the overall loss the sum of the three losses, which are trained jointly: Latt = Lall + LKL Lrecon-

---

Published as a conference paper at ICLR 2023
Table 1: Comparison on nuScenes test set. Best in bold, second best in underline.
Table 2: Comparison on Argoverse val/test set. Best in bold, second best in underline.
rejjectron**
Paper
mADE;
1.88
mADE11
1.51
MR;
0.70
1R mFDE;
0.57
9.52
Paper
mADEs
Val set
mFD55
mADEE
Test set
mi mFD55
Salzmann al.
2020
TNT
221
Deo & Trivedi
2020
1.45
1.16
0.64
0.46
10.5
Zhao
LancKt NN
0.73
1.29
0.94
1.54
AgentFornns
1,86
1.45
Zong
2021
1.19
TPC.N
LaPred
0.53
Yo et
2021
1.15
Kim et al
2021
MalnFaah
Girgis al.
2021
0.73
1.10
0.89
1.41
Chai ct al
2020
GOROME
Gilles et
2022a
1.42
1.15
0.57
0.47
6.99
S0000 Trahtst
Lio et ml
202
0.72
1.21
0.80
0.84
1.34
1.23
1.37
1.03
0.62
0.44
8.19
Varadarajan
2022
Girgis al.
2021
Miltipalii+
]HOMAAS
PGP
Gilles
Deo
2022
2022h
1.27
1,33
0.94
1.04
0.55
0.52
0.42
0.34
6.71
7.17
IV
Baselme
Varadarajan
3022
2022
0.71
0.66
0.96
1.03
0.79
0.77
0.86
1.17
1.21
1.30
Ours
1.18
0.88
0.48
0.30
6.59
Ours
0.68
0.99
0.82
1.27
EXPERIMENTS
We train and evaluate method two popular real-world trajectory datasets: nuScenes (Caesar iet al. (2020) and Argoverse ( (Chang et al. (2019).. nuScenes/Argovert datasets provide the 2/2 seconds of past and require 6/3 seconds of future trajectory 0.5/0.1 second intervals, respectively. Training/validationnee
sets consist of wollwwrld driving scenes of 32, /8668,560/9900 in nuScenes and 205, 11.99942299921111 143 in Argoverse. For the baseline model in ablation, we follow TNT for goal conditioned model, and MHA encodes interaction from past trajectories. For implementation and computation details, please refer to the supplementary material.
5.1 QUANTITATIVE RESULT
Our method outperforms SoTA models in all nuScenes benchmark metries, as shown in Tab. Specifically, our model outperforms the runner-up method, PGP (Deo et al. (2022)), by substantial margin. This result indicates that our explicit interaction modeling via inferring waypoint occupancy helps scene understanding compared to the implicit interaction modeling of PGP. When predicting 10 samples, our model shows improvements of 5.3% and 8.8% in terms of mADE and MR. Previously, THOMAS Gilles et al. (2022b)) was ranked first in mFDE; by proposing recombination module that post-processed marginal predictions into the joint predictions that are aware of other agents. However, model performs better than THOMAS in mFDE,, indicating better interaction modeling ability without post-processing. This is possible because inferring future relationships helps to better understand the future interaction with other agents; details provided in the ablation study.
We also evaluated our method on the Argoverse dataset. While our model does not achieve SoTA performance, it still shows remarkable performance improvement in both validation and test sets. Moreover, except for the HiVT, our method make competitive performance in mADE. Please note that our model (0.82) is still comparable SceneTranrorrerr (0.80) and Multipath+ (0.79) in the test set results. However, HiVT uses the surrounding vehicles' trajectories for training, resulting in increased training data. Therefore, direct comparison to HiVT would be rather unfair.
We do not achieve SoTA in Argoverse because the proposed method is less effective than in nuScenes. We attribute this disparity to the differences in dataset configurations, where nuScenes requires predicting longer future trajectory than Argoverse. As intuition suggests, interaction modeling has more significant impact on longer-range prediction tasks. To validate this assumption, we conducted an ablation study by measuring the performance gain on nuScenes when predicting the same length of future as Argoverse. The results, presented in Tab. shows that our interaction modeling method improves mADE, by over 10% in 6-second prediction task, but its effect was halved in 3-second prediction task, which is similar to the results obtained in Argoverse. This finding suggests that our interaction modeling method more effective in longer-range prediction tasks.

---

Published as a conference paper at ICLR 2023
Table 4: Ablation studies on nuScenes.
F=1
F=5
mADE/mFDE
mADE/mFDE
Table 3: Impact of prediction time the pro- posed modeling in terms of /ADE /
Impact of model design
Baseline
3.23/7.66
1.26/2.49
Ours w/n FR
3.2 .9
1.26/2.50
Baseline
Ours
Improvement
nuScenes (6xec)
3.23/1117
2.89/1100 10
10.5%v/6.0%
Ours wio GCN
3.04/6.94
1.22/2.41
muScenes (3ecc)
1.26/0.50
1.19/048
5.6%
Ours w/ Sym
1,22//.35
Argoverse (src)
1.41/0.71
1.330.68
5.7%/4.2%
Importance of multimodal stochastic interaction
Ours w/ GP
2.98/6.78 78
1.20/2.33
Ours w/ Deterministic
2.96/6.80
1.28/2.52
Ours (Full)
2.89/6.61
1.19/2.30
Figure 6: Qualitative results of the proposed method. The green solid line is past trajectories, the red lines are 6 predicted samples by baseline (left) and our method (right). The blue line is GT future trajectory of the surrounding vehicles. Lane centerlines are gray dashed lines. In complex road scenes, baseline generates spatially uniform samples regardless of interaction with surrounding vehicles. On the other hand, our method generates diverse yet interaction-awar samples; wait or surpass other vehicles that would join in the future.
5.2 QUALITATIVE RESULT
In Fig. 6 we present prediction samples (F=6) from the baseline (left) and our method (right). To assess the efficacy of our method, we brought the samples with two agents and plotted the prediction of single agent per scene. The green, blue, and red solid lines indicate the past trajectories of the both agents, future trajectories of the surrounding agents, and prediction samples of the target agents, respectively. In two scenes, each target agent sets its intention to where the other agent is likely to pass in the future. Our method generates prediction samples that incorporate and leverage the Future Relationship with other agents. Which means, unlike the baseline method that ignores other agents and generates spatially uniform trajectories, our model surpasses or waits for the other agents accounting for interaction. Moreover, not only considering two modes of interaction; surpass wait, we also allow stochasticity within a single mode of interaction. Consequently, our model generates diverse yet interaction-aware samples.
Furthermore, our method can incorporate stochastic interaction when multiple agents are present. In the experiment shown in Fig. 7 we predict the trajectories of the target agents (denoted 0) with multiple interacting vehicles. In each scene, the intention of the target vehicle is fixed (denoted in green) and two interaction edges are sampled. The corresponding predicted trajectory samples and the degree of interaction (l) are plotted on the right. In the first row of the figure, the target agent (0) infers significant interaction with agent 2 in sample direction and is predicted to move ahead, our model generates an accelerating trajectory to follow agent In contrast, in sample 2, the interaction with agent 1 is sampled as significant because they are expected to be in the same intersection. In this case, our model generates decelerating trajectory considering the future motion of agent 1. Importantly, all predicted trajectories in these samples are appropriately constrained within the goal lane segments the intention is set to the green colored lane. This indicates that our training strategy effectively restricts the role of interaction features to momentary motion.
agent 2 is moving in the same
8

---

Published as a conference paper at ICLR 2023
Input
GT other vehicles
Intention vehicle
Sample
Sample
Figure 7: Qualitative results of the proposed method in multi-agent scene.
5.3 ABLATION STUDY
Ablation the impact of model design is shown in the upper part in Tab. The Ours w/o FR variant does not consider Future Relationship interaction modeling, and only uses past trajectories to infer the relation, similar the NRI. This variant performs almost identically the baseline, which MHA of past trajectories to model interaction. This result shows the importance of leveraging Future Relationship for plausible interaction inference. The Ours w/o GCN variant omits the smoothing waypoint occupancy leading to inaccurate inter-agent proximity (PR) estimation, especially in the posterior distribution. Since GT waypoint occupancy is a binary value, computing PR from it can result in inaccurate proximity and lower prediction performance. In the proposed model, we allow asymmetric interaction between two agents. The Ours w/ Sym applies hard symmetric interaction modeling ( (Li et al. (2019)), and it shows that our asymmetric design is more suitable for modeling the driver relation.
The importance of multi-modal stochastic interaction modeling is shown in the lower part of Tab. The Ours w/ GP variant models the prior distribution as Gaussian distribution instead of GM, considering only single modality of interaction, which leads to performance decline compared to the full model with multi-modal interaction. The Ours w/ Deterministic variant predicts only the mean of interaction edges in Eq. 4 Although it can model prone to be limited compared to the stochastic counterpart especially when the sample size is large. The result shows that stochastic modeling is critical for prediction performance, and deterministic modeling significantly degrades the prediction performance when predicting more samples. In contrast, the Ours w/ GP variant shows relatively less performance drop as it maintains stochasticity even after removing the GM prior.
interaction, the diversity is
CONCLUSION
In this paper, we propose Future Relationship to effectively learn the interaction between vehicles for trajectory prediction. By explicitly utilizing lane information addition to past trajectories, our FRM can infer proper interactions even in complex road structures. The proposed model generates diverse yet socially plausible trajectory samples by obtaining interaction probabilistically. which provides explainable medium such waypoint occupancy or inter-agent proximity. We trained our model using CVAE scheme and validated it on popular real-world trajectory prediction datasets. Our approach achieved SoTA performance in long-range prediction task, nuScnes, and brings remarkable performance improvement in a short-range prediction task, Argoverse. Modeling Future Relationship novel approach, and anticipate that using more sophisticated training methods (Ye al. (2022); Zhou et al.] (2022)) or a better baseline model (such as GANet (Wang et al. (2022) may further improve prediction performance.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by Institute of Information & Communications Technology Planning & (valuation(IITP) grant funded by the Korea governmen((MSITT 20..001-3-0 Development of High Performance Visual BigData Discovery Platform for Large-Scale Realtime Data Analysis.
REFERENCES
Inhwan Bae, Jin-Hwi Park, and Hae-Gon Jeon. Non-probability sampling network for stochastic human trajectory prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 6477-6487, 2022.
Alexander Barth and Uwe Franke. Where will the oncoming vehicle be the next second? In 2008 IEEE Intelligent Vehicles Symposium, pp. 1068-1073. IEEE, 2008.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In 2020 JEEF/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 11618-11628 IEEE Computer Society, 2020.
Defu Cao, Jiachen Li, Hengbo Ma, and Masayoshi Tomizuka. Spectral temporal graph neural network for trajectory prediction. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1839-1845, IEEE, 2021.
Sandra Carrasco, D Fernández Llorea, and MA Sotelo. Scout: Socially-consistent and understandable graph attention network for trajectory prediction of vehicles and vrus. 202/ IEEE Intelligent Vehicles Symposium (/V), pp. 1501-1508. IEEE, 2021.
Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In 2020 European Conference Computer Vision (ECCV). Springer, 2020.
Yuning Chai, Benjamin Sapp. Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In Conference on Robot Learning, pp. 86-99. PMLR, 2020.
Rohan Chandra, Tianrui Guan, Srujan Panuganti, Trisha Mittal, Uttaran Bhattachayya, Aniket Bera, and Dinesh Manocha. Forecasting trajectory and behavior of road-agents using spectral clustering in gssph- IEEE Robotics and Automation Letters, 5(3):4882-498 2020.
Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, al. Argoverse: 3d tracking and forecasting with rich maps. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8740-8749. IEEE Computer Society, 2019.
Siyuan Chen, Jiahai Wang, and Guoqing Li. Neural relational inference with efficient message passing mechanisms. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 7055-7063, 2021.
Nachiket Deo and Mohan M Trivedi. Convolutional social pooling for vehicle trajectory prediction. In 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1549-15498. IEEE, 2018.
Nachiket Deo and Mohan Trivedi. Trajectory forecasts unknown environments conditioned on grid-hased plans. arXiv preprint arXiv:2001.00733 2020.
Nachiket Deo, Eric Wolff, and Oscar Beijbom. Multimodal trajectory prediction conditioned
lane-graph traversals. In Conference on Robot Learning, pp. 203-212. PMLR, 2022.
Nat Dilokthanakul, Pedro AM Mediano, Marta Gamelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. arXiv preprint arXiv: 1.06448, 2016.
10

---

Published as a conference paper at ICLR 2023
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectoret:: Encoding hd maps and agent dynamics from vectorized representation. In 2020 IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Gohome: Graph-oriented heatmap output for future motion estimation. In 2022 International Conference on Robolics and Automation (ICRA), pp. 9107-9114. IEEE, 2022a.
Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde.
Thomas: Trajectory heatmap output with learned multi-agent sampling. In International Confer-
ence on Learning Representations, 2022b.
Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D' D'Souz, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Latent variable sequential set transformers for joint multi-agent motion prediction. In International Conference on Learning Representations, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In 202/ Advances in neural information processing systems (NeurIPS), 2014.
Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In 2021 IEEF/CVF International Conference on Computer Vision (ICCV), pp. 15283-15292. IEEE, 2021.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2255-2266 2018.
Boris Ivanovic and Marco Pavone. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1375-2384 2019.
ByeoungDo Kim, Scokhwan Lee, Scong Hyeon Park, Elbek Khoshimjonov. Dongsuk Kum, Junsoo Kim, and Jun Won Choi. Lapred: Lane-aware prediction of multi-modal future trajectories of dynamic agents. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pp. 14636-14645 IEEE Computer Vision and Pattern Recognition, 2021.
Diederik P Kingma and Max Welling. 1312v:1312.611 14,, 2013.
Auto-encoding variational bayes. arXiv preprint
Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference Machine Learning, pp. 2688- 2697. PMLR, 2018.
Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martin, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. Advances in Neural Information Processing Systems, 32, 2019.
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 336-345, 2017.
Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. Advances in neural information processing systems, 33:19783- 19794, 2020.
Longyuan Li. Jian Yao, Li Wenliang, Tong He, Tianjun Xiao, Junchi Yan. David Wipf, and Zheng Zhang. Grin: Generative relation and intention network for multi-agent trajectory prediction. Advances in Neural Information Processing Systems, 34:27107-271 18, 2021a.
Xiao Li, Guy Rosman, Igor Gilitschenski, Cristian loan Vasile, Jonathan A DeCastro, Sertac Karaman, and Daniela Rus. Vehicle trajectory prediction using generative adversarial network with temporal logic syntax tree features. IEEE Robotics and Automation Letters, (2::3499-3466 2021b.

---

Published as a conference paper at ICLR 2023
Yaguang Li, Chuizheng Meng, Cyrus Shahabi, and Yan Liu. dtructure-infommt graph auto encoder for relational inference and simulation. In ICML Workshop on Learning and Reasoning with Craph-Structured Data, volume 8, pp. 2, 2019.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In 2020 European Conference on Computer Vision (ECCV). Springer, 2020.
Chiu-Feng Lin, A Galip Ulsoy, and David LeBlanc. Vehicle dynamics and external disturbance estimation for vehicle path prediction. IEEE Transactions Control Systems Technology, 8(3): 508-518, 2000.
Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction with stacked transformers. 2021 IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), pp. 7573-7582. IEEE Computer Society, 2021.
Yecheng Jason Ma, Jeevana Priya Inala, Dinesh Jayaraman, and Osbert Bastani. Likelihood- diverse sampling for trajectory forecasting. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 13279-13288, 2021.
Jean Mereat, Thomas Gilles, Nicole EI Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Pita Gil. Multi-head attention for multi-modes joint vehicle motion forecasting. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9638-9644. IEEE, 2020.
Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tier Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David Weiss, Ben Sapp, Zhifeng Chen, and Jonathon Shlens. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In 2022 International Conference on Learning Representations net h
(ICLR), 2022. URL
Tung Phan-Minh, Elena Corina Grigore, Freddy A Boulton, Oscar Beijbom, and Eric M Wolff. Covernet: Multimodal behavior prediction using trajectory sets. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 140022-4071. IEEE, 2020.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically- feasible trajectory forecasting with heterogeneous data. In European Conference on Computer Vision, pp. 683-700. Springer, 2020.
Charlie Tang and Russ R Salakhutdinov. Multiple futures prediction. Advances in Neural Information Processing Systems, 32, 2019.
Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Multi- path++: Efficient information fusion and trajectory aggregation for behavior prediction. In 2022 International Conference on Robotics and Automation (ICRA), pp. 7814-7821. IEEE, 2022.
Anirudh Vemula, Katharina Muelling, and Jean Oh. Social attention: Modeling attention in human crowds. In 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018.
Mingkun Wang, Xinge Zhu, Changqian Yu, Wei Li, Yuexin Ma, Ruochun Jin, Xiaoguang Ren, Dongehun Ren, Mingxu Wang, and Wenjing Yang. Ganet: Goal area network for motion forecast- ing. arXiv preprint arXiv:2209.09723 2022.
Max Welling and Thomas N Kipf. Semi- cumi-super classification with graph convolutional networks. In J. International Conference on Learning Representations (ICLR 20/7), 2016.
Maosheng Ye, Tongyi Cao, and Qifeng Chen. Tpen: Temporal point cloud networks for motion forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11318-11322 2021.
Maosheng Ye, Jiamiao Xu, Xunnong Xu, Tongyi Cao, and Qifeng Chen. Dems: Motion forecasting with dual consistency and multi-pseudo-eee supervision. arXiv preprint arXiv:2244.00 2204.0589, 2022.
12

---

Published as a conference paper at ICLR 2023
Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting. In Proceedings of the IEEE/CVE International Conference Computer Viston, pp. 9813-9823, 2021.
Wenyuan Zeng, Ming Liang, Renjie Liao, and Raquel Urtasun. Lanerenn: Distributed representations for graph-centric motion forecasting. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 532-539. IEEE, 2021.
Lingyao Zhang, Po-Hsun Su, Jerrick Hoang, Galen Clark Haynes, and Micol Marchetti-Bowice Map-adaptive goal-based trajectory prediction. In Conference on Robot Learning, pp. 1371-1383. PMLR, 2021.
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp. Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, al. Tnt: Target-driven trajectory prediction. In Conference on Robol Learning, pp. 895-904. PMLR, 2021.
Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for mueti-agent motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2823-8833 2022.
13


Reviews:
[{"review_id": "uB7t6XSng_", "reviewer": "Reviewer_Y1Xx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors utilize the lane information to model the future relationship among agents for trajectory prediction. Specifically, the probability of lane-level waypoint occupancy of vehicles is predicted, and a future relation module is developed for sampling interaction edges. Finally, interaction features are obtained via message passing for decoding. Empirical results on nuScenes and Argoverse are promising.", "summary_of_the_review": "I list my concerns in [Weaknesses].", "strength_and_weaknesses": "**Strengths**:  \n1. Well-organized and clearly written.  \n2. Promising results.  \n\n\n**Weaknesses**:  \n1.  Some details are missing. What is the temporal probability of passing the same lane? and how to learn interaction based on that probability? This sentence is confusing and ambiguous.  \n\n\n2. Novelty is limited. Almost all the techniques are well-studied in the vehicle trajectory prediction task, i.e., the motivation of using HD map (lane information), occupancy via probability, many prior works follow this approach now. The differences between the proposed method and existing baselines are not highlighted, nor the contributions are emphasized.\n\n\n3. Minor mistakes:  \n(1) In Fig. 4, I believe it is Future Trajectory Samples instead of F Trajectory Samples.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rMpVh2UHF8P", "reviewer": "Reviewer_6gEW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Problem: Vehicle Trajectory Prediction as commonly defined and evaluated.\n\nKey insight: agents usually follow lanes, so explicitly use lane graph structure to find likely future agent-agent interactions.\n\nNovel contributions: Future Relationship Module that consumes waypoint occupancy estimates + lane graph and produces a prior distribution of interactions conditioned on the historical trajectory, trained against a posterior distribution conditioned on the full trajectory for training. Ablations show that this module is the key to system performance, as without it performance deteriorates to baseline which only uses historical trajectories.\n\nEvaluation: competitive / SotA performance on standard metrics for popular public Trajectory Prediction benchmarks (NuScenes and Argoverse 1)\n\n", "summary_of_the_review": "Tackles the standard problem of trajectory prediction via a key insight that is actualized into a method that's competitive on popular public benchmarks. Likely will have an impact on the problem's community, as the actualized key insight forms a module that should be compatible with other SotA methods and will likely produce better performing future methods.\n\nConditioned on\n1) expansion of the quantitative results section to include the additional methods mentioned (plus other methods deemed relevant from benchmark performance)\n2) clean-up of the presentation + English\n\nI would be happy to see this paper presented at ICLR 2023, and I think it would make a nice addition to the trajectory prediction community.", "strength_and_weaknesses": "Strengths: Good performance on popular public Trajectory Prediction benchmarks (NuScenes and Argoverse 1). The novel contribution, the Future Relationship Module, is shown to benefit the performance of the network. Furthermore, Future Relationships Module appears to provide useful features that are orthogonal to the modules of other competitive methods, opening up the possibility of future work that combines these methods together.\n\nWeaknesses: Paper's actualized key insight is \"agents usually follow lanes\" but they do not do any analysis to show how often this assumption is violated. Due to the fact that the method performs well, presumably it's not very often, but it would be nice to know often it emerges in the train/val dataset and qualitatively what happens to the predictions under these conditions.\n\nNuScenes results not fully SotA due to missing models (https://www.nuscenes.org/prediction?externalData=all&mapData=all&modalities=Any). DGCN_ST_LANE has significantly lower MinADE_5 (1.092) vs the proposed method (1.19 as reported in the paper), a gap larger than reported against the runner up PGP, and DGCN_ST_LANE was submitted on May 25th 2022, which is significantly before the Sept 28 2022 ICLR submission deadline and thus should be mentioned in the results table. However, note that the authors' proposed method *does* outperform DGCN_ST_LANE on MinADE_10 (0.89 in the paper vs 0.974) and so it does maintain some claim to SotA.\n\nArgoverse 1 results were reported on the *validation* set, not the test set on the public leaderboard. This important to note because the proposed method is outperformed on the validation set by HiVT. HiVT performs significantly worse on the test set -- HiVT-128, the method described in Zhou et al., 2022, has significant hits to both mADE_6 (0.66 val vs 0.77 test) and mFDE_6 (0.96 val vs 1.17 test), and is outperformed on this test set by many methods such as DCMS (Ye et al., https://arxiv.org/abs/2204.05859, posted April 12th, 2022) Wayformer (Nayakanti et al., https://arxiv.org/abs/2207.05844, posted July 12, 2022). Certainly the transitive performance degradation from the Argoverse 1 validation set to test set of HiVT may not apply to the authors' method, but this does leave a lingering question of how well the proposed method actually stacks up relative to the state of the art. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KZjKVoFx8E_", "reviewer": "Reviewer_2v4y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed to leverage future relationships for vehicle trajectory prediction. Particularly, the method first predicts future waypoint occupancy on lane segments. Then, a future relationship module defines an interaction edge between every two agents based on the proximity and features of the two agents, and the interaction edges follow Gaussian mixture distributions. Conditioned on the interaction features and intention features, the decoder predicts the trajectory. The results is SoTA on nuScenes dataset and on par to SoTA on Argoverse dataset.", "summary_of_the_review": "The paper proposed to use vehicles' future relationships to improve trajectory prediction. The paper has clear science contribution and has achieved SoTA results on public dataset.", "strength_and_weaknesses": "Strength:\n\n1. Compared with previous methods which directly use the history to predict the future, this method add an intermediate step, which first predict future waypoint occupancy on lane segments, and then derive explicit future relationships between agents. The paper show that such explicit future relationships are very helpful for predicting the final trajectory of an agent, and the model achieve SoTA results on nuScenes dataset.\n\n2. To derive future relationship, the authors proposed to calculate inter-agent pair-wise feature vectors that follow Gaussian Mixture distributions, and ablation study shows that modeling the stochastic of this feature vector is important.\n\nTherefore, this paper has good methodological contribution. Meanwhile, the experiments are thorough with quantitative, qualitative and ablation results.\n\nWeaknesses:\n1. The authors should really work on simplifying the notations. There are too many sup- and subscript and it is hard to follow.\n\n2. Why is the waypoint occupancy conditioned on the sampled goal lane features? Can we integrate out the goal? The Gaussian Mixture  in FRM should be able to capture the stochasticity of goals. This will reduce inference time.\n\n3. In section 4.4, the authors says \"the goal feature (\\hat{h}^{-}_{\\ell})\" is sampled F times from the inferred waypoint occupancy, however, in section 4.1 the waypoint occupancy is derived from the goal feature. The two statements seem contradictory.\n\n4. z_{e}^{ij} in equation (7) already contains information about h_x^j, why do we need another element-wise product with h_x^j? Does experiment show that this is necessary?\n\n5. Why is the Argoverse test set results not included in Table 3?\n\n6. The method does introduce quite a lot of computation compared with other baselines such as MultiPath++. It would be better if the authors can report training and inference cost compared with baselines. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster