JFP: Joint Future Prediction with Interactive Multi-Agent Modeling for Autonomous Driving
Authors: Wenjie Luo
Cheol Park
Andre Cornman
Benjamin Sapp
Dragomir Anguelov
Venue: CoRL 2022 Poster (2022)
Keywords: Self-driving
motion forecasting
interactive prediction

Abstract: We propose \textit{JFP}, a Joint Future Prediction model that can learn to generate accurate and consistent multi-agent future trajectories. For this task, many different methods have been proposed to capture social interactions in the encoding part of the model, however, considerably less focus has been placed on representing interactions in the decoder and output stages. As a result, the predicted trajectories are not necessarily consistent with each other, and often result in unrealistic trajectory overlaps. In contrast, we propose an end-to-end trainable model that learns directly the interaction between pairs of agents in a structured, graphical model formulation in order to generate consistent future trajectories. It sets new state-of-the-art results on Waymo Open Motion Dataset (WOMD) for the interactive setting. We also investigate a more complex multi-agent setting for both WOMD and a larger internal dataset, where our approach improves significantly on the trajectory overlap metrics while obtaining on-par or better performance on single-agent trajectory metrics.

on-par or better performance on JFP: Joint Future Prediction with Interactive Multi-Agent Modeling for Autonomous Driving
Wenjie Luo" wen
Cheolho Park
Andre Cornman cornman wwymo com
com
cheol wwwolhop@wayeeeeee hop@waymo com
Benjamin Sapp
Dragomir Angueloy
bensapp@waymo... com
dr Aragomirrewayee com
Abstract: We propose JFP, Joint Future Prediction model that can learn to gen- erate accurate and consistent multi-agent future trajectories. For this task, many different methods have been proposed to capture social interactions in the encod- ing part of the model, however, considerably less focus has been placed on rep- resenting interactions in the decoder and output stages. As result, the predicted trajectories are not necessarily consistent with each other, and often result in unre- alistic trajectory overlaps. In contrast, we propose an end-to-end trainable model that learns directly the interaction between pairs of agents in structured, graph- ical model formulation in order to generate consistent future trajectories. It sets new state-of-the-are results on Waymo Open Motion Dataset (WOMD) for the in- teractive setting. We also investigate a more complex multi-agent setting for both WOMD and a larger internal dataset, where approach improves significantly on the trajectory overlap metrics while obtaining on single-agent trajectory metrics.
Introduction
Motion forecasting of multiple agents is of particular importance in the autonomous vehicle (AV) domain, since safe and comfortable driving requires understanding of the future trajectory distri- bution of all agents on the road including but not limited to vehicles, pedestrians and eyclists.
This interesting and high-impact motion forecasting task has resulted in many proposed models to date (e.g. [1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]), spurred by the release of multiple popular public benchmarks for this problem [20, 21, 22, 23). Most of the focus to-date has been on input representations and encoding of the complex world states, including an agent's relationships with other agents and the road network. With a few exceptions [19, 24J. there has been almost no focus on output representations is driven at least in part by the primary metrics in most major public benchmarks, which are based on trajectory distance error to the ground trutt future trajectories, computed independently for each agent.
model the joint probability of multiple agents. This
Letting s; be the future trajectory for the 2th agent in the scene, the marginal distribution of s; typically represented by K future trajectories with corresponding probabilities: {p((s)) s 1 There are clear shortcomings for using marginal prediction for autonomous driving. As a simple illustrative example shown Fig. (a), assume two vehicle agents coming into intersection with equal prior probability of going straight or turning left. Representing the future set of outcomes with marginal probabilities. one cannot tell that two of four possible outcomes in (Agent A. Agent B} x {going straight, turning left} is infeasible. In practice, K is often set to be 6 [22, 20], and there could be even more unrealistic predictions when considering more than two agents.
Instead, one could consider modeling the joint distribution of all A agents in a scene as p(S1, $2...., SA). While keep the same number of states K for each agent, the total size of
"Equal contribution
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

s IB
Simple Example
b: Model Illustration
Figure 1: (a): A simple two-agent example showcasing the importance of using a joint model that can output consistent future trajectories. (b): We propose formulate the problem using pairwise graphical model, where the model first generates trajectories for all agents as unary (Sec. 3.2) and constructs the underlying interaction graph, pairwise potential computed for the selected pairs followed by message passing to get final joint trajectories (Sec. 3.3).
the output space grows exponentially w.r.t. number of agents, i.e. KA, This is computationally intractable, given that common scenarios can have dozens or hundreds of agents. Towards this full joint modeling, the research community has recently explored some limited versions: the Waymo Open Motion Dataset [20] provides a joint prediction track that focuses only on the given pair of interactive agent 52); The ACTION dataset [23] has conditional prediction track fo- cusing on p(s, s,). Practitioners have addressed these tasks by directly modeling a pairwise output space p( 52) or conditional space p(s, (s,) [24, 9]. While these methods are targeting two agents interaction cases, others have tried to approximate the full joint distribution directly by multi-agent trajectory sample rollouts [25, 8] or directly predicting a weighted set of K joint predictions [19] where the joint output is tied and limited to queries.
To better model the joint probability in the multi- setting, we formulated it as a pairwise graphi- cal model with dynamic interactive graph. where nodes correspond agents, and edges correspond to agents' pairwise relationships. Intuitively, the pairwise potential between each pair of agents can represent the consistency between their predicted trajectories. The underlying interaction graph can be built dynamically based on unary predicted trajectories. This is a more tractable solution com- pared with the one directly modeling everything jointly. As one special case, we also experiment with a star graph that intuitively takes ego-centric approach and primarily focuses modeling the interactions of the AV (Autonomous Vehicle) and other agents. Furthermore, using energy-based models allows us easily perform conditional inference. This is important for real-world AV stacks since one would need to reason about joint futures conditioned on a specific AV trajectory.
We evaluate our approach using the public Waymo Open Motion dataset (WOMD) [20], where we set the new state-of-the-ee for the interactive setting. Furthermore, we investigate the multi-agents setting where the model needs to predict consistent trajectories for all agents presented in the scene. Our approach achieves significant improvements on both WOMD and a much larger internal dataset.
2 Related Work
Feature Encoding The input of the motion forecasting task involves time-series data such the status of agents and traffic lights, as well as static data such as road graph elements. For encoding agents motion. LSTMs/GRU have been used [26, 8, 5, 27], VectorNet [14] adapted the polyline representation, laneGCN [2] utilized ID convolution, [7] rasterized the agents' bounding boxes explicitly and used CNNs encode the motion. On the other hand, [28, 3] took raw sensor data as input and implicitly learned the motion of agents with CNNs. Recently, attention layers have also been used to encode the motion of agents with promising results [19, 29, 30]. For encoding
road graph data, while rasterized map images have been used in [5, 26, 28, 3, 1, 8), [2, 31] shows that graph convolution can used to encode a map to better capture the relation between each map element. Furthermore, [15, 14, 9, 32, 33] utilized polylines to represent the road graph, which proved be an effective approach. In our work, we also use polylines to represent the road graph data.

---

Interaction Modeling While DSDNet [34] proposed a fixed fully connected graph to model in- teraction among agents, RAIN[35] trained RL module output the agents' connectivity graph. [27] showed that attention can be used to learn effectively the relation between agents. Scene former [19] and AutoBots [29] adapted axial attention layers [36] on agent state tensors' time and social dimensions separately, [37, 30] also adapted attention layers into their model for better over- all performance. However, these models focus mostly on the interaction modeling in the encoding part to learn better agent features, the final trajectory candidates are predicted independently often using simple MLP from those agent feature embeddings. For generating joint outputs, MFP [38] uses discrete latent variables to represent the different modes of the joint future, ILVMM[39] uses la- tent variables and GNN for interaction modeling and optimizes with ELBO, Scene Transformer [19] uses learnable queries to decode different joint futures. In contrast, we build graphical model on top of the trajectory candidates with fully learned potentials in order to generate consistent future predictions. Different from [34, 35], we construct the interactive graph dynamically using predicted trajectories from the backbone network.
Conditional Setting Different previous methods have explored conditioning properties for gen- erating joint predictions. M21 [40] focuses on agent pairs, where one of the agents is predicted as influencer and the other as reactor. Then utilizes a conditional model generate consistent future trajectories for the reactor based on the influencer's marginal future predictions. MTP[41] first se- lects topology (homotopy) among few predefined ones, and then predicts trajectories conditioning on that, while our method doesn't I have the topology assuppiio/requirementt Different approaches have also investigated the setting of conditioning on AV's s actions. In particular, [9] aimed gen- erate diverse predictions conditioned on hypothetical road lanes and interactions. [24] focused on agents' motion forecasting conditioned on future hypothetical trajectory (query) of ego-car. A sim- ilar setting has also been explored in [42], which conditions the agent predictions on the planning trajectory. In contrast, our approach is modeling the joint probability and can be converted into conditional setting easily via conditional inference on a graphical model.
3 Method
In this paper, employ encoder-decoder architecture [7, 27, 3, 19, 32] and propose joint future prediction model for multi-agent interactive modeling (named JFP). Specifically, we take as input the past trajectories of all agents, the static road elements (i.e. road graph), well as the dynamic road elements (i.e. traffic lights) to generate per-agent feature embeddings. These agent embeddings are fed into decoder that learns to generate the trajectory candidates for each agent, and unary and pairwise potentials. We further apply a graphical (belief propagation) to combine these learned unary and pairwise potentials, and generate scene- consistent joint futures. In this section, we first explain the base model for trajectory generation and consistency learning, then introduce the graphical model formulation and the construction of the interaction graph. Finally, we cover the training procedure including losses for optimization and gradient computation. Fig. (b) provides a high-level illustration of our model.
and run message passing
3.1 Input Representation
The inputs of our model consist of three parts: agent history, road graph, and traffic light status. Agent history contains each agent's 's 2D location, bounding box shape, yaw, velocity, agent type. We pre-process the data be agent-centric provide a normalized input for the neural network. This is done by centering each agent at its last timestep's location and rotating it to have zero yaw. We model the road graph using the efficient and effective polyline representation, where each polyline segment is created using the Ramer Saaeeer-Doglls--- -Peckkrr algorithm [43] from the raw road graph data. For each agent, we pre-select the subset of polyline segments that are closest and convert them into the corresponding agent-centric coordinate frame. The final polyline segment feature includes its 2D location, direction, length, and type. The traffic light features include type, states over time and 2D location, which have been similarly converted into the agent-centric coordinate system.
3

---

3.2 Backbone Network
Multiple different backbone networks have been proposed fuse information among agents and context information based on LSTMs [44], CNNs [28, 1, 45], GNNs(31], ContextGating [32] or transformers [29, 19, 30J. Given its outstanding performance, we adopt MultiPath++ [32] as the backbone network to generate marginal predictions for each agent. The key component of Multi- Path++ is the Multiple Context Gating (MCG) layer, which conceptually simplified attention layer. Compared with attention, the MCG layer maintains global attention ability while keeping the computation linear in the number of tokens.
3.3 Interaction Modeling
We represent interactions using an energy model with parameters 8, given input data X. The proba- bility is defined as:
p(s|2 0) = 2
0))
Specifically, a pairwise graphical model is used to compute energy as:
E(s| 0) Σ (s (2, +
Σ
Exair (S; (s,
(1))E9
where unary potential is defined using predicted probability q(s:|XX 0) from backbone Eerry (s:(2X,)) 0) (X,
Different from previous approaches, we utilize a learned pairwise potential Epair with dynamic underlying graph G. For each agent pair (i, we first project the predicted trajectories s; into agent-j-centered coordinate system, denoted as Suzy. and vice-versa for Sjzt. Note both Sunj and sja, have K trajectories for jth and agent respectively. Thus, the pairwise potential, as a x K matrix, can be simply computed as:
Epsir (s, 1, 0) = MLP((MLP((sss)) s.)), MLP((saa) s,Dl),
where | represents concatenation. The two inner MLPs share the weights. In our experiments, each MLP consists of two layers, the first with 128 and the second with 64 feature dimensions.
Intuitively, a pairwise potential provides the compatibility of the trajectory combinations between its two agents. For example, if the ah predicted trajectory of agent i is colliding with the 8th predicted
trajectory of agent j, then Epoir (s. Sa) [a, b] should have a much higher value, specifying that those two trajectories should not be selected the same time form one joint future prediction.
Although is clear that not necessarily all agents in the graph G interact with each other, we do not have ground truth labels for the interactivity. While [35] proposed to reinforcement learning to estimate the interactive graph, our core idea is to use the predicted marginal trajectory candidates. In particular, for each pair of agents (i, j). if the most likely predicted trajectories s sh are close with each other (i.e. their center locations are within the distance of their length and width), we connect these two agents in the interactive graph, iee. (i.j) G. In this way, the graph structure can be adapted for the specific scenarios in a dynamic fashion, thus we denote it as dynamic graph approach. In practice, we found this works especially well when the predicted trajectories are produced by well trained backbone model. In one special case related to AV planning, where the task is to predict the AV's future trajectories and the relevant futures for the other agents, we utilize a star-shaped graph that is centered on the AV with connections to all other agents.
Given the full definition of the graphical model, apply standard sum-produc, message passing to obtain the approximated joint probability as defined in Eq. and max-produet message passing to obtain the best joint trajectories for all agents. In practice, we instantiate 3 iterations of mes- sage passing in our models' compute graphs; we saw little benefit beyond this, which is a similar observation found in prior work with GNN interaction encoding [4].
Conditional Setting: Generating future trajectories explicitly conditioned on specific agents (usu- ally AV) action is important for planning in practice. This has been explored by a few prior works [42, 46, 19, 24, 40], however, their focus mostly on modeling the future of a single agent pair, instead of the full multi-agent setup. One advantage of leveraging graphical model in our

---

Method
Overlap(4) minSADE(()
minSFDE())
SMissRate(
mAP(f)
LSTM baseline [20]
1.91
5.03
0.78
0.05
HeatlRm4 [47]
1.42
3.26
0.72
0.08
OgeneeTrasformerr/ [19]
0.98
2.19
0,49
0.12
Test
M21 [40]
1.35
2.83
0.55
0.12
DenseTNT [48]
1.14
2.49
0.54
0.16
MulriPath++(321
1.00
2.33
0,54
0.17
JFP [AV-Star]
0.88
1.99
0.42
0.21
Scene [ranssrrrrmmm [19]
0.091
1.12
2.60
0.54
0.09
S [19]
0.046
0.97
2.17
0.49
0.12
Val
[33
0.064
1.00
2.33
0.54
0.18
JPP [AV-Star]
0.030
0.87
1.96
0.42
0.20
Table I: WOMD Interactive Split: We report scene-level joint metries numbers averaged for all
object types over 3, 5, 8 seconds. Metries minSADE, minSFDE, SMissRate, and mAp are from the benchmark [20]. In addition, we compute Overlap as defined in Sec. 4.1.
multi-agent interaction formulation that easy to perform conditional inference, i.e. to gen- erate joint future trajectories for other agents conditioned specific trajectory j for some agent We can compute joints of all other agents using the same procedure as before but with modified unary potential for agent i; i.e. by setting the unary potential other than the conditioned trajectory to infinity (Etraj(s, (s; * j) = oo) to make agent always select trajectory j during inference. Note this does not require separate training steps and is natural extension of our inference procedure.
3.4 Loss Function and Optimization
We train our multi-agent motion forecasting model to minimize the negative joint log-likelihood of given dataset D:
min Σ - logp(s/XXX
S
Instead of optimizing this objective function directly, previous methods [1, 9, 31] optimize the marginal probability for each agent, effectively decomposing the objective function into ΣxEDE, p(,,|X 8). Then for each agent, classification task with cross entropy loss used for determining which one of the K trajectories is favored, and regression task with Huber loss is used make the selected predicted trajectories closer to ground-truth.
In our work, while we utilize the same regression loss Lreg as in [32], we differ from previous methods on the classification loss, which optimizing directly Eq. The gradients of Eq. w.r.t. all trainable weights can be computed with chain rule where first we derive that the gradients w.r.t. the unary and pairwise terms (i.e. Etraj- Epair) are 2:
LE = cross_entropy (14s + stop-gradient(ix. - pt))
Ep cross_entropy (Us,) + stop- -stt--gradient(eeeee (ū. vs,)))
Note to simplify the notation, we denote μ4 as the unary softmax logits and Viy as the pairwise softmax logits, i.e. log(g(s,|2{, (x) = -EEraa (s, 0)) and tig = - Epair (s., s,l. f). and fus, Uj,, are corresponding marginal logits from message passing. Secondly, since Erraj and Epair are directly output from the backbone network, we can easily backpropagate their gradients to all trainable weights. Thus, the optimization target of Eq. 2 can be replaced by the following loss function: L =
L + L
4 Experiments
As we are proposing models for multi-agent motion forecasting in self-driving scenarios, it is essen- tial to use large and comprchensive datasets with appropriate metrics. We first show experimental
For detailed derivation, please refer supplementary material.
5

---

Overlap
Marginal
Pairwise
All
AV
minADE
minFDE
MissRate
mAP
minADE
minFDE
MissRate
(eeneerranfoomme((M) [19]
1.60 0.12
0.35
0.81
0.14
0.18
0.49
1.27
0.37
(eeneTTansfommer() [19]
0.83 0.04
0.61
1.58
0.28
0.24
0.56
1.49
0.40
MultiPath++ [32]
0.99 0.05
0.30
0.74
0.03
0.47
0.37
1.00
0.11
JFP [AV-Star]
0.99 0.02
0.30
0.74
0.03
0.49
0.36
0.94
0.10
JFP [Dynamic]
0.79 0.02
0.31
0.76
0.04
0.49
0.36
0.96
0.11
Table 2: WOMD Extended: All-agents setting using standard track data.
results on Waymo Open Motion Dataset (WOMD) [20] for the interactive split, where we set the new state-of-the-art compared with the existing public methods. Next, we experiment with more demanding "all-agent" setting using WOMD data, denoted as WOMD Extended described in detail below. Our method achieves significant improvements on joint metries and on-par better on single-agent metrics. We also do an ablation study to investigate the choice of underlying interaction graph to provide more insights into our model. To further validate the effectiveness of our model, we experiment on much larger (1000x) internal dataset and show improvements on all joint metrics, as well demonstrate our model's applicability the conditional prediction setting.
4.1 WOMD Interactive
This dataset provides 104K different scenarios, including 7.64 million unique tracks. Each data sample is 9-seconds segment with 10Hz, where the first 1 second is used as inputs and the later 8 seconds are considered for prediction. High-definition 3D maps including traffic lights were pro- vided as inputs. While the standard track focuses on single agent prediction, the interactive track is designed for evaluating joint prediction for one pair of agents in each scenario. Note that the selection process for the agent pair is designed pick interacting agents [20].
For this dataset, we utilize a star-graph centered at a random agent for training, and star- star-graph centered at one of the target agents for inference (i.e. effectively only one edge since there are only two agents for ovaluation). The prediction head of the backbone network consists of 5 ensemble heads that each provides 64 trajectories. NMS (Non-Max Suppression) is applied for each agent independently to generate the final 6 trajectories. We refer interested readers to [32] for more details. A star-graph is applied on top of the output trajectories. Our model is trained on 16 TPU-v3 cores with batch size of 1616 for 350K iterations (== 20 hours). We use ADAMW with a learning rate of 44 and decay to half only once at 200K iterations.
Table. shows the results on both the validation and test sets. Numbers are shown for the official metrics averaged over object types and time steps. Our model clearly outperforms other public methods across all metrics.
Overlap Metric 3 We measure the consistency of predicted joint trajectories using the overlap metiic. It is computed among the most likely trajectories for all target agents (two for each scene in this case). If the trajectories of a pair of agents overlap at any timestep, it is counted as one overlap. The final number is the sum of all overlaps in one scene, averaged over the dataset. The overlap metrics numbers in Tab. show that our model is able to generate significantly more consistent predictions than the baselines.
4.2 WOMD Extended
While the WOMD interactive track provides good insights about two interactive agents, we further experiment with an all-agent setting where the number of target agents can be up 40. The same data from WOMD is used for training and the validation data is obtained from the standard track, but includes all agents in the scene target agents for joint modeling (up 40 agents). This setup
"Note that the WOMD official metrics tool has an overlap entry, however that's only computing overlap between prediction and ground -ruth but not among predicted trajectories from different agents. Thus it is not ideal for evaluating consistency among predictions.

---

Overlap
Marginal
Pairwise
GraphType
All
AV
minADE
minFDE
MissRate
mAP
minADE
minFDE
MissRate
None
1.00 0.05
0.31
0.76
0.04
0.47
0.38
1.01
0.12
Random-Sta
0.97 0.04
0.30
0.75
0.03
0.49
0.36
0.95
0.10
AV-Star
0.99 0.02
0.30
0.74
0.03
0.49
0.36
0.94
0.10
Dynamic
0.79 0.02
0.31
0.76
0,04
0.49
0.36
0.96
0.11
Fully- molly-Connected
0.92 0.03
0.31
0.75
0.03
0.48
0.37
0.98
0.11
Table 3:
WOMD ablation on using different interaction graphs.
Pairwise
Conditional Marginal
Overlap
minADE
minFDE
MissRate
mAP
Overlap
minADE
minFDE
MissRate
mAP
MP++* [32]
0.03
1.27
3.12
0.33
0.29
0.05
1.14
3.45
0.34
0.48
CBP [24]
0.03
1.23
3.02
0.32
0.30
0.02
1.05
3.24
0.33
0.51
JFP [AV-Star]
0.01
1.19
2.91
0.31
0.31
0.01
1.04
3.04
0.31
0.57
Table Internal Dataset: JFP with AV-Star interaction graph shows consistent improvements over both joint metrics and conditional metrics, compared with competitive baselines including single- agent state-of-the-arr Multipath++ [32] and model designed for conditional setting CBP [24].
closer to the full AV driving setting, and can provide extra insights for evaluating joint prediction models. In this experiment, do not use ensembles for fast training speed.
We use the aforementioned overlap metric, with All for overlap among all targets and AV for overlap involving the AV. However, overlap alone is insufficient, since model can simply predict stationary futures for all agents to get 0 overlap. Thus standard single-agent marginal metrics included well, i.e. minADE, minFDE, MissRate, and mAP [20]. To better evaluate the joint model, we also introduce pairwise metrics, which naturally generalize the marginal metrics. For each pair of agents, we ask the model provide a top-6 set of joint predictions, and compute the minSADE, minSFDE, RMissRate, and mAP. We only consider those pairs, whose agents have ever appeared within 10 meters each other (according to ground-truth). This filtering can better differentiate interactive cases, since pairwise metrics for agents far away from each other highly correlate with marginal metries.
Metrics:
Tab. shows our models outperform MultiPath++[32] and Scene Treesforrooommr//// 19] by a large margin for overlap metries while maintaining on-par better marginal and pairwise metries. Note model with dynamic graph improves significantly over AV-Star on the overlap metrics, since better captures the interactions among all agents.
Ablation on Interaction Graph: The performance of our model with different underlying inter- action graphs are shown in Tab. 3, where Kandom-Star uses a star-graph centered at random agent, Fully-Connected uses a fully-connected graph. We can see that the dynamic model significantly improves upon the baselines the overlap metrics, without regressing the rest. While the Fully- Connected model has connections among all agents, could be too complex and less accurate for message passing especially with dozens of agents, resulting in worse performance compared the dynamic model.
Qualitative Results: In Fig. 2, we provide few representative scenarios demonstrate the effec- tiveness of our approach. Our model able to output consistent future trajectories even for very crowded scenes.
4.3 Internal Dataset
Next, we conduct experiments on our internal dataset, which is about 1,000 larger than WOMD, to show how the methods generalize on industry-scale datasets. With this experiment, we are focusing on the joint prediction performance relative to the AV, which is most relevant for robust and safe planning. Thus we measure 1) pairwise metrics involving AV and 2) AV-conditional metrics. In this setup, we experiment with star-graphs compared against baseline models since they naturally fit to AV-centric applications.
7

---

W/O Graph
Dynamic Graph
120
(a)
(b)
(c)
Figure 2: Visualization of most likely future predictions (8 seconds) for all agents on crowded scenes from Waymo Open Motion Dataset. We visualize each agent's predicted future trajectories as dark gray waypoints and its history trajectories as light gray waypoints. Trajectory overlaps among agents marked with red for all corresponding future steps. Compared with the baseline model (top row), our approach (bottom row) can output consistent joint prediction without trajectory overlap scenarios such (a) intersection, (b) for turning, and (c) with heavy traffic stack.
The dataset is prepared with the same setup as WOMD and our model trained with the same hyperparameters in Sec. 4.1 except 32 TPU-v3 cores used with learning rate 1e The MultiPath++ backbone is upgraded to handle the industrial-level dataset. The capacity of the encoders and the predictors implementation [24] using the same backbone. Joint probability for CBP is calculated by p(AV, other agent) best AV trajectories (i.e. the prediction with the smallest minADE w.r.t. ground-truth) and compute metrics on all other the agents conditioned on these trajectories. Note for JFP, can be computed as running conditional MRF inference to generate most likely trajectories for non-AV agents, while no extra change needed for the MultiPath+ baseline since agents are independently predicted.
increased. The conditional model, CBP in Tab. 4, is
p( (AV)p(other agent|
For the conditional metries,
select the
As shown in Tab, 4, our model consistently outperforms the baseline Multipath++ and CBP across all metrics by large margin, even for conditional metrics. This shows approach consistently has benefits on industrial-lewt dataset and models as well.
Limitation
The performance of our model could be bounded by how accurate the underlying interaction graph is, especially in sophisticated scenarios. While we proposed various graph types for different cases using human domain knowledge, it is still interesting to see we can learn the interaction graph from data directly. However, this is non-trivial since there is no ground-truth for the graph. On the other hand, our approach does not do regression after message passing steps, which might limit the performance. We leave it to future work for the exploration.
6 Conclusion
We have proposed a deep structured model that can perform joint future predictions of multiple agents in scene. Our model is based pairwise MRF and can be trained end-to-end by opti- mizing directly the joint log-likelihood. The underlying interactive graph is dynamically generated from unary trajectories. Importantly, show that model can learn avoid trajectory overlaps without corresponding domain knowledge. Experiments on public and larger scale internal datasets show significant improvement joint motion forecasting metrics.

---

References
[1] B. Sapp, Chai, M. Bansal, and D. Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In Conference on Robot Learning, pages 86-99. PMLR, 2020.
[2] M. Liang, B. Yang, R. Hu, Y. Chen, Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forecasting, In European Conference on Computer Vision, pages 541-556. Springer, 2020.
[3] S. Casas, W. Luo, and R. Urtasun. Intentnet: Learning predict intention from raw sensor data. In Conference on Robot Learning, pages 947-956. PMLR, 2018.
[4] S. Casas, C. Gulino, R. Liao, and R. Urtasun. Spagnn: Spatially-aware graph neural networks for relational behavior forecasting from sensor data. IEEE, 2020.
[5] N. Rhinehart, K. Kitani, and P. Vernaza. R2P2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In ECCV, 2018.
[6] N. Lee, W. Choi, Vernaza, C. B. Choy, H. S. Torr, and M. Chandraker. DESIRE: Distant future prediction in dynamic scenes with interacting agents. In CVPR, 2017.
[7] H. Cui, V. Radosavljevie, F.-. Chou, T.-H. Lin, T. Nguyen, T.-K. Huang, J. Schneider, and N. Djurie. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 International Conference on Robotics and Automation (ICRA), pages 2090- 2096. IEEE, 2019.
[8] C. Tang and R. R. Salakhutdinov. Multiple futures prediction. Advances in Neural Information Processing Systems, 32: 52:1544--55444 2019.
[9] S. Khandelwal, W. Qi, J. Singh, A. Hartnett, and D. Ramanan. What-if motion prediction for autonomous driving. arXiv preprint arXiv:2008. 10587, 2020.
[10] T. Phan-Minh, E. C. Grigore, F. A. Boulton, O. Beijbom, and E. M. Wolff. CoverNet: Multi- modal behavior prediction using trajectory sets. arXiv:191.110298, 2019.
[11] Y. Yuan and K. Kitani. Diverse trajectory forecasting with determinantal point processes. ICLR, 2020.
[12] J. Liang, L.. Jiang, K. Murphy, T. Yu, and A. Hauptmann. The garden of forking paths: Towards multi-future trajectory prediction. In CVPR, 2020.
[13] M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03077 2018
[14] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11525-11533, 2020.
[15] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Shen, Chai, Schmid, et al. Tnt: Target-driven trajectory prediction. arXiv preprint arXiv:2008.08294 2020.
[16] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. Gohome: Graph-oriented heatmap output for future motion estimation. arXiv preprint 1rXiv:2109.01827, 2021.
[17] F. Marchetti, F. Becattini, L. Seidenari, and A. D. Bimbo. Mantra: Memory augmented net- works for multiple trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7143-7152, 2020.
[18] W. Zeng, W. Luo, Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In CVPR, 2019.
[19] Ngiam, Caine, V. Vasudevan, Z. Zhang, H. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, D. Weiss, B. Sapp. Z. Chen, and J. Shlens. Scene transformer: A unified multi-task model for behavior prediction and planning. CoRR, abs/2106.08417,, 2021. URL. https: / / raxii vvv.vvv org/ htgg/bbs/2100 08417.

---

[20] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp. C. Qi, Y. Zhou, Z. Yang. A. Chouard, P. Sun, Ngiam, Vasudevan, A. McCauley, Shlens, and D. Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. arXiv preprint arXiv:2 204. 10133, 2021.
[21] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv: 1903. 1027, 2019.
[22] M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019.
[23] W. Zhan, L. Sun, D. Wang, H. Shi, A. Clausse, M. Naumann, J. Kümmerle, H. Königshof, C. Stiller, A. de La Fortelle, and M. Tomizuka. INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic 2019.
Maps.
arXiv:
[24] E. I. Tolstaya, R. Mahjourian, C. Downey, B. Varadarajan, B. Sapp, and D. Anguelov. Iden- tifying driver interactions via conditional behavior prediction. In IEEE International Con- ference on Robotics and Automation, ICRA 2021, Ki'an, China, May 30 June 5, 2021, pages 3473-3479, IEEE, 2021. doi: 00000///AAAA550022100000000 //doi.org/11 10. 1109/ ICRA4850 2021. 9561967.
URL https:
[25] N. Rhinehart, R. McAllister, K. Kitani, and S. Levine. PRECOG: Prediction conditioned on goals in visual multi-agent settings. ICCV, 2019.
[26] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-feasibbe trajectory forecasting with heterogeneous data. Computer Vision-ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 683-700, Springer, 2020.
[27] J. Mercat, T. Gilles, N. EI Zoghby, G. Sandou, D. Beauvois, and G. P. Gil. Multi-head attention for multi-modal joint vehicle motion forecasting. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9638-9644 IEEE, 2020.
[28] W. Luo, B. Yang, and R. Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 3569-3577, 2018.
[29] R. Girgis, F. Golemo, F. Codevilla, J. A. D' D'Souza, M. Weiss, S. E. Kahou, F, Heide, and C. Pal. Autobots: Latent variable sequential set transformers. arXiv preprint arXiv:2104.00563 2021.
[30] Y. Liu, J. Zhang, L. Fang, Q. Jiang, and B. Zhou. Multimodal motion prediction with stacked transformers. Computer Vision and Pattern Recognition, 2021.
[31] W. Zeng, M. Liang, R. Liao, and R. Urtasun. Lanerenn: Distributed representations for graph- centric motion forecasting. arXiv preprint arXiv:2101.0666 2021.
[32] B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cornman, K. Chen, B. Douillard, C. P. Lam, D. Anguelov, et al. Multipath++:: Efficient information fusion and trajectory aggregation for behavior prediction. arXiv preprint (..::2/111..9///1 2021.
[33] N. Nayakanti, R. Al-Rfou, A. Zhou, K. Goel, K. S. Refaat, and B. Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. arXiv preprint arXiv:2207.05844, 2022.
[34] W. Zeng, S. Wang, R. Liao, Y. Chen, B. Yang, and R. Urtasun. Dsdnet: Deep structured self-driving network. In European conference on computer vision, pages 156-172. Springer, 2020.
[35] J. Li, F. Yang, H. Ma, S. Malla, M. Tomizuka, and C. Choi. Rain: Reinforced hybrid atten- tion inference network for motion forecasting. Proceedings of the IEFF/CV International Conference on Computer Vision, pages 16096-161 16106, 2021.
10

---

[36] J. Ho, N. Kalchbrenner, D. Weissenborn, and T. Salimans. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180, 2019.
[37] M. Bhat, Francis, and J. Oh. Trajformer: Trajectory prediction with local self-attentive contexts for autonomous driving. arXiv preprint 1rXiv:2011.14910 2020.
[38] C. Tang and R. R. Salakhutdinov. Multiple futures prediction. In NeurIPS. 2019.
[39] S. Casas, C. Gulino, S. Suo, K. Luo, R. Liao, and R. Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pages 624-641. Springer, 2020.
[40] Q. Sun, X. Huang, J. Gu, B. C. Williams, and H. Zhao. M2i: From factored marginal trajectory prediction interactive prediction. arXiv preprint arXiv:2202. 1884, 2022.
[41] Roh, C. Mavrogiannis, Madan, D. Fox, and S. Srinivasa. Multimodal trajectory prediction via topological invariance for navigation at uncontrolled intersections. In Conference Robot Learning, pages 22-22227 PMLR, 2021.
[42] H. Song, W. Ding, Y. Chen, S. Shen, M. Y. Wang, and Q. Chen. Pip: Planning-informed trajectory prediction for autonomous driving. In European Conference on Computer Vision, pages 598-614, Springer, 2020.
[43] A. Saalfeld. Topologically consistent line simplification with the douglas-peucker algorithm. Cartography and Geographic Information Science, 26(1):7-18, 1999.
[44] A. Alahi, K. Goel, Ramanathan, Robiequet, Fei-Fei, and Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-971, 2016.
[45] J. Gu, C. Sun, and H. Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15303- 15312, 2021.
[46] S. Khandelwal, W. Qi, J. Singh, A. Hartnett, and D. Ramanan. What-if motion prediction for autonomous driving. 2020.
[47] X. Mo, Z. Huang, and C. Lv. Multi-modal interactive agent trajectory prediction using hetero- geneous liggeeenhanced graph attention network. In Workshop on Autonomous Driving, CVPR, volume 6, page 7, 2021.
[48] Gu, Q Sun, and H. Zhao. Densetnt: Waymo open dataset motion prediction challenge 1st place solution. CoRR, abs/2106. 14160, 2021. URL https: wwtp:/iv.rrr/ab///111e 1 141 60.


Reviews:
[{"review_id": "npv8yb2FwW", "reviewer": "Reviewer_DT7Z", "summary": "The paper introduces a new trajectory prediction method. It works as a layer on top of an existing forecasting method and creates a graph of agent-agent interaction that is used to improve prediction accuracy.\nI overall like the paper and I verified that the authors submitted a new high score to the leaderboard but I don't fully understand the method.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n### Strengths\n\n- **S.1** The writing is mostly great and the structure is good. (Except for the method section as noted below.)\n- **S.2** The authors do beat the high score on the WOMD Interaction Challenge. This is a hard task, so I think this is great evidence that this method works.\n- **S.3** The accompanying video is good at visualizing the differences between a baseline and this method\n\n\n### Weaknesses\n\n- **W.1** I'm going to have to veto the title. \"SMART\" is already taken as acronym in the motion forecasting literature and it's from 2 years ago: https://arxiv.org/pdf/2007.13078.pdf Creating another method with the \"SMART\" acronym will only cause confusion in the community. Plus for their title, the acronym works significantly better and they don't have to pull the \"R\" from the middle of a word. ;)\n- **W.2** Clarity. Even on the second read, the method isn't intuitively clear to me and I don't know how much of that is my fault. I don't hold this fully against the authors and I hope they can clarify this during the rebuttal. From what I understand, you predict pairwise interactions and then you use some message-passing magic to use these pairwise interaction probabilities to improve predictions. But how? (a) The pair-wise interaction predictor is supervised by distance, right? If 2 agents are within each other's threshold at timestep, then what's the supervisory signal for their \"pairwise potential\"? (b) And how does the message-passing work? How is that implemented, what data travels back and forth between agents? (c) How is either of that used to generate new/different trajectories? Are you fine-tuning the backbone with this somehow or are you modifying the predicted trajectories? (d) I really think it'd help to present a bit of code here, either as proof-of-concept Jupyter notebook or as pseudocode.\n- **W.3** Results on internal dataset/reproducibility. We obviously can't reproduce the results from your internal dataset. We don't have any specifics for this dataset. We don't know how much work was put into the baselines. For all intents and purposes, you could have completely made up these numbers. I would just cut that part of the paper and replace that with implementation details.\n- **W.4** Generality. If I understand correctly, this method is supposed to work with any kind of backend, right? Why don't you show that? SceneTransformer outperforms MultiPath++ on the WOMD Interactive dataset, so could you run an experiment with that as backend? If the argument is that code isn't available, then why not the similar AutoBots [29]? If your method does not work with other backends or if you can't show that, then I'd rephrase some sections of your paper to reflect that it's an extension of MultiPath++.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I'm happy to adjust my rating if the authors explain their method a bit more and respond to the other weaknesses noted above. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The method clearly works. Otherwise, it wouldn't be on the WOMD leaderboard.\nBut the writing and reproducibility need to be improved for CoRL standards.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "afAYKsyMpER", "reviewer": "Reviewer_jdv8", "summary": "This paper proposes a graph based decoder to do joint motion prediction by optimizing the pair-wise consistency by energy loss function. The performance of the proposed method is impressive and the 1000x larger experiments are interesting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed energy-based optimization goal for pair-wise consistency on the joint motion predicon task is novel.\n- The performance on the larger scale dataset is impressive.\n- The 1000x larger (compared to Waymo Open Motion) dataset is interesting\n\nWeaknesses:\n- The only open test protocol is Waymo Interaction Prediction, which only predicts for 2 agents jointly and might not be enough to fully examine the joint prediction performance in the real world setting. The Waymo Extended and internal dataset are both difficult for the community to follow. More experiments on public dataset/leaderboard would significantly improve the impacts of the work, for example, as mentioned in the paper, the INTERACTION challenge.\n- Some of technical details are unclear. Please see Issues section for details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In summary, the proposed method is intuitive and effective with impressive performance, which is why I give it accept. I appreciate the information about its 1000x larger internal dataset. However, it seems that it is not accessible to the research community.  Thus, I think some open source protocal code for Waymo Extended and experiments on other public datasets would make the work more influencial and easy to follow by the community. I would further raise my score if some of the two aforementioned efforts could be done or promised to be done.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- Unclear technical details: in Line 152-156, how the sum-product and max-product are performed, with what as input and output? How the iterations are done? Some experiments about the number of iterations would be appreciated.\n\n- Missed reference:  [1, 2] are both joint motion prediction methods. Some comparison with them would be appreciated; in Line 134, you mentioned projection between different agents in multi-agent system, which is similar to [3, 4, 5] \n\n- More datasets: as mentioned in the Weakness part, if experiments could be conducted on more public datasets, the work would have much larger impact and be easy for the community to follow. For example, I notice that [1,2] results on the INTERACTION challenge.\n\n- More details: The 1000x internal dataset is impressive.  In Line 260-261, you mentioned that Multipath++ and the predictor has been upgraded to handle the industrial-level dataset. I am wondering under such large dataset, how the model scale. Some details such as hidden-dimension, number of layers, and number of iterations would be appreciated.\n\n[1] Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction. ICLR 2022\n\n[2] THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling. ICLR 2022\n\n[3] Multi-Agent Trajectory Prediction by Combining Egocentric and Allocentric Views. CoRL 2021\n\n[4] Roto-translated Local Coordinate Frames For Interacting Dynamical Systems. NeurIPS 2021\n\n[5] MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction. ICRA 2022\n\n"}}, {"review_id": "QBi7MZqSa_e", "reviewer": "Reviewer_CkUV", "summary": "The paper proposed a pairwise MRF on top of MultiPath++ to have consistent trajectory predictions that any pair of trajectories are encouraged not to overlap each other. It uses various types of graphs that determine the connection between trajectories in terms of energy and provides an ablation study. It achieves state-of-the-art performance in WOMD interactive and provides additional experiment results on additional datasets (WOMD extended, internal dataset).\n\n[1] B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cornman, K. Chen, B. Douillard, C. P. Lam, D. Anguelov, et al. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. arXiv preprint arXiv:2111.14973, 2021.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Simple but an effective extension on MultiPath++ [1] for better modeling of interaction with a joint probability modeling.\n* Show the improvement on various interactive datasets including a public benchmark.\n\nWeaknesses\n* Some details are missing: details of the model and hyperparameters (alpha and beta) for the reproduction of the numbers.\n* Hard to distinguish what was coming from MultiPath++ and what was added as a contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Though I had confused about the details of the method, the high-level idea and contribution to predicting trajectories of high interactions are clear and sound. While MultiPath++ provides a strong backbone for marginal trajectory prediction, the method provides a novel way to involve pairwise interactions in choosing the final trajectories with the pairwise MRF formulation and a joint probability modeling. It seems the proposed method is effective in predictions for interactive cases but similar to its backbone in marginal predictions since it uses MultiPath++ results as its initial candidates. As the details of the architecture (for MultiPath++) are not clear, the contribution of the paper might be limited to highly interactive cases; therefore I recommend weak acceptance.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "I have a few questions and comments.\n\n* The main idea of the paper resembles the paper \"Multimodal Trajectory Prediction via Topological Invariance for Navigation at Uncontrolled Intersections\" [2] from CoRL 2020 as it modeled a joint probability of trajectories and considered a pair-wise relationship for consistent trajectory predictions. Adding descriptions that provide differences between the proposed method and the paper would be helpful for understanding the contribution. \"Implicit Latent Variable Model for Scene-Consistent Motion Forecasting\" [3] from ECCV 2020 also employed scene-level prediction with a joint probability model for trajectory prediction.\n* For the dynamic graph construction, the connection is determined as described in line 145. I wonder if the affinity matrix or E_{pair} is initialized as an NxN matrix (like a fully-connected graph) and the connection determines non-zero values in the matrix. Also, \"their center locations are within the distance of their length and width\" sounds a little confusing. Is the trajectory s_{i} already predicted from MultiPath++? While defining the pairwise energy of s_{i} and s_{j}, it requires realized trajectories in its energy term. Is this determined by the ground truth trajectories? I am also curious how the distance metric is defined between trajectories. Even some trajectories that are not close enough may affect each other. For instance, if a vehicle in front of the AV is running fast, it may affect the AV's behavior of left turn.\n* It seems that L_{reg} is not coming from Eq. 2 and is not defined in the paper. Is this for MultiPath++ model? Could you specify the term? Then how is this different from updating E_{traj}? Providing a clear distinction from MultiPath++ may help readers understand the paper.\n* (optional) Visualization of pairwise energy from examples may help readers to see the effect of the term.\n* (optional) What is the computation overhead of the graph approach? Though this is not the usual case to provide the speed of inference or training in this domain, providing some comparison of computation or speed of training or inference may help readers to understand the trade-off.\n* (optional) Effective or the average number of connections may be another interesting point for readers. Different graphs were compared and the dynamic graph was effective since it reduces the complexity to attend to meaningful connections. Comparing effective connecting numbers of the dynamic graph w.r.t the distance threshold can help understand the trade-off in the number of connections vs performance. Also comparing the optimal numbers between different datasets can provide an indirect measure of the level of interactions for each dataset.\n\n\n[2] Roh, J., Mavrogiannis, C., Madan, R., Fox, D., & Srinivasa, S. S. (2020). Multimodal trajectory prediction via topological invariance for navigation at uncontrolled intersections. arXiv preprint arXiv:2011.03894.\n[3] Casas, S., Gulino, C., Suo, S., Luo, K., Liao, R., & Urtasun, R. (2020, August). Implicit latent variable model for scene-consistent motion forecasting. In European Conference on Computer Vision (pp. 624-641). Springer, Cham."}}, {"review_id": "HEmWDb7Bs5-", "reviewer": "Reviewer_2W5p", "summary": "This paper proposed a deep structured model for multi-agent interactive trajectory prediction. It built upon typical multi-agent trajectory prediction model which predicts the marginal distributions of agents independently, and used an energy-based graphical model to predict the joint distribution of agents' trajectories. By adding the joint prediction layer, the proposed method outperformed the backbone MultiPath++ model and achieved the state-of-the-art performance on the interactive split of Waymo Open Motion Dataset. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The proposed method is a sound and effective way to model the joint distribution of agents' trajectories. It can be easily plugged into any backbone models to improve the prediction performance in interactive scenarios. \n\nWeaknesses: \n1. While it makes sense to use the proposed energy-based graphical model to model the joint distribution of interacting agents, it is unclear if it is necessary and better than some simple heuristics. For example, what if $E_{pair}(\\mathbf{s}_i,\\mathbf{s}_j|\\mathcal{X}, \\theta)$ is simply set as zero if the predicted trajectory pairs do not overlap, and $+\\infty$ if the predicted trajectory pairs overlap. The operation to verify trajectory overlapping is already inherited in the construction procedure of the dynamic graph, so it should a straightforward alternative of the proposed method. By doing so, it is equivalent to rule out all the combinations of trajectory candidates with collision and renormalize the joint distribution. I suspect it could work equally well as learning the energy function, because overlap rate seems to be the only metric that can be substantially improved by adding the graph structure, and the improvement is most effective when the model is informed which trajectory pair candidates overlap via the dynamic graph. I suggest the author to include this ablation study in the rebuttal or any other convincing evidences to show that the proposed method learns more than removing colliding candidates. \n\n2. It is not clearly explained how to approximate the joint probability and sample from the joint distribution. While the authors briefly explain it in line 152-154, I am particularly curious about how the final predicted trajectories are selected from the modeled joint distribution. My current understanding is that $K$ trajectory candidates are drawn from the marginal distribution of each agent, the joint distribution is defined over $K^N$ combinations of candidates for $N$ agents. While the energy-based graphical representation and the heuristic dynamic graph reduce the computational complexity in computing $E(\\mathbf{s}|\\mathcal{X}, \\theta))$, the probabilities of all the $K^N$ combinations need to be computed to find the ones with the highest probabilities. The dimension $K^N$ grows exponentially with the number of agents. The authors mentioned the number of agents could be up to 40 in their experiments. It will be great if the authors can provide more details on how the predicted trajectories are selected in their experiments. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is generally well-written. The proposed method is sensible and can effectively improve prediction performance in interactive scenarios. However, it is still unclear to me whether the additional modules are indeed necessary and can be practically used for real-time trajectory prediction. At this stage, I am inclined to reject the paper since it is difficult to precisely evaluate the value of the paper with these unclear issues, but I am happy to change my score if the authors can provide convincing evidences in the rebuttal. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please try to address the two concerns I mentioned before. ", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)