GRIN: Generative Relation and Intention Network for Multi-agent Trajectory Prediction
Authors: Longyuan Li
Jian Yao
Li Kevin Wenliang
Tong He
Tianjun Xiao
Junchi Yan
David Wipf
Zheng Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Trajectory prediction
variational Autoencoder
deep learning
probabilistic

Abstract: Learning the distribution of future trajectories conditioned on the past is a crucial problem for understanding multi-agent systems. This is challenging because humans make decisions based on complex social relations and personal intents, resulting in highly complex uncertainties over trajectories. To address this problem, we propose a conditional deep generative model that combines advances in graph neural networks. The prior and recognition model encodes two types of latent codes for each agent: an inter-agent latent code to represent social relations and an intra-agent latent code to represent agent intentions. The decoder is carefully devised to leverage the codes in a disentangled way to predict multi-modal future trajectory distribution. Specifically, a graph attention network built upon inter-agent latent code is used to learn continuous pair-wise relations, and an agent's motion is controlled by its latent intents and its observations of all other agents. Through experiments on both synthetic and real-world datasets, we show that our model outperforms previous work in multiple performance metrics. We also show that our model generates realistic multi-modal trajectories.

GRIN: Generative Relation and Intention Network for Multi-agent Trajectory Prediction
Longyuan Lif-
Jian Yao² Junchi Yan't
Li K. Wenliang Tong He
Tianjun Xiao
David Wipf Zheng Zhang
'Dearrment of Computer Science and Engineering.
MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University
FFudan University
Gatsby Unit, University College London
*Amazon Web Services
{jeffli, yanjunchi/asjtan eduuu.edd.cccm en, kevinli@gataby. ucl. ac.lacc.ae uk (htong, tianjux, daviwipf, zhaz) (haz))amazzo.coo com
Abstract
Learning the distribution of future trajectories conditioned on the past is crucial problem for understanding multi-agent systems. This challenging because humans make decisions based on complex social relations and personal intents, resulting in highly complex uncertainties over trajectories. To address this problem, we propose conditional deep generative model that combines advances in graph neural networks. The prior and recognition model encodes two types of latent codes for each agent: inter-agent latent code to represent social relations and an intra-agent latent code represent agent intentions. The decoder carefully devised to leverage the codes in disentangled way predict multi-modal future trajectory distribution. Specifically, a graph attention network built upon inter-agent latent code is used to learn continuous pair-wise relations, and an agent's 's motion is controlled by its latent intents and its observations of all other agents. Through experiments both synthetic and real-world datasets, show that model outperforms previous work in multiple performance metrics. We also show that our model generates realistic multi-mode trajectories.
Introduction
Generative trajectory forecasting requires learning the distribution of future trajectories conditioned on the past. It can provide an important foundation of path planning and tracking in many domains [I In a highly interactive multi-agent environment, such as team sports, each individual takes actions based on its intentions, relations with, and observation of other agents 77. For example, an attacking basketball player acts based on his/her strategy along with observations of teammates and the defending team. Similarly, an experienced driver identifies potential dangers from observed actions and motions of other drivers, thereby planning ahead of time (27). Notably, in many such real-world examples, intentions, relations, and interactions are all dynamic, hidden, and stochastic, leading to highly variable future trajectories.
We argue that to model uncertainty correctly we should account for factors that arise from both individual and social (i.e. relation) perspectives, and do so explicitly. That is to say, individual decision/intention and social relation/observation are inductive biases that should be explicitly
WWork completed during internship at AWS Shanghai AI Labs. *Corespondence authors are Junchi Yan and Tong He.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

introduced into a model design. Existing works model either intention [ 32 Or relations 24 3 36
not both, lead to compromised performance under highly interactive and uncertain environments 22
To better address the problems above, we propose Generative Relation and Intention Network (GRIN) that explicitly models uncertainties of both the intention and relations, the design comes with an added benefit of being more interpretable. Specifically, our contributions are:
1) We present GRIN, conditional generative model for multi-agent trajectory prediction, trained by variational inference and learning [5];
2) We propose disentangled design of the model architecture such that the uncertainty from the intention and relations are modelled separately;
3) We evaluate our model on both a deterministic synthetic dataset 18 and a highly stochastic real-world dataset [41] by single-path and distributional metrics, quantifying the ability to predict accurately while being aware of multiple possibilities. Our model consistently outperforms advanced baselines on all datasets and metries tested, highlighting the critical advantage of generative modeling of intentions and relations.
2 Related Works
Our work is closely related to multi-agent trajectory prediction, relation discovery from trajectory data, and multi-modal prediction, which are discussed in below.
2.1 Multi-Agent Trajectory Prediction
Multi-agent trajectory prediction is an actively rescarched problem due its broad applications in robot planning 34 20]. traffic prediction [24 38 26). sport video analysis [5] and so on. In
practical situations, two major observations challenge trajectory prediction: 1) the way agents interact with each other affects the future trajectories, and 2) given very similar past trajectories. the future paths the same agents may still be very uncertain, producing diverse and multi-modal outcomes. Next, we survey existing works from these two perspectives in detail.
2.2 Relation Discovery
Relationship between agents can affect their behaviors through two mechanisms: the type or state of the relation and the communications among agents contingent on the relations. As such, modeling the interactions between agents using graphs and the message passing mechanism they afford becomes natural choice. Ivanovic et al. [12] uses heuristics from spatial proximity to get the graph before making predictions. Most other work start from a fully connected graph, then learn the graph edges in self-superviser way. Kosaraju et al. [19 run graph attention network (39 on the fully connected graph and use the attention weight to decide how much information to share between agents. Kamra et al. [14] developed a dedicated attention mechanism for trajectory prediction from the inductive bias of motion and intents. Instead of using soft-attention, neural relational inference (NRI) proposed by Kipf et al. [18] takes the form of variational auto-encoder; the edges are modeled independent, categorical posterior distributions produced by an encoder trained through a biased reparametrization gradient. Jiachen et al. 24 and Graber et al. [8] improved NRI by using graphs that evolve over time. Compared with (24). which is a error- based model with two-stage training pipeline, our model is
mrobabiiisticbbasee model with more customized end-to-end training procedure. [24] g compute
discrete edge types, which have difficulty modeling continuously varying relationships.
In summary, most work can be considered as RNN-based where the RNN is combined with the graph structure to model both spatial, temporal and inter-agent relations.
2.3 Multi-modal Prediction
A distributional estimation of possible trajectories provides more information for downstream tasks, especially for those with risk-sensitive processes. Deep generative models are suitable for such tasks. Most existing works build on the generative adversarial network (GAN) 22 21 10 and variational auto-encoder (VAE)
6 18 R. Social-GAN [10] is an early attempt address this issue. However,
it only learns single mode of behavior with high variance. Bocial-BiGAT [9] explicitly accounts

---

HRNK
HOON
la X
RNN
GCN
MLP
GAT MLP
4x
21
Prior/Encoder
Decoder
Figure 1: Model architecture of GRIN. It takes in sequence of X and infers posteriors the relations the graph of agents (Zg) and for each individual agent (2.)). Samples of zg enter a GAT module obtain inter-agent features before combining with 2 the intra-agent features. The GAT computes inter-agent features using the observations as well.
for the multi-modal nature of the prediction problem forming reversible transformation between each scene and its latent noise vector. For VAE-based methods, conditional- VAEss (35) provides an uppropriate formulation of trajectory prediction, as the past trajectory can be considered as the conditioning variable. The works [T] 34 33 explore a serials of work using CVAE-bast methods. Beyond Gaussian likelihoods, [24] proposes to use a Gaussian mixture as the likelihood generate multi-modal predictions, which may be helpful given limited flexibility of neural networks.
Previous works autonomous driving 2 25 37
also graph neural networks to learn the correlation of agents. However, for [37 21. the relation and intention of agents are entangled into single latent variable. In (25), it does involve probabilistic generative model design handle multi-modality.
These recent works have mostly considered multi- modality and built mechanisms to handle interaction between agents. However, to our knowledge, none of them disentangle the multi-modalee effeets from the two sources: inter-agent and intra-agent (or within-agent), which influence the trajectory in distinct ways. We address this point through a careful architecture design and thorough empirical validation in the rest of the paper.
3 Methods
We specify notations, describe the task, and introduce the model architecture and training strategy.
Notations In each scene, trajectory data this scene, and xf the location of the i-th agent at the 1-th step. Given Tohs & T as the number of historic frames, we have the observation Xp = X 1I:oos, and the task is to predict XF
given N agents across T steps. Denote by X NXT all the
(Tss 1): based on the observation.
Our motivation is to model the uncertainty of future trajectory given the past and to disentangle the two effects from the intra-agent intention and inter-agent relations, both of which are unobserved. Thus, we model the data by conditional generative model with latent variables z and zg that describe agent intention and relational graph. respectively. This model can be trained by maximizing the evidence lower bound (ELBO, (35)), which is usually used as a surrogate for the intractable log- likeliiooodd. During training, when past and future trajectories are available, using an encoder given X and Xx and optimize the ELBO with SGD. During prediction, we do require the encoder and predict the future trajectories using the generative model given X.p. Our We describe the details in the following subsections.
infer the latents
model, GRIN, is illustrated in Figure [
3.1 Encoder
The encoder computes approximate posterior q tioned on Xp and X To allow disentanglement between the two types of latents, we explore an inductive bias built into our decoder architecture described in Section 3.3
Xx) with samples l zg] condi-
To start, we concatenate Xp and X along the time dimension and use a recurrent neural network (RNN) compute feature of the full-length trajectory of each agent:
H E RNN RNN( Xx)),
(I)

---

where H ANN E R.WM,,, where N is the the number of agents and M, is the feature dimension. Next, we assume a complete graph G for the N agents, and further improve the feature via graph convolutional network (GCN):
H GCN(G, RNN),
where H GCN E Nx,,, Assuming z follows multi-variate Gaussian distribution N(, Io), with the reparameterization trick [31 [6], we sample e ~ (N, I), and obtain z 6. x a + μs, where the and a are predicted by Multi-Layer Perceptrons (MLP):
= E F,, (H)
(3)
_ = Fo (HE
The encoder's outputs are zA E RN zG E RNxx where Mz is the dimension of ZA. ZG-
3.2 Prior
The prior ppz characterize the conditional distribution of given only Xp. In our implementa- tion, the prior shares the model weights with the encoder. this case, the encoder and prior are only different in terms of the input data the RNN in Eq IT By weight-sharing the same RNN sees inputs with different lengths, and the KL-divergence term in the loss enforces the encoder and decoder to extract similar information from the input. Therefore, the RNN is encouraged to learn encode the same information from Xp and [x,, y.). Besides, weight-sharing sharing reduces its memory cost.
HEN RNN(X,)
H cCN GCN(G, H
(4)
"P = (HENN)
O" = Fo (HGEN)
3.3 Decoder
j
The decoder p(XF |ZA, ZG, Xp) produces fu- ture trajectories conditioned on the random variable z and the past trajectory Xp. Fol- lowing [8]. we assume that that latent z is sufficient summary of the distance history and define the likelihood as p(X++1 z, X x1#)= p(X* X Therefore, given and X', the decoder can predict X For > Tobs, the input to the decoder is its own prediction X
k
Neighbors
Figure 2: The modified GAT module.
The decoder is built with a simple inductive bias that is nonetheless important for capturing inter-agent relations, which describe In many practical situations, the relation between two agent largely depend on the agents themselves and is thus correlated. For example, for three particles A. B and each with either positive or negative charge, if one knows that A attracts B, and B attracts C, then A must repels C. Thus, the posterior over the relations are highly correlated. Such correlation over the relations can be more difficult to characterize, since dimensionality of relations scales as 2 Thus, we propose to model the relations as function of each pair of agent, rather than independent as assumed in previous work, e.g. [8] which yields correlated uncertainty over the relations.
To incorporate this intuition, assume complete graph for the N agents, and apply modified Graph Attention Network (GAT) module (39 to describe their pairwise relations via the attention mechanism. Specifically,, we use zg compute the attention weight on edges and update node features with observation X This module is illustrated in Figure 2 which implements:
Cj = (Fa(=a,) F.(2a.)))
(5)
0ij softmax,
where (.) the dot product, F, and F, are two MIP layers for source (zG.)) and destination (=g.) node features respectively, and ag is the edge attention weight for the later graph convolution

---

computation. Thus, the edges become correlated through the graph latents of each individual. By definition, the attention thus computed is not constrained to be symmetric but contains the set of symmetric relationships. When the true relationship is indeed symmetric, such as the force between charged particles, we expect this attention matrix by almost symmetric.
Further, one agent may have its own intention that is usually private and not available to the others. The agent-specific latent A models such private intentions and does not enter the GAT module. Altogether, it is likely that our model disentangles an agent's ' own intention (z₂) and the relations through (zg), which we carefully validate in experiments.
The rest of the decoder follows similar from previous work. Given aij and X', we perform one round of message passing extract the feature for each agent:
h1
Σ ans (
(6)
Next, we incorporate hद with A,, to predict the Ax. = x 2t. We assume A follows Gaussian distribution (A oz). where or is for the entire dataset. Then og is estimated by single learned parameter oz, and the mean is predicted by:
A P. (M =A,I))
The GAT module models the relationships among the agents given zg and their location X
3.4 Training
For training, we sample z = [ZA, zc] from the encoder, q(z) Xp, Xp), and then optimize the ELBO:
ELBO₂ -E[log(p(Xx Xp))] + (L X XP) X
(8)
Since our decoder predicts [Arif ā.] for agent at step I, and it follows Gaussian distribution. We approximate the expectation with the sample mean:
log (og(2 + E(st - Arr)²
E[log(((Xr/+
Σ
Σ
2
Toohh+1 1=1
where Z is the normalization term, and n the dimension of Azt. Since the outputs of prior and the encoder are Gaussian, the second term can be computed analytically:
[L[g(z|Xp X [( = (L[ Io (I IIo")]
(10)
Following 10 we sample multiple z's for one batch, and only compute the gradient from the lowest ELBO,;
L min (ELBO,)
(11)
3.5 Trajectory Generation
To generate future trajectories, we first sample z from the prior, p(z(Xr), and then roll out the prediction from the decoder for any t > Tobs:
A Decoder(z, X') x
(12)
x + AR
The first input of roll out the ground truth observation Tos, For t > Tobs, we use X as the input.
4 Experiments
In this section, we analyze the performance of GRIN against several baselines on two datasets. First, we introduce the datasets, baselines, and the evaluation metrics. Next, show quantitative results comparing GRIN with baselines and ablation study on GRIN. Finally, we conduct qualitative case-study of GRIN's behavior on the trajectory prediction task
*Code and data available at httt:::://gtttb..om/////gg/uuuu/
5

---

4.1 Datasets
Charged Dataset [18]. We experiment with a simulated deterministic systems: charged particles, which controlled by simple physics rules. In each scene, there are 5 charged particles. Each particle has either a positive or negative charge with equal probability. Particles with the same charge repel each other, and vise versa. We set T' = 100 and Tabs 80 for each scene. We generate 50K scenes for training, and 10K each for validation and test respectively.
NBA Dataset [41]. This dataset contains tracking data from the 2012-2013 NBA season, and is accessible [30]. In this dataset, each trajectory contains the 2D positions the basketball and the players from both sides. Each team is consisted of 5 players. We preprocess the data such that each scene has 50 frames and spans approximately 8 seconds of play, and the first 40 frames are historic, i.e. T = 50 and Tobs 40.
4.2 Evaluation Metries
The potential uncertainty and multi modiiimmddlltyy in the trajectory datasets makes the evaluation difficult. Instead of just measuring the point-wise absolute match between prediction and ground truth, we check both point-wise accuracy and distributional distance by the following different metrics.
1) Average Displacement Error (ADE) and Final Displacement Error (FDE), Average Displace- ment Error is the L2 distance between the ground truth and predicted trajectories. Final Displace- ment Error measures the L2 distance between the ground truth final destination and predicted final destination. For stochastic models, we report the best-of- 100 displacement error of each trajec- tory/destination.
2) Conditional Maximum Mean Discrepancy (MMD). MMD 9 is a non non-parametric measure of distributional distance between two distributions. Here, our goal is to learn a conditional distribution of the future path given the past, so we used the conditional version defined in 13 See Appendix for details.
3) Negative Log Likelihood (NLL). With the test dataset, we compute ELBO, an upper bound on the NLL, on the predicted trajectories. Since the computation of ELBO asks for sample of the random variables for each trajectory, we compute 100 ELBO values and report the mean and the standard deviation. We only report this metric on models that are trained with ELBO.
4.3 Baselines
To make the comparison complete, we decouple the designs to several aspects and compare with the representatives. While there are methods that integrate tracking with advanced perceptual back- bone [9] 33 here we mainly validate our approach of modeling inter- and intra-agent uncertainty, while the integration with other components is also worth future investigation.
Instance Interaction. Some existing works take the form of RNN plus interaction module each timestamp. Fuzzy Query Attention (FQA) [4] is a recent work following this paradigm, which models the inductive bias from relative motion, intent and interaction.
Inferred Latent Graph. Works in this paradigm take the form of variational auto-encoder, infer the latent graph from past trajectory. Message passing the decoder is conducted on the inferred graph. Representative works include Neural Relational Inference (NRI) 18 and its dynamic version Dynamic Neural Relational Inference (dNRI) (8)
Generative Model. Generative model is a suitable tool to handle multi multi-modality and uncertainty. One classic work following this line is Social- [10), as it is GAN-based method. It can be considered as a general building block, and can be extended by introducing visual input.
Implementation Details. We implement our model using PyTorch 29 which has BSD-style license. The graph neural network modules are implemented using DGL 40. which has Apache 2.0 license. We train the model with the Adam [17] optimizer with the learning rate of 0.001. For all datasets, we set the dimension of 2G and ZA to be 2, the hidden dimension of all MLPs the encoder and decoder to be 128. The batch size is set to be 128 for charged particle dataset and 64 for NBA dataset. Following common practice VAE training, we let the predicted variance å parameter that trained along with all other parameters in the model. We estimate the total amount of computation is

---

Table 1: Performance comparison with baseline models. MMD with p 0.05 are marked with Deterministic models are marked with f.
Charged
NBA
Model
ADE
FDE
ADE
FDE
MMD 2 X
NLL
NRI
[8
0.63
1.30
2.10
4.56
1.11
586.9#0..0
dNRI g
0.94
1.93
2.02
4.52
1.49*
662.6++0.0
FQAt [4]
0.82
1.76
2.42
4.81
1.38*
Social-GAN
10
0.66
1.25
1.88
3.64
1.18*
GRIN (Ours)
0.52
1.09
1.72
3.59
1.04
507.5+00.2
Table 2: Disentanglement ablation results of the proposed GRIN on NBA dataset. MMD with p 0.05 are marked with Deterministic models marked with f.
Model
ADE
FDE
MMD² x10
Deterministict
2.03
4.26
1.13
Deterministic zg
1.89
3.85
1.12
Deterministic 2,4
1.87
3.87
1.17*
Full Model
1.72
3.59
1.04
in total around 500 hours on an AWS p3.8xlarge EC2 instance with 4 Nvidia V100 GPU. The full details on the architecture and baseline implementations are discussed in the Appendix.
4.4 Quantitative Evaluation
Baseline Comparison. We quantitatively compare our proposed model against the others. In Table I we compare on Charged and NBA datasets. Since there's no multi-modality in Charged dataset, we only report ADE and FED. Since FQA is deterministic, and Social-GAN is not optimized via ELBO, we not compute NLL for these two models. GRIN achieves the best or on par performance under all the settings. We run hypothesis test of MMD scores and the difference statistically significant between our prediction and the predictions from dNRI, FQA and Social-GAN, Although the ADE and FDE performance of Social-GAN is similar to GRIN, their MMD is significantly less optimal than ours. This indicates GRIN learns a better distribution of future trajectory.
Disentanglement Ablation Study. We examine the quantitative impact of zg and n in the model. We make 2g or A deterministic by only predicting the mean as deterministic latent vector, and remove the KL penalty of this term. Under this setting, the baselines are not computing valid ELBO thus we don't 't report the Score score. The experiments are conducted on NBA dataset, and the results are available in Table Obviously, the deterministic model is less capable of making predictions in an uncertain environment. The two models with one fixed z both give compromised performance comparing the full model. Especially for deterministic ZA, only sampling the zg the MMD is significantly larger than the full model. The results shows that having zg and Z together improves on both the discrepancy error and the distribution distance. We also provide perturbation experiments of z and zg to show the disentanglement of latent space, which can be found the Appendix.
4.5 Qualitative Analysis with Visualization
We illustrate the predicted trajectories for scenes for both datasets. The figures support that GRIN can predict trajectories accurately in the deterministic environment, and can also capture uncertainty in the stochastic environments. Note that we just illustrate a few number of examples due to space limit, and more can be found in Supplementary show that the observations is general.
Interactive Scene
Deterministic Environment. scenario that some particles start with close trajectories, thus it is highly interactive. Moreover, although we don't have neutral particles in the training and test environment, a neutral particle with no charge is inserted into the scene. The prediction from GRIN is illustrated in Figure 3 We observe
Charged dataset, we manually design a
7

---

Truth
Prediction
Latent Attention
target
-4
source
-2
Figure 3: Trajectory prediction on charged particles dataset. The left two figures are ground truth and predicted trajectories. Diamond, plus and circle marks negative-charged, positive-charged and neutral particles respectively. The right plot is the learned edge attention Oijz where the i-th row records the attention weights from the i-th node's neighbors, where the cell on row and column j records the attention weights from the j-th node to the i-th node. Higher attention scores are in lighter color.
Ground Truth
Generations
Full model b
Fixed ZA
|
Fixed Zg
Figure Samples of generated trajectories on NBA dataset. For each trajectory, the light solid line the past trajectory, and the dotted line is the future trajectory. Attacking players pink, defending players are blue, and the basketball brown. The first column is the ground truth trajectories, and the following three are the generated samples. The first row shows the generated trajectories from the full model, that we sample from both 2g and Z The second row shows the generated trajectories from the model that only samples different zg with fixed sample of z, and the third contains the results from a fixed sample zg and different samples of Z
that for closed pair of particles (1 and 2), our model accurately captures the attracting forces as illustrated attention matrix. Interestingly, for the neutral particle 0, the model only pieks up minor effect from particle 1, and it doesn't impact the other particles either. This empirically proves that our attention module capable to capture interpretable relationships among agents, and ignores the less relevant pairs.

---

Uncertainty and Multi-modality in Real-world Environment. We visually show the multi- modality in the real-world NBA dataset. First, to show that our model is capable to capture the uncertainty, we illustrate a few scenarios where the prediction trajectories are highly diverged in Figure
4 To further inspect the effect from 2G and 2 separately, we sample 2g repeatedly with one fixed random sample of A and vise versa, then plot the trajectories under the same scenario. From the ground truth, we see that player 2 carries the ball in the past, and he passes the ball to player 3. From the generation in the first two rows, himself. In the third row, the model only predicts trajectories with a smaller variance. We argue that this case demonstrates that zg has more significant impact on the uncertainty of trajectories. This observation aligns with the common sense of basketball, as it is highly interactive and players' movements are mostly affected by the inter-agent relationships.
that player 2 could either pass the ball or carry
5 Conclusion and Discussions
oIn this paper, we present GRIN for multi-agent trajectory prediction. Our model achieves state-of- the-art performance on synthetic and real-world datasets across multiple metrics. Moreover, GRIN explicitly disentangles the sources of future uncertainty in the decoder, and produces convincing interpretable outputs in real-world scenarios.
5.1 Outlook and Future Works
Based on this work, we aim at several meaningful directions as future works.
1) Vision-based Trajectory Prediction. GRIN takes trajectories as inputs directly. While for unstructured vision data which is an important application scenario for our approach, e.g. video, an end-to-end pipeline is often needed to perform tracking and extract inter- and intra-agent information from video data to feed into our model, as done in 19 33
2) Dynamic Trajectory Prediction. Real-world environment is highly interactive and dynamic. This can be done by extending our model to continuously update and Zg as time rolls forward.
3) Sparse Relationship Modeling. For simplicity, have used fully connected graph to model agent interactions. Real-world environments often induce sparse connectivity, so connectivity can change over time. Leveraging sparsity dynamically can potentially improve efficiency and scalability.
5.2 Limitations
Our approach may still have limitations in modeling and predicting very complex multi agent systems. Although it captures the correlations in the relations, we cannot guarantee that the relations are consistent with the rules of the context (e.g. laws of physics between binary charged particles). One can either build in a particular decoder to represent such constraints or use surrogate model that learns and approximate the rules.
5.3 Potential Negative Social Impacts
Trajectory prediction can be privacy-sensitive when the targets are human individuals. Our model may further mine the underlying interactions and even relations from the observed behaviors which raise some concerns.
Acknowledgement
Longyuan Li and Junchi Yan were partly supported by National Key Research and Development Program of China (2020AAA0107600), Shanghai Municipal Science and Technology Major Project (201ISHZZZ00102, and NSFC (61972250, 7206 127003).
References
[1] Haoyu Bai, Shaojun Cai, Nan Ye, David Hsu, and Wee Sun Lee, Intention- aware online pomdp planning for autonomous driving in crowd. In 2015 IEEE International Conference on Robotics and Automation (ICRA). pages 454 460. IEEE, 2015.

---

[2] Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pages 624-641. Springer. 2020.
[3] Chiho Choi and Behzad Dariush. Learning to infer relations for future trajectory forecast. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[4] Chiho Choi and Behzad Dariush. Looking relations for future trajectory forecast. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 921-930, 2019.
[5] Panna Felsen, Pulkit Agrawal, and Jitendra Malik. What will happen next? forecasting player moves in sports videos. Proceedings of the IEEE International Conference on Computer Vision, pages 3342-3351, 2017.
[6] Panna Felsen, Patrick Lucey, and Sujoy Ganguly. Where will they go? predicting fine-grained adversarial multi-agent motion using conditional variational autoencoders. In Proceedings of the European Conference on Computer Vision (ECCV), pages 732-747, 2018.
[7] Chris D Frith and Uta Frith. Interacting minds biological basis. Science, 286(5445):1692 1695, 1999.
[8] Colin Graber and Alexander Schwing. Dynamic neural relational inference for forecasting trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pages 1018-1019, 2020.
[9] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773 2012.
[10] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEFE Conference on Computer Vision and Puttern Recognition, pages 2255-2264, 2018.
[11] Boris Ivanovic, Karen Leung, Edward Schmerling, and Marco Pavone. Multimodal deep generative models for trajectory prediction: A conditional variational autoencoder approach. IEEE Robotics and Automation Letters, 6(2):295-302, 2020.
[12] Boris Ivanovic and Marco Pavone. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2375-2384, 2019.
[13] Wittawat Jitkrittum, Heishiro Kanagawa, and Bernhard Schölkopf. Testing goodness of fit of conditional density models with kernels. In Conference Uncertainty in Artificial Intelligence, pages 221-230. PMLR, 2020.
[14] Nitin Kamra, Hao Zhu, Dweep Kumarbhai Trivedi, Ming Zhang, and Yan Liu. Multi-agent trajectory prediction with fuzzy query attention. Advances Neural Information Procesinng Systems, 33, 2020.
[15] Vasiliy Karasev, Alper Ayvaci, Bernd Heisele, and Stefano Soatto. Intent-aware long-term prediction of pedestrian motion. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 2543 2549. IEEE, 2016.
[16] Diederik Kingma and Max Welling, Auto-encoding variational Bayes. In ICLR, 2014.
[17] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint vrXiv::412.000 22800, 2014.
[18] Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pages 2688-2697. PMI.R, 2018.
[19] Vincet Kosaraju, Amir Sadeghian, Roberto Martin-Martin, Ian Reid, S Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan sanellggnnnn and graph attention networks. Advances in neural information processing systems, 2019.
[20] Henrik Kretzschmar, Markus Kuderer, and Wolfram Burgard. Learning to predict trajectories of cooper- atively navigating agents. In 2014 IEEE international Conference on Robotics and Automation (ICRA), pages 40200. IEEE, 2014.
10

---

[21] Alex Kuefler, Jeremy Morton, Tim Wheeler, and Mykel Kochenderfer. Imitating driver behavior with generative adversarial networks. In 2017 IEEE Intelligent Vehicles Symposium (IV). pages 204-211. IEEE, 2017.
[22] Stéphanie Lefévre, Dizan Vasquez, and Christian Laugier. A survey on motion prediction and risk assessment for intelligent vehicles. ROBÔMECH jouraal 1(1):1114, 2014.
[23] Jiachen Li, Hengho Ma, and Masayoshi Tomizuka. Interaction-aware multi-agent tracking and probabilistic behavior prediction via adversarial learning. In 20/9 International Conference on Robotics and Autonation (ICRA). pages 6658-6664. IEEE, 2019.
[24] Jiachen Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi- trajectory prediction with dynamic relational reasoning. Proceedings of the Neural Information Processing Systems (NeurIPS), 2020.
[25] Lingyun Luke Li, Bin Yang, Ming Liang, Wenyuan Zeng, Mengye Ren, Sean Segal, and Raquel Urta- sun. End-to-end contextual perception and prediction with interaction transformer. In 2020 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 5784-5791. IEEE, 2020.
[26] Binbing Liao, Jingqing Zhang, Chao Wu, Douglas Mcllwraith, Tong Chen, Shengwen Yang, Yike Guo, and Fei Wu. Deep sequence learning with auxiliary information for traffic prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 537-546, 2018.
[27] Andrew Liu and Dario Salvucci. Modeling and prediction of human driver behavior. In Intl. Conference on HCI, pages 1479-1483, 2001.
[28] Abduallah Mohamed, Kun Qian, Mohamed Elhoseeny, and Christian Claudel. Social-stgenn social spatio-temponal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 14424- 14432, 2020.
[29] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Veto Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[30] Stats Perform. Stats perform baskethall research datasets. httpa://asat www.:// amazon. com/maarkepllace/
pp/
[31] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic hackpropagation and approxi- mate inference in deep generative models. In ICMI, 2014.
[32] Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction conditioned goals in visual multi-agent settings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2821-2830, 2019
[33] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically- feasible trajectory forecasting with heterogeneous data. arXiv preprint arXiv:2001.0300 030093, 2020.
[34] Edward Schmerling, Karen Leung, Wolf Vollprecht, and Marco Pavone. Multimodal probabilistic model- based planning for human-robon interaction. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3399 -406. IEEE, 2018.
[35] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Leamingg structured output representation using deep conditional generative models. Advances in neural information processing systems, 28:3483-3491, 2015.
[36] Chen Sun, Per Karlsson, Jiajun Wu, Joshua Tenenbaum, and Kevin Murphy. Stochastic prediction of multi-agent interactions from partial observations. In International Conference on Learning Representa- tions, 2018
[37] Charlie Tang and Russ R Salakhutdinov. Multiple futures prediction. Advances in Neural Information Processing Systems, 12:15424-15434 2019.
[38] Luca Anthony Thiede and Pratik Prabhanjan Brahma. Analyzing the variety loss in the context of probabilistic trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9954-9963, 2019.
[39] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference Learning Representations, 2018
11

---

[40] Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li. Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He. George Karypis. Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-periee perfooraat package graph neural networks. arXiv preprini urXiv:1909.01314 2019.
[41] Yisong Yue. Patrick Lucey. Peter Carr, Alina Bialkowski, and lain Matthews, Learning fine- fine-grained spatial models for dynamic sports play prediction. In 2014 IEEE International Conference on Data Mining, pages 670-679. IEEE, 2014.
12


Reviews:
[{"review_id": "ut0ws4etht5", "reviewer": "Reviewer_8qCr", "summary": "This work proposes GRIN, an architecture that explicitly models the latent uncertainty of social and individual aspects of future prediction of multi-agent systems. The architecture is trained using a conditional variational inference framework, and uses graph attention networks to allow for social attention during the future generation stage. The architecture is evaluated on a Charged particles dataset, an NBA motion forecasting dataset and pedestrian datasets (appendix). Quantitative and qualitative results show that the proposed approach is better than the baselines, and showcases the utility of disentangling the latent space into social and individual components.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The major innovation in this work is the decomposition of the latent space into two parts: the first addresses inter-agent relationships and the second addresses intra-agent intention. The effectiveness of this approach is shown both quantitatively (Tab. 2) and qualitatively (Fig. 4).\n- The introduction, related work and method sections are well-written and clearly communicate the message to the reader.\n\nWeaknesses:\n- The related work section does not address recent motion forecasting approaches, some of which use a similar message passing mechanism [1,2, 3]. These methods have been explored in the context of autonomous driving.\n- Other than disentangling the latent space, I am not entirely sure if the architecture as a whole is novel. The architecture seems quite similar to the one proposed in [1].\n- The quantitative improvement in Table 1 over Social-GAN seems marginal. Given that there are more recent approaches that outperform Social-GAN on motion forecasting tasks, it would've been interesting to compare to those methods (e.g., [4]).\n- The disentanglement Ablation study performed (line 242 and Table 2) seems somewhat counter-intuitive. It seems to me that the correct ablation would be to 1) remove $z_A$ and double the size of $z_G$; 2) remove $z_G$, double the size of $z_A$ and compute the attention weights using the hidden states $h_i^t$. This would allow one to more accurately assess the effect of disentangling the latent space.\n\nReferences:\n- [1] Casas, Sergio, et al. \"Implicit latent variable model for scene-consistent motion forecasting.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII 16. Springer International Publishing, 2020.\n- [2] Li, Lingyun Luke, et al. \"End-to-end contextual perception and prediction with interaction transformer.\" 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.\n- [3] Tang, Charlie, and Russ R. Salakhutdinov. \"Multiple futures prediction.\" Advances in Neural Information Processing Systems 32 (2019): 15424-15434.\n- [4] Sadeghian, Amir, et al. \"Sophie: An attentive gan for predicting paths compliant to social and physical constraints.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\nClarifications:\n- Figure 4: The color of the ball (brown) is very difficult to distinguish given the background of the court. \n- The explanation of Figure 4 at line 266 can be improved by outlining that with a fixed $z_G$ the ball always remains with agent 2. It's not entirely clear what happens when we fix $z_A$.\n- For the NBA dataset, is there a special identifier for the ball to differentiate it from players in the encoding? If not, how does the model know to treat it's movement differently?\n\nMinor corrections:\n- References 18 and 19 are the same.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and social impacts are adequately discussed in the conclusion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "VM_J1DwBRr", "reviewer": "Reviewer_WhRr", "summary": "This work presents a deep VAE model for the multi-agent trajectory prediction problem. The assumption that social relationship between agents and each intention is a bias that should be introduced into the network explicitly.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper is moderately clear in terms of the content display and the proposed idea is original. The ablation study between Z_A and Z_G was helpful to support the claim of the inductive bias. \n\n- This raises a question, what if Z_A and Z_G are treated as one tensor Z_AG? \n- Also, I didn't quite understand what makes Z_A agent only and Z_G graph only? is it by design? More details on this are appreciated. \n- One main concern: In eq(1) X_P and X_F are concatenated in the encoder. Practically, you would only use the X_P in the encoder, isn't this like introducing the target to be predicted into the network? How does this works while testing if this is an only training mechanism? \n- Other works such as Social-STGCNN, EvolveGraph, Trajectron++ and similar lines of works do model each agent as a graph and define the relationship between them through the means of graph edged. They define it either by direct handmade function or by learning such a relation. Also, in Trajectron++ an agent graph is created (similar to Z_G). Where does GRAIN stands between these line of works, especially the concept of Z_A and Z_G seems to exists in previous lines of works in a sense?\n- The Qualitative analysis misses failure cases and why it happens.\n\n=+=+=+=+=+=+=+=+=+ POST REBUTTAL =+=+=+=+=+=+=+=+=+\n\nI believe that all the concerns where answered in a proper manner except one point. This point is the ADE/FDE 100 samples on ETH/UCY datasets in which most of the papers use the 20 samples metric and rarely uses the 100 samples metrics. Sharing the same concern as pxYX reviewer. Yet, I will raise my score. I believe posting the 20 samples ADE/FDE will improve the paper. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "4ZOzX8lLkJV", "reviewer": "Reviewer_C2Xv", "summary": "This paper develops a multivariable multimodal trajectory prediction model titled GRIN which is based on a CVAE framework. Specifically, future trajectories are conditioned on two latent vectors: one (the “interaction” latent) which is intended to represent the interactions between all pairs of entities, and one (the “intent” latent) which is intended to represent the “intent” of each individual entity. These meanings are enforced by the model architecture, as the interaction latent is used to compute attention values which are used to aggregate hidden representations for all pairs of variables, while the intent latent only impacts the prediction of its corresponding entity. Experiments are run on a few trajectory prediction datasets; GRIN outperforms other approaches which model interactions using latent variables.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality: \nThe novel component of this paper is the model architecture, as the CVAE formulation for trajectory prediction has been developed/used in prior work. The idea of interpreting a per-entity latent variable as an intent is not novel (e.g. [A]), nor is the idea of modeling relations between agents as latent variables (e.g. [7, 17]). To my knowledge, the combination of the two employed here is novel.\n\n### Quality:\n- The data used for comparison and the baseline models chosen are appropriate. \n- The novelty of this approach regards the use of the different types of latent variables; however, the predicted latent variables and what they encode are not explored in much detail. The only visualization provided is in Figure 3, for the synthetic data. It would be additionally interesting to explore whether the latent variables encoded any meaningful information for the other datasets.\n- This model uses very small latent vector sizes (2 per latent “type”, for a total of 4 dimensions). No experiments are presented which investigate how the size of these vectors indicate performance.\n- The appendix presents experimental results on additional data (ETH/UCY), where an additional baseline (Trajectron) is used for comparison. This model performs comparably on this dataset; it would be useful to understand how Trajectron performs on the other data as well (does it still perform comparably? Better? Worse?)\n- The latent variable sampling procedure described in lines 168/169 are somewhat strange to use in this context, since the ELBO contains expectations which are computed over the latent variables ([9] doesn’t run into this issue, since it uses a GAN loss). No experiments are run which investigate this choice - does doing this “min sampling” improve performance?\n\n\n### Clarity: \nThe approach is presented clearly, and builds sensibly on prior work related to multimodal trajectory prediction and interaction modeling. Code is provided, and the paper is detailed enough to ensure reproducibility.\n\n### Significance: \nThe model outperforms related approaches which model interactions between entities using latent variables. It is unclear whether this approach could be considered “state of the art” compared to other multimodal trajectory forecasting approaches (as mentioned previously, the experimental results indicate it performs comparably to other approaches). It is unclear whether the predicted latent variables are meaningful beyond leading to improved performance, as only one set of latent variables for one trajectory for one dataset are visualized.\n\n\n### Final Thoughts:\nThough GRIN performs well compared against related approaches, it is not completely clear why a practitioner should use this model over alternatives. There is not enough evidence presented to call this approach \"state-of-the-art,\" and there is not enough evidence presented that the predicted latent vectors are useful for interpretation purposes. If the goal is to present a model which explicitly models interactions between entities (such as NRI or dNRI) and performs better compared to prior approaches which do so, then more comparisons need to be made to better understand which component of the model leads to improved performance and whether the latent variable representation leads to meaningful interpretation.\n\n\n[A]Salzmann, Tim, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. \"Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data.\" ECCV 2020.\n\n\n\n# POST-REBUTTAL UPDATE\nThanks for the comment. It's great to see you were able to fix a few things and get better results. Additionally, the additional ablations regarding the impact of the different latent variables on predictions are a valuable addition to the paper. I'm increasing my score based on these additions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations described are somewhat general and vague, and would apply to most (if not all) of the baselines as well. Are there any limitations to this approach that are unique to it? Are there observed scenarios where the model tends to fail?\n\nThe raised negative societal impacts are valid, and there are no ethical concerns raised by this work in its current form.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "1Yvo7ls_pKq", "reviewer": "Reviewer_pxYX", "summary": "This paper proposes to model interactions together with intention using the conditional generative models. For this, the authors created a model to encode two sets of latent variables and provided how they can be used for trajectory prediction. The evaluation is conducted using three datasets (two in the main manuscript and one in the supplementary).  ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The work is well-motivated, and introduction clearly describes it.\n- The methods sounds technical. \n\nMajor concerns:\n- One of contributions seems that it samples two types of latent variables, $z_G$ for interaction modeling and $z_A$ for intention modeling. However, I cannot find the validation of such categorization, except verbal description. I would like to see the influence of $z_G$ and $z_A$. \n- In addition, the encoder of GRIN already extracts certain global features as a result of GCN in Eqn.2, considering the interactions of agents. In this view, it is questionable if it is good to categorize $z_G$ and $z_A$ into inter- and intra-agent latent. I don't see any intention information from the generation of $z_A$, and it seems more like spatial, temporal and inter-agent relations similar to [24,7]. \n- Following the previous concerns, one ablative study is that the demonstration of using two latent variables instead of one. Simply, the authors use $z_G,i$ in Eqn. 7. Since both $z_G$ and $z_A$ are sampled from the same latent space, I believe the result might not be huge different from such a replacement. \n- It will be interesting to see the comparison of [24] in the experiments section in addition to [7,17], as it can be categorized as one of the closest work. \n- The evaluation is made using two datasets. However, the Charged dataset provides simple toy examples, and it seems that the NBA dataset is tailored by authors. I am concerned about the validity of these evaluations. The readers of this paper would want to see more thorough comparisons with more diverse works (not only NRI-families but also recent methods) evaluated on more standard datasets. Although the authors provide the evaluation on one of the frequently used benchmark dataset (ETH/UCY) in the supplementary material, its evaluation criteria is different from the standard evaluation in literature, as they sample 100 trajectories. This also makes me doubt about the validity of this method. \n\nMinor concerns:\n- Figure 1 is missing the condition $X_p$ in the decoder.\n\n\n\n-----------------\nAfter read the authors' rebuttal, I raise my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a discussion on privacy-related societal impact. However, the authors should be more thoroughly discuss about it considering more broader aspects. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work explores two types of latent variables for multi-agent trajectory forecasting, an inter-agent latent code representing social relations and an intra-agent latent code representing agent intentions.   Scores for this paper were initially split between rejection and accept. Additional clarifications and (ablation) experiments provided during the rebuttal caused multiple reviewers to increase their scores.  While the experiments here are not extensive, they were perceived as being sufficient by the reviewers.  After the rebuttal period all reviewers recommend accepting this paper.

The AC recommends acceptance.
Decision: Accept (Poster)