GRIN: Generative Relation and Intention Network for Multi-agent Trajectory Prediction

1 Introduction . Generative trajectory forecasting requires learning the distribution of future trajectories conditioned on the past . It can provide an important foundation of path planning and tracking in many domains [ 28 , 1 ] . In a highly interactive multi-agent environment , such as team sports , each individual takes actions based on its intentions , relations with , and observation of other agents [ 7 ] . For example , an attacking basketball player acts based on his/her strategy along with observations of teammates and the defending team . Similarly , an experienced driver identifies potential dangers from observed actions and motions of other drivers , thereby planning ahead of time [ 27 ] . Notably , in many such real-world examples , intentions , relations , and interactions are all dynamic , hidden , and stochastic , leading to highly variable future trajectories . We argue that to model uncertainty correctly we should account for factors that arise from both individual and social ( i.e . relation ) perspectives , and do so explicitly . That is to say , individual decision/intention and social relation/observation are inductive biases that should be explicitly ∗Work completed during internship at AWS Shanghai AI Labs . †Correspondence authors are Junchi Yan and Tong He . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . introduced into a model design . Existing works model either intention [ 15 , 32 ] or relations [ 24 , 3 , 36 ] , not both , lead to compromised performance under highly interactive and uncertain environments [ 22 ] . To better address the problems above , we propose Generative Relation and Intention Network ( GRIN ) that explicitly models uncertainties of both the intention and relations , the design comes with an added benefit of being more interpretable . Specifically , our contributions are : 1 ) We present GRIN , a conditional generative model for multi-agent trajectory prediction , trained by variational inference and learning [ 35 ] ; 2 ) We propose a disentangled design of the model architecture such that the uncertainty from the intention and relations are modelled separately ; 3 ) We evaluate our model on both a deterministic synthetic dataset [ 18 ] and a highly stochastic real-world dataset [ 41 ] by single-path and distributional metrics , quantifying the ability to predict accurately while being aware of multiple possibilities . Our model consistently outperforms advanced baselines on all datasets and metrics tested , highlighting the critical advantage of generative modeling of intentions and relations . 2 Related Works . Our work is closely related to multi-agent trajectory prediction , relation discovery from trajectory data , and multi-modal prediction , which are discussed in below . 2.1 Multi-Agent Trajectory Prediction . Multi-agent trajectory prediction is an actively researched problem due to its broad applications in robot planning [ 34 , 20 ] , traffic prediction [ 24 , 38 , 4 , 26 ] , sport video analysis [ 5 ] and so on . In practical situations , two major observations challenge trajectory prediction : 1 ) the way agents interact with each other affects the future trajectories , and 2 ) given very similar past trajectories , the future paths of the same agents may still be very uncertain , producing diverse and multi-modal outcomes . Next , we survey existing works from these two perspectives in detail . 2.2 Relation Discovery . Relationship between agents can affect their behaviors through two mechanisms : the type or state of the relation and the communications among agents contingent on the relations . As such , modeling the interactions between agents using graphs and the message passing mechanism they afford becomes a natural choice . Ivanovic et al . [ 12 ] uses heuristics from spatial proximity to get the graph before making predictions . Most other work start from a fully connected graph , then learn the graph edges in a self-supervised way . Kosaraju et al . [ 19 ] run graph attention network [ 39 ] on the fully connected graph and use the attention weight to decide how much information to share between agents . Kamra et al . [ 14 ] developed a dedicated attention mechanism for trajectory prediction from the inductive bias of motion and intents . Instead of using soft-attention , neural relational inference ( NRI ) proposed by Kipf et al . [ 18 ] takes the form of a variational auto-encoder ; the edges are modeled as independent , categorical posterior distributions produced by an encoder trained through a biased reparametrization gradient . Jiachen et al . [ 24 ] and Graber et al . [ 8 ] improved NRI by using graphs that evolve over time . Compared with [ 24 ] , which is a error-based model with two-stage training pipeline , our model is probabilistic-based model with more customized end-to-end training procedure . [ 24 , 8 ] compute discrete edge types , which have difficulty modeling continuously varying relationships . In summary , most work can be considered as RNN-based where the RNN is combined with the graph structure to model both spatial , temporal and inter-agent relations . 2.3 Multi-modal Prediction . A distributional estimation of possible trajectories provides more information for downstream tasks , especially for those with risk-sensitive processes . Deep generative models are suitable for such tasks . Most existing works build on the generative adversarial network ( GAN ) [ 23 , 21 , 10 ] and variational auto-encoder ( VAE ) [ 11 , 6 , 18 , 8 ] . Social-GAN [ 10 ] is an early attempt to address this issue . However , it only learns a single mode of behavior with high variance . Social-BiGAT [ 19 ] explicitly accounts for the multi-modal nature of the prediction problem by forming a reversible transformation between each scene and its latent noise vector . For VAE-based methods , conditional-VAEs [ 35 ] provides an appropriate formulation of trajectory prediction , as the past trajectory can be considered as the conditioning variable . The works [ 11 , 34 , 33 ] explore a serials of work using CVAE-based methods . Beyond Gaussian likelihoods , [ 24 ] proposes to use a Gaussian mixture as the likelihood to generate multi-modal predictions , which may be helpful given limited flexibility of neural networks . Previous works on autonomous driving [ 2 , 25 , 37 ] also use graph neural networks to learn the correlation of agents . However , for [ 37 , 2 ] , the relation and intention of agents are entangled into a single latent variable . In [ 25 ] , it does not involve a probabilistic generative model design to handle multi-modality . These recent works have mostly considered multi-modality and built mechanisms to handle interaction between agents . However , to our knowledge , none of them disentangle the multi-modality effects from the two sources : inter-agent and intra-agent ( or within-agent ) , which can influence the trajectory in distinct ways . We address this point through a careful architecture design and thorough empirical validation in the rest of the paper . 3 Methods . We specify notations , describe the task , and introduce the model architecture and training strategy . Notations In each scene , we are given N agents across T steps . Denote by X ∈ RN×T all the trajectory data in this scene , and xti the location of the i-th agent at the t-th step . Given Tobs ≤ T as the number of historic frames , we have the observation XP = X1 : Tobs , and the task is to predict XF = X ( Tobs+1 ) : T based on the observation . Our motivation is to model the uncertainty of future trajectory given the past and to disentangle the two effects from the intra-agent intention and inter-agent relations , both of which are unobserved . Thus , we model the data by conditional generative model with latent variables zA and zG that describe agent intention and relational graph , respectively . This model can be trained by maximizing the evidence lower bound ( ELBO , [ 35 ] ) , which is usually used as a surrogate for the intractable log-likelihood . During training , when past and future trajectories are available , we infer the latents using an encoder given XP and XF and optimize the ELBO with SGD . During prediction , we do not require the encoder and predict the future trajectories using the generative model given XP . Our model , GRIN , is illustrated in Figure 1 . We describe the details in the following subsections . 3.1 Encoder . The encoder computes an approximate posterior q ( z|XP , XF ) with samples z = [ zA , zG ] conditioned on XP and XF . To allow disentanglement between the two types of latents , we explore an inductive bias built into our decoder architecture described in Section 3.3 . To start , we concatenate XP and XF along the time dimension and use a recurrent neural network ( RNN ) to compute feature of the full-length trajectory of each agent : HERNN = RNN ( [ XP , XF ] ) , ( 1 ) where HERNN ∈ RN×Mh , where N is the the number of agents and Mh is the feature dimension . Next , we assume a complete graph G for the N agents , and further improve the feature via a graph convolutional network ( GCN ) : HEGCN = GCN ( G , H E RNN ) , ( 2 ) where HEGCN ∈ RN×Mh . Assuming z follows a multi-variate Gaussian distribution N ( µ , Iσ ) , with the reparameterization trick [ 31 , 16 ] , we sample ∼ N ( 0 , I ) , and obtain z = × σ + µ , where the µ and σ are predicted by Multi-Layer Perceptrons ( MLP ) : µE = Fµ ( HEGCN ) σE = Fσ ( HEGCN ) ( 3 ) The encoder ’ s outputs are zA ∈ RN×MZ , zG ∈ RN×MZ where MZ is the dimension of zA , zG . 3.2 Prior . The prior p ( z|XP ) characterize the conditional distribution of z given only XP . In our implementation , the prior shares the model weights with the encoder . In this case , the encoder and prior are only different in terms of the input data to the RNN in Eq 1 . By weight-sharing , the same RNN sees inputs with different lengths , and the KL-divergence term in the loss enforces the encoder and decoder to extract similar information from the input . Therefore , the RNN is encouraged to learn to encode the same information from XP and [ XP , XF ] . Besides , weight-sharing reduces its memory cost . HPRNN = RNN ( XP ) HPGCN = GCN ( G , H P RNN ) µP = Fµ ( HPGCN ) σP = Fσ ( HPGCN ) ( 4 ) 3.3 Decoder The decoder p ( XF |zA , zG , XP ) produces future trajectories conditioned on the random variable z and the past trajectory XP . Following [ 18 ] , we assume that that latent z is a sufficient summary of the distance history and define the likelihood as p ( Xt+1|z , X1 : t ) = p ( Xt+1|z , Xt ) . Therefore , given z and Xt , the decoder can predict X̂t+1 . For t > Tobs , the input to the decoder is its own prediction X̂t . The decoder is built with a simple inductive bias that is nonetheless important for capturing inter-agent relations , which we describe now . In many practical situations , the relation between two agent largely depend on the agents themselves and is thus correlated . For example , for three particles A , B and C each with either positive or negative charge , if one knows that A attracts B , and B attracts C , then A must repels C. Thus , the posterior over the relations are highly correlated . Such a correlation over the relations can be more difficult to characterize , since dimensionality of relations scales as N2 . Thus , we propose to model the relations as a function of each pair of agent , rather than independent as assumed in previous work , e.g . [ 18 ] , which yields correlated uncertainty over the relations . To incorporate this intuition , we assume a complete graph for the N agents , and apply a modified Graph Attention Network ( GAT ) module [ 39 ] to describe their pairwise relations via the attention mechanism . Specifically , we use zG to compute the attention weight on edges and update node features with observation Xt . This module is illustrated in Figure 2 which implements : eij = 〈Fd ( zG , i ) · Fs ( zG , j ) 〉 αij = softmaxj { eij } ( 5 ) where 〈·〉 is the dot product , Fs and Fd are two MLP layers for source ( zG , j ) and destination ( zG , i ) node features respectively , and αij is the edge attention weight for the later graph convolution computation . Thus , the edges become correlated through the graph latents zG of each individual . By definition , the attention thus computed is not constrained to be symmetric but contains the set of symmetric relationships . When the true relationship is indeed symmetric , such as the force between charged particles , we expect this attention matrix to by almost symmetric . Further , one agent may have its own intention that is usually private and not available to the others . The agent-specific latent zA models such private intentions and does not enter the GAT module . Altogether , it is likely that our model disentangles an agent ’ s own intention ( zA ) and the relations through ( zG ) , which we carefully validate in experiments . The rest of the decoder follows similar from previous work . Given αij and Xt , we perform one round of message passing to extract the feature for each agent : hti = 1 N − 1 ∑ j αijFg ( [ xti , x t j ] ) ( 6 ) Next , we incorporate hti with zA , i to predict the ∆x t+1 i = x t+1 i − xti . We assume ∆x̂ti follows a Gaussian distribution N ( ∆xti , σx ) , where σx is for the entire dataset . Then σx is estimated by a single learned parameter σ̂x , and the mean is predicted by : ∆x̂t+1i = Fx ( [ h t i , zA , i ] ) ( 7 ) The GAT module models the relationships among the agents given zG and their location Xt .

Summary: This work proposes GRIN, an architecture that explicitly models the latent uncertainty of social and individual aspects of future prediction of multi-agent systems. The architecture is trained using a conditional variational inference framework, and uses graph attention networks to allow for social attention during the future generation stage. The architecture is evaluated on a Charged particles dataset, an NBA motion forecasting dataset and pedestrian datasets (appendix). Quantitative and qualitative results show that the proposed approach is better than the baselines, and showcases the utility of disentangling the latent space into social and individual components.