Dynamic Relational Inference in Multi-Agent Trajectories

DYNAMIC RELATIONAL INFERENCE IN MULTI-AGENT TRAJECTORIES Anonymous authors Paper under double-blind review ABSTRACT Unsupervised learning of interactions from multi-agent trajectories has broad applications in physics , vision and robotics . However , existing neural relational inference works are limited to static relations . In this paper , we consider a more general setting of dynamic relational inference where interactions change over time . We propose DYnamic multi-Agent Relational Inference ( DYARI ) model , a deep generative model that can reason about dynamic relations . Using a simulated physics system , we study various dynamic relation scenarios , including periodic and additive dynamics . We perform comprehensive study on the trade-off between dynamic and inference period , the impact of training scheme , and model architecture on dynamic relational inference accuracy . We also showcase an application of our model to infer coordination and competition patterns from real-world multi-agent basketball trajectories . 1 INTRODUCTION . Particles , friends , and teams are multi-agent relations at different scales . Learning multi-agent interactions is essential to our understanding of the structures and dynamics underlying many systems . Practical examples include understanding social dynamics among pedestrians ( Alahi et al. , 2016 ) , learning communication protocols in traffic ( Sukhbaatar et al. , 2016 ; Lowe et al. , 2017 ) and predicting physical interactions of particles ( Mrowca et al. , 2018 ; Li et al. , 2018 ; Sanchez-Gonzalez et al. , 2020 ) . Most existing work on modeling relations assume the interactions are observed and train the models with supervised learning . For multi-agent trajectories , the interactions are hidden and thus need to be inferred from data in an unsupervised fashion . While one could impose an interaction graph structure ( Battaglia et al. , 2016 ) , it is difficult to find the correct structure as the search space is very large ( Grosse et al. , 2012 ) . The search task is computationally expensive and the resulting model can potentially suffer from the model misspecification issue ( Koopmans & Reiersol , 1950 ) . Neural Relational Inference for Interacting Systems Thomas Kipf * 1 Ethan Fetaya * 2 3 Kuan-Chieh Wang 2 3 Max Welling 1 4 Richard Zemel 2 3 4 Abstract Interacting systems are prevalent in nature , from dynamical systems in physics to complex societal dynamics . The interplay of components can give rise to complex behavior , which can often be explained using a simple model of the system ’ s constituent parts . In this work , we introduce the neural relational inference ( NRI ) model : an unsupervised model that learns to infer interactions while simultaneously learning the dynamics purely from observational data . Our model takes the form of a variational auto-encoder , in which the latent code represents the underlying interaction graph and the reconstruction is based on graph neural networks . In experiments on simulated physical systems , we show that our NRI model can accurately recover ground-truth interactions in an unsupervised manner . We further demonstrate that we can find an interpretable structure and predict complex dynamics in real motion capture and sports tracking data . 1 . Introduction A wide range of dynamical systems in physics , biology , sports , and other areas can be seen as groups of interacting components , giving rise to complex dynamics at the level of individual constituents and in the system as a whole . Modeling these type of dynamics is challenging : often , we only have access to individual trajectories , without knowledge of the underlying interactions or dynamical model . As a motivating example , let us take the movement of basketball players on the court . It is clear that the dynamics of a single basketball player are influenced by the other players , and observing these dynamics as a human , we are * Equal contribution 1University of Amsterdam , Amsterdam , The Netherlands 2University of Toronto , Toronto , Canada 3Vector Institute , Toronto , Canada 4Canadian Institute for Advanced Research , Toronto , Canada . Correspondence to : Thomas Kipf < t.n.kipf @ uva.nl > . Proceedings of the 35 th International Conference on Machine Learning , Stockholm , Sweden , PMLR 80 , 2018 . Copyright 2018 by the author ( s ) . Observed dynamics Interaction graph Figure 1 . Physical simulation of 2D particles coupled by invisible springs ( left ) according to a latent interaction graph ( right ) . In this example , solid lines between two particle nodes denote connections via springs whereas dashed lines denote the absence of a coupling . In general , multiple , directed edge types – each with a different associated relation – are possible . able to reason about the different types of interactions that might arise , e.g . defending a player or setting a screen for a teammate . It might be feasible , though tedious , to manually annotate certain interactions given a task of interest . It is more promising to learn the underlying interactions , perhaps shared across many tasks , in an unsupervised fashion . Recently there has been a considerable amount of work on learning the dynamical model of interacting systems using implicit interaction models ( Sukhbaatar et al. , 2016 ; Guttenberg et al. , 2016 ; Santoro et al. , 2017 ; Watters et al. , 2017 ; Hoshen , 2017 ; van Steenkiste et al. , 2018 ) . These models can be seen as graph neural networks ( GNNs ) that send messages over the fully-connected graph , where the interactions are modeled implicitly by the message passing function ( Sukhbaatar et al. , 2016 ; Guttenberg et al. , 2016 ; Santoro et al. , 2017 ; Watters et al. , 2017 ) or with the help of an attention mechanism ( Hoshen , 2017 ; van Steenkiste et al. , 2018 ) . In this work , we address the problem of inferring an explicit interaction structure while simultaneously learning the dynamical model of the interacting system in an unsupervised way . Our neural relational inference ( NRI ) model learns the dynamics with a GNN over a discrete latent graph , and we perform inference over these latent variables . The inferred edge types correspond to a clustering of the interactions . Using a probabilistic model allows us to incorporate prior beliefs about the graph structure , such as sparsity , in a principled manner . ar X iv :1 80 2 . 04 68 7v 2 [ s ta t.M L ] 6 Ju n 20 18 Figure 1 : Neural R lational I ference for learning th interaction graph . Picture taken from ( Kipf et al. , 2018 ) Relational inference aims to discover hidden interactions from data and has been studied for decades . Statistical relational learning are based on probabilistic graphical models such as probabilistic relational model ( Kemp & Tenenbaum , 2008 ; Getoor et al. , 2001 ; Koller et al. , 2007 ; Shum et al. , 2019 ) . However , these methods may require significant feature engineering and high computational costs . Recently , Battaglia et al . ( 2016 ) ; Santoro et al . ( 2017 ) propose to reason about relations using graph neural networks but still requires supervision . One exception is Neural Relational Inf rence ( NRI ) ( Kipf et al. , 2018 ) , a flexible deep generative mo el that can infer pot ntial relations in an unsupervised fashi n. As shown in Figure 1 , NRI simul- taneously learns the dynamics from multi-agent trajectories and infers their re ations . In particular , NRI builds upon variational auto-encoder ( VAE ) ( Kingma & Welling , 2013 ) and introduces latent variables to represent the hidden relations . Despite its flexibility , a major limiting factor of NRI is that it assumes the relations among the agents are static . Th t is , two agents ar either interacting or n t interacting regardless of their states at different time steps , which is rath r restrictive . 1 In this paper , we study a more realistic setting : dynamic relational inference . For example , in game plays , players can coordinate and compete dynamically depending on the strategy . We propose a novel deep generative model , which we call DYnamic multi-Agent Relational Inference ( DYARI ) . DYARI encodes trajectory interactions at different time steps . It utilizes deep temporal CNN models with pyramid pooling to extract rich representations from the interactions . DYARI infers the relations for each sub-sequence dynamically and jointly decode a sequence of relations . As relational inference is unsupervised , we use simulated dynamics physics systems as ground truth for validation . We find that the performance of the static NRI model deteriorates significantly with shorter output trajectories , making it unsuitable for dynamic relational inference . In contrast , DYARI is able to accurate infer the hidden relations with various dynamics scenarios . We also perform extensive ablative study to understand the effect of inference period , training schemes and model architecture . Finally , We showcase our DYARI model on real-world basketball trajectories . In summary , our contributions include : • We tackle the challenging problem of unsupervised learning of hidden dynamic relations given multi-agent trajectories . • We develop a novel deep generative model called DYARI to handle time-varying interactions and predict a sequence of hidden relations in an end-to-end fashion . • We demonstrate the effectiveness our method on both the simulated physics dynamics and real-world basketball game play datasets . 2 RELATED WORK . Deep sequence models Deep sequence models include both deterministic models ( Alahi et al. , 2016 ; Li et al. , 2019 ; Mittal et al. , 2020 ) and stochastic models ( Chung et al. , 2015 ; Fraccaro et al. , 2016 ; Krishnan et al. , 2017 ; Rangapuram et al. , 2018 ; Chen et al. , 2018 ; Huang et al. , 2018 ; Yoon et al. , 2019 ) . For GAN-like models , ( Yoon et al. , 2019 ) combine adversarial training and a supervised learning objective for time series forecasting . Liu et al . ( 2019 ) propose a non-autoregressive model for sequence generation . Compared with GANs , VAE-type models can provide explicit inference and are preferable for our purpose . For instance , Chung et al . ( 2015 ) introduces stochastic layers in recurrent neural networks to model speech and hand-writing . Rangapuram et al . ( 2018 ) parameterizes a linear state-space model for probabilistic time series forecasting . Chen et al . ( 2018 ) ; Huang et al . ( 2018 ) combine normalizing flows with autoregressive models . However , all existing models only model the temporal latent states for individual sequences rather than their interactions . Relational inference Graph neural networks ( GNNs ) seek to learn representations over relational data , see several recent surveys on GNNs and the references therein , e.g . ( Wu et al. , 2019 ; Goyal & Ferrara , 2018 ) . Unfortunately , most existing work assume the graph structure is observed and train with supervised learning . In contrast , relational inference aims to discover the hidden interactions and is unsupervised . Earlier work in relational reasoning ( Koller et al. , 2007 ) use probabilistic graphical models , but requires significant feature engineering . The seminal work of NRI ( Kipf et al. , 2018 ) use neural networks to reason in dynamic physical systems . Alet et al . ( 2019 ) reformulates NRI as meta-learning and proposes simulated annealing to search for graph structures . Relational inference is also posed as Granger causal inference for sequences ( Louizos et al. , 2017 ; Löwe et al. , 2020 ) . Nevertheless , all existing work are limited to static relations while we focus on dynamic relations . Multi-agent learning Multi-agent trajectories arises frequently in reinforcement learning ( RL ) and imitation learning ( IL ) ( Albrecht & Stone , 2018 ; Jaderberg et al. , 2019 ) . Modeling agent interactions given dynamic observations from the environment remains a central topic . In the RL setting , for example , Sukhbaatar et al . ( 2016 ) models the control policy in a fully cooperative multi-agent setting and applies a GNN to represent the communications . Le et al . ( 2017 ) models the agents coordination as a latent variable for imitation learning . Song et al . ( 2018 ) generalizes GAIL ( Ho & Ermon , 2016 ) to multi-agent through a shared generator . However , these coordination models only capture the global interactions implicitly without the explicit graph structure . Tacchetti et al . ( 2019 ) combines GNN with a forward dynamics model to model multi-agent coordination but also requires supervision . Grover et al . ( 2018 ) directly models the episodes of interaction data with GNs for learning multiagent policies . Our method instantiates the multi-agent imitation learning framework , but focuses on relational inference . Our approach is also applicable to dynamic modeling in model-based RL .

Summary: This paper builds on Kipf et al. (2018)’s Neural Relational Inference. In particular, this work introduces a latent variable model which treats the interactions (i.e. relations) between different agents as dynamic and time-varying. As in NRI, the interaction variable between any two agents is conditioned on the history of those agents’ states. An agent’s future state is conditioned on its history of states as well as its interaction variables with other agents.