Neural Relational Inference with Node-Specific Information 
Authors: Ershad Banijamali
Venue: ICLR 2022 Poster (2022)
Keywords: Graph Neural Networks
Variational Inference
Trajectory Prediction

Abstract: Inferring interactions among entities is an important problem in studying dynamical systems, which greatly impacts the performance of downstream tasks, such as prediction. In this paper, we tackle the relational inference problem in a setting where each entity can potentially have a set of individualized information that other entities cannot have access to. Specifically, we represent the system using a graph in which the individualized information become node-specific information (NSI). We build our model in the framework of Neural Relation Inference (NRI), where the interaction among entities are uncovered using variational inference. We adopt NRI model to incorporate the individualized information by introducing private nodes in the graph that represent NSI. Such representation enables us to uncover more accurate relations among the agents and therefore leads to better performance on the downstream tasks. Our experiment results over real-world datasets validate the merit of our proposed algorithm. 

Published as a conference paper at ICLR 2022
NEURAL RELATIONAL INFERENCE WITH NODE-SPECIFLL INFORMATION
Ershad Banijamali Amazon Alexa AI'
Toronto, Canada
ebani jame ama zon. com
ABSTRACT
Inferring interactions among entities is important problem in studying dynamical systems, which greatly impacts the performance of downstream tasks, such as prediction. In this paper, tackle the relational inference problem in setting where each entity can potentially have set individualized information that other entities cannot have access to. Specifically, we represent the system using a graph in which the individualized information become node-specific information (NSI). We build our model in the framework of Neural Relation Inference (NRI), where the interaction among entities are uncovered using variational inference. We adopt NRI model to incorporate the individualized information by introducing private nodes in the graph that represent NSI. Such representation enables us to uncover more accurate relations among the agents and therefore leads to better performance on the downstream tasks. Our experiment results over real-world datasets validate the merit of our proposed algorithm.
INTRODUCTION
Our world includes many different types of systems that involve multiple entities interacting with each other, from biology to sports, from social media to driving situations. Modelling the behaviour of such dynamical systems is challenging task, which requires uncovering different types of interactions among the entities and how they affect each other.
Recent approaches in machine learning learn the interactions among the entities through graph-based models and attention- models, in which the representation of an entity is updated through its relationship with other entities. Such relationships are usually either predefined learning. Kipf et al. (2018) proposed neural relational inference (NRI) for relationship uncovering in the framework of variational inference (Kingma & Welling, 2013; Rezende et al., 2014). NRI has an encoder-decoder structure in which the latent codes represent different types of interactions among the entities. The distributions over the latent variables are inferred based the input features of the entities and graph structure is formed by sampling from these distributions. The decoder of the model is a based algorithm that runs over the uncovered graph to update the features of entities. The model shows prominent performance over many synthetic and real-world datasets.
uncovered by
Uncovering the interaction among entities in NRI and other models in the GNN literature is studied in problems where the features of the entities are completely observable and the GNN-based algorithm also run on the observable features. However, in many real-world problems there is set of hidden features that affect the way entities interact with each other. For example consider the problem of predicting the future position of vehicles in a driving scenario. In order to uncover the relation of target vehicle with other vehicles not only should observations from vehicles, e.g. their trajectories up to the current time, but also we should take into account the intention of the target vehicle. In fact, intention, which can be immediate action or a longer term goal, forms a set of features that is only accessible by the target vehicle and other vehicle cannot know it. In this paper we call such feature the individualized features Node-Specific Information (NSI) in a graph representation of the system. Formally, NSI is a set of features that is
consider the features extracted from the
Work was done prior joining Amazon.

---

Published as a conference paper at ICLR 2022
only accessible by one node in graph structure but affects the interactions of that node with other nodes. Our goal is to efficiently exploit NSI to build more accurate graph structures and consequently achieve better performance in the downstream tasks. Towards this goal, the first step is to find proper representation for NSI in our graph structure. We propose introducing a new set of nodes in the graph that earry NSI and call them private nodes. Observable features of the entities are then denoted by public nodes. Therefore each entity can be represented by a public and private node. We introduce our model in the framework of NRI, i.e. a variational inference model that uncovers different types of interaction among the entities. We carefully design the encoder and decoder part of our model to ensure that NSI remains an individual feature for one entity and does not affect the interaction modelling of the other entities. At the same time, through experiment we show that such modelling of NSI is very efficient and can result in significant improvement on the performance on downstream tasks. The main contributions of this work are the followings:
To the best our knowledge the problem having individualized features for each entity has not been previously studied in the framework of relational inference. We tackle this problem by introducing a new set of nodes in a graph structure that represents the individualized information. These nodes are used during the process of relational inference well as performing the downstream task, i.e. trajectory prediction in our case.
We show that our proposed model can exploit NSI efficiently by introducing minimum additional computational complexity compared to the original NRI model and its variants.
The results of our experiments on real-world datasets show that our model can outperform the baselines and achieve the state-of of-the--ttt-tt-- art results on the defined tasks.
2 RELATED WORK
Interaction modelling: Relational learning is a popular approach for the problems with dependency structure among the data points. Such dependency can be predefined in some domains. But in many domains it has to be learned. For example approaches like locally linear emhedding (LLE) (Roweis & Saul, 2000) and Isomap (Tenenbaum et al., 2000) use ANN for forming such relationships based on different measures of similarities among the data points. More recently, neural networks have become the dominant tools for learning these dependencies based on different architectures and paradigms (Kipf & Welling, 2017; Hamilton et al., 2017; Garcia Duran & Niepert, 2017; Monti et al., 2017; Veličković et al., 2017; Franceschi et al., 2019).
The most relevant works to our proposed model are NRI (Kipf et al., 2018) and dynamic NRI (dNRI) (Graber & Schwing, 2020), which are discussed in more details in the next section. Recently, Li et al. (2020) introduced similar ideas for multi-modal trajectories prediction. Extensions of NRI in other directions than ours also appeared in the literature. For example, Li et al. (2019) tries to uncover interactions by imposing some structural constraints on the prior and Webb et al. (2019) introduces the idea of factorized graph for NRI.
The other common approach for uncovering relationships among the entities is based the idea of attention. This idea has been used in Narasimhan et al. (2018); Hoshen (2017); Van Steenkiste et al. (2017); Garcia & Bruna (2017); Monti et al. (2017); Veličković et al. (2017), where the attention mechanism is the main tool for interaction uncovering, however, it is also used as a building block for GNNs.
Future trajectory prediction as evaluation metric: We define our problem as uncovering the interactions of entities in multi-agent dynamical system, where the evaluation is based the accuracy of future trajectory prediction. Trajectory prediction is in fact an important problem in many multi-agent systems, including the ever growing area of autonomous driving. In fact, our approach falls into the category of nultivariate time-series prediction (Yu et al., 2018; Wu et al., 2019; Sen et al., 2019; Salinas et al., 2020; Rangapuram et al., 2018; Li et., 2018; Bai al., 2018), which the prediction is based on the relationship among the series. Specifically we use the relationship of the entities in the GNNs framework. GNN has been widely used in the trajectory prediction, especially in the application of autonomous driving and significantly improved the performance in this area. For example, Salzmann et al. (2020) uses GNN to capture the relationship among different road users (vehicles and pedestrian). Gao et al. (2020) uses graph attention networks (GATs) to learn the relationship among agents and different components of map data, and Liang et al. (2020) uses graph convolutional networks (GCNs) to learn the interaction among lanes and vehicles.
2

---

Published as a conference paper at ICLR 2022
In our experiment consider scenarios in which the goal (final) position of the entities is given as the individualized information. In the context of goal-aware prediction, there have been some effort in the area of autonomous driving that are not based on explicit relational learning (Rhinehart et al., 2019; 2018). In these papers, the prediction is based on an autoregressive flow-hased model that considers a collective observation of all entities make prediction for each of them. The goal is then added at the inference time and the latent codes are optimized in a way that the target entity reaches its goal.
3 BACKGROUND: NEURAL RELATIONAL INFERENCE (NRI)
NRI is an unsupervised model that learns infer the interaction types among entities in multi-agent systems in order to model the dynamics of the system. The model is defined as the problem of predicting the future trajectory of entities given the past trajectories. Formally, the trajectories of
N entities are given for T time steps. Entity is denoted by x; = =x,, ..... x, The set of all trajectories at time / is denoted by x' = {x], xh,... xyy and x = (x* x² ,..., x²) denotes the whole trajectories for all agents. NRI tries model the system by maximizing the log-likelihood of
the observations, log p(x), in the framework of variational inference, i.c. maximizes the evidence lower-bound (ELBO):
((,,0) () = (xtx)) po []] (l.(x((2))))lpgg
where latent variable z has a categorical distribution and represents the interaction among entities. More specifically, Z,z is K K-dimensional vector that denotes the type of interaction between entities x; and x, The entities in the NRI model are represented using nodes of graph' and therefore the interactions are directed edges on this graph. Parameters of p(.) and q(.) models are denoted by 0 and @, respectively. The three probability distributions in Eq. 1 are:
The variational posterior, 90 (z)xx, is implemented using amortized inference parameter- ized by neural network, namely the encoder network. Given the input trajectories, the encoder network predicts the type of edges on the graph. The latent variable z is assumed to have categorical distribution. Samples from this distribution form the edges of the graph. In order to backpropagate the error signals to the encoder layers, we need to make the sampling process differentiable. In NRI this is done by approximating the posterior distribution and reparamterezaatin of Gumbel distribution (Jang et al., 2017; Maddison et al., 2017):
Zij softmax (() + g)/r)
where h( is the last output of the encoder before the softmax layer and g E R shows i.i.d. samples drawn from Gumbel 1) and 7 is a hyperparameter that controls the smoothness of the distribution.
The prior, p(z) = Ilizy p(Zig) ), is assumed to be a factorized uniform categorical distribu- tion over the edges.
The likelihood, (x|),, is implemented by the decoder network and predicts the future trajectories given the uncovered structure of the graph.
The prediction in NRI is done in an autoregressive fashion. However, the ground truth trajectory is fed to the model for few steps during the training to improve the performance of the decoder (teacher forcing). NRT in its original form has two main shortcomings:
1. The latent variable which defines the edge types, is fixed for the whole prediction horizon. That is, the uncovered interactions among the agents are assumed to be fixed over the next time steps. This is not necessarily valid assumption as the agents can dynamically change their interactions in the system.
Thhouuhhot the paper, entities and nodes as well interactions and edges are used interchangeably, based on the context.
3

---

Published as a conference paper at ICLR 2022
2. The prior distribution is assumed to be uniform and not conditioned on the previous obser- vations. Both of these assumptions can degrade the performance of the model in longer prediction horizons since samples from the prior provides minimum information about the input.
More recently, Graber & Schwing (2020) pointed out the above issues and addressed them in dynamic neural relation inference (dNRI) model. The conditional prior distribution in dNRI is defined as:
T
p(z|x) == II p(z
bit z 2
(3)
1=1
which is implemented by another set of MLP and LSTM layers that form the encoder of the po(.) model. The experiments show that, by resolving those issues, dNRI achieves better prediction performance than NRI.
4 MODEL DESCRIPTION
4.1 PROBLEM STATEMENT
We define our problem in the setting of NRI. However, we assume that, in addition to the observable feature set x, each entity can potentially have access to a set of individualized features cf at each time step f. of cannot be observed by other entities 2. Similar to the observable features, we denote the set of individualized features for each agent by C, (c) c², ..... ).),, set of individualized features for all agents at time / by c = {cf, cs..., ..., cyy ). and set of all individualized features by (c*, c², ...,, c²'). We still study the problem of modelling the dynamics of the system through predicting the future trajectories of the entities. However, interaction among the entities are inferred more accurately and, therefore, provide a better model for the underlying dynamics of the system, which can lead to better prediction performance.
aim to exploit the individualized features in a way that the
4.2 REPRESENTATION OF INDIVIDUALIZED FEATURES IN THE GRAPH: PRIVATE vs PUBLIC NODES
In order to build the interaction inference model, need to first represent the individualized features in the graph. Note that simply augmenting the observation features x; with the individualized features and building a new set of features, e.g. y: = f(x, c,). is not solution here, as this will affect the interaction uncovering among the agents, which is in contrary with our initial assumption about the accessibility of individualized features.
Sniky
x]
Here, we propose adding a new set of nodes to represent the individualized features and we call these nodes pri- vate nodes. We also refer to the nodes that represent the observable nodes as public nodes for clarification. There- fore each entity i at each time t can be shown using a public node and private node corresponding to x² and cf, respectively. A private node is only accessible by its corre- sponding public node while public node is accessible by all other public nodes. Interaction among public nodes and their corresponding private node is always fixed, while interaction among public nodes are learned. Fig. shows the uncovered graph of an example system time step Note that there is always an edge between the public node and its corresponding private node, which shown in black in the figure. This edge represents all types of interactions. Representing the individualized features using the private nodes allows us to employ unified framework to uncover the graph, without creating a computational overhead.
Blue dots and black dots show the public and private nodes, respectively. Dif- ferent type interactions among entities are depicted as directed edges among the nodes with different colors.
Figure 1:
Although, in some cases the effect of individualized features might be observed in the future times steps through x (k > 0). For example the intention of driver is a hidden feature but its effect can be observed through the future trajectory of the car. Note that not all hidden features have this property. Whether this property holds does not affect the performance of model
4

---

Published as a conference paper at ICLR 2022
Unchared pal.)
ESTR
pola' x!" (ehe
GNN Vyp
Public mode escoding
GNN (shared escodee)
Sumple i.
Muhti-head graph encoding
enceding
Syta
JEASEM Undhared encoder
Uncovered graph with all edge types
Figure 2: Proposed model. Encoded observable individualized features are fed GNN model to update the node features. The output of the GNN is fed to both pa (.) and qo(.) encoders to provide the uncovered interactions among the agents. The parameter of pa(.) and ga(.) distributions are learned to minimize their KL divergence. Moreover, during the training the edges randomly drawn from the pa(.) encoder to better optimize the parameters of this model. The sampled edges together with the input features fed the decoder model to output the prediction at cach time step.
4.3
MODEL COMPONENTS
In order to learn the interactions in our model we maximize the ELBO of the following form:
NRI-NSI (0, () = Ex [log plog (x (x|z, z, c)] - KL (2|x, c)llpa x, c)]-
(4)
The conditional probability distributions are parameterized by neural networks and factorized as:
4e (2/2x, c)
II ls ( (x
2 1:))
a x 1:T 1:t-1 .c.")) 1.7)),
(5)
J#
po(z|x, c)
II po (z" 2 1tt1 1 ,.,,
po
po(zlylx* lit 1:--1 ,,,,),
(6)
f=
N T
po(x|z, c) = II po (x"
x
1:t
Pa
X
x
1:t
.2", c):t).
j=1t=1
Note that at each time step, the edge uncovering and prediction for each public node depend only on its own private node. Here we describe the details of each component.
4.3.1 CONDITIONAL PRIOR NETWORK Po (z(z|x,c) c)
The prior distribution forms the different types of interactions given the past and current status of public and private nodes at each time step using the autoregressive model in Eq. 6. We use a GNN architecture for message passing where the graph is fully-connected for the set of public nodes and for the private nodes there is only an edge towards their corresponding public node.
node embedding:
m S (xx((x) (x)) n³ = Jemb (c)),
(8)
h (as),1 = So. ([m) m2)).
(9)
e + D:
hj = f.( Σ b(c.).)),
(10)
h (. = f2 (2h, o f2 (b), h/. n()),
(11)
where denotes Hadamard product and v and represent nodes and edges the graph, respectively. Note that using our proposed represenetation, can handle the general setting in which not all entities necessarily have individualized features. In the case that these features are not provided for an entity, f2(.) is masked and replaced by an all one vector. By using two levels of message
5

---

Published as a conference paper at ICLR 2022
passing make that all observable nodes are considered to infer an edge. In the first level only the observable features are used so that the final relationship uncovering for each entity is not affected by other entities' individualized features. We use multilayer perceptron (MLP) layers for all of the f(-) functions. In order to take into account the previous interactions, the emhedding h h( at each time step is fed to layers of LSTM followed by MLP and softmax to output the actual conditional prior distribution:
h (1)),p LSTM(h (1.).2) h (.)))))
(12)
po (olg)) xiit z 2itt-1 c..t) softmax (b
(13)
4.3.2 VARIATIONAL POSTERIOR NETWORK (z)x)x,c) c)
The variational posterior approximates the true posterior. During the training the graphs are con- structed based on the samples from this distribution. According to Eq. 5, we use the whole set x and from time 1 to 7" to infer the edges from 4s (z|x, c). This will result in predicting more accurate edge types during training, compared to the prior model po (z)zs xx c). Minimizing the KL divergence in Eq. 4 allows to transfer knowledge about the future to the prior model by minimzing the gap between po(2) x, c) and go [2|x,,0 c). We employ two ideas to help this minimization:
Parameter sharing between (z)x,, x, c) and po(z)x, c): We use the same networks in Eq. 8-11 to obtain the edge information. Then a bi-directional LSTM is used to combine this information from previous and future time steps. The output goes to MLP layer followed by softmax. This will allow identical representation learning for the shared part and the KL divergence minimization should be only applied the unshared parts of the two models. Note that this is essentially different from the model in Graber & Schwing (2020), in which Po (z(z) c) is basically sub-module of qo (z)x, c) and therefore variational posterior cannot contribute in learning a better edge uncovering beyond the prior model. Our variational distribution given by:
h Has)) = Bi-LSTM (1.). h (o (15).g)
(14)
9s (z(ziz) |x
1t-11 c.tt) softmax (b(ss)))))
(15)
We use reparameterization trick with Gumbel distribution to backpropagate error during the training.
Sampling from the prior during the training: In order to further close the gap between the two distributions, a% of the time the edges are sampled from the prior distribution and fed to the decoder. Therefore the parameters of the prior encoder are trained with the prediction loss as well as the KL divergence loss.
4.3.3 DECODER NETWORK po(x|z,
The decoder network, takes the uncovered graph as the input and makes predictions by running a GNN-based algorithm on different edge types. The outputs are then concatenated and go through a decoder function, fa(-), which an LSTM for autoregresive prediction. Note that at each time step the private nodes affect both edge uncovering in the encoder and updating nodes states for prediction in the decoder. Similar to NRI, in order to improve the optimization of the decoder parameters during the training, we use teacher forcing and feed the model with ground truth (instead of previous predictions) for the first 10 time steps.
+ e
h b(S (ft x(I) (x c,D)).
(16)
6-> U::
11/4 - 2 (onn
(17)
11 xf + LLMMM(Cnncat, (ha (h), 14),
(18)
(x z c.,,)) = - -N
(19)
where k is the type of interaction, is fixed variance and is set during the training using the validation set, and all (() functions implemented by layers of MI..

---

Published as a conference paper at ICLR 2022
5 EXPERIMENTS
We perform our experiments on two different tasks, i.e. goal-conditional prediction and action- conditional prediction. For both tasks we consider a multi-agent system in which at least one agent has access to its individualized features. Both of these tasks are of great interest in the context of trajectory prediction, with important downstream applications such planning. For the goal- conditional task the individualized feature is the final goal (position) of the agent. Therefore, this information is fixed for the whole prediction horizon or at least for multiple time steps, c = g for I 1. For the action-condition conditional task the individualized feature is the next action of the agent, which changes at every time step, cf = u². We refer to our model as NRI-NSI. In all of our experiment we use ADAM optimizer (Kingma & Ba, 2015) with learning rate 0.0001.
Metrics: Since our final task is trajectory prediction of the entities, we use minimum average displacement error (minADE) and minimum final displacement error (FDE) as the evaluation metrics. ADE: average mean square error (MSE) over all time steps between the ground truth future trajectory and the predicted trajectory. FDE: MSE between the final ground truth position and the predicted final position. Note that since we are using a stochastic model in the decoder, the minADE and minFDE are the closest sample to the ground truth over 20 different sampled predictions. We follow this scheme for all of the baselines, too.
Baselines: For both action-conditional and goal-conditional prediction, we compare our model with NRI and dNRI where the individualized features are fed to the model at the last stage of decoder for each the entities, i.c. before outputting the distribution of the predictions.
5.1 ACTION-CONDITIONAL PREDICTION
For this task we further consider FM-MPUR (Henaff et al., 2019) baseline. FM-MPUR is also an action-conditional prediction model based on conditional variational autoencoders (CVAEs) that aims maximize the lower bound of log p(x*+1:7) x u.:T-1). ). The model uses digh-dimensional features (images) of the environment that help reasoning about the interaction of the agents.
5.1.1 NGSIM I-80 DATASET:
We study this problem in the domain of autonomous driving. The Next Generation Simulation program's Interstate 80 (NGSIM 1-80) (Halkias & Colyar, 2006) dataset consists of 3 batches of 15-minute recordings from traffic cameras mounted over stretch of a highway in the US. Driving behaviours are complex with complicated interactions between vehicles moving at high-speed. We consider 7 different agents at each time step, where the state of each agent given by its coordinates and velocity. We assume that for one of the agents (ego-agent), the action is given for every time steps. Actions are two-dimensional vectors that represent acceleration and rotation of the steering wheel. The other 6 vehicles are the closest ones to the ego-agent in different lanes (same lane/left lame/right lane) at time t = 1. We keep the state of these vehicles until the end of prediction horizon. We consider 4 relation types for NRI, dNRI, and NRI-NSI.
Table 1 shows the results of different models. We sample the data at 2Hz and perform the prediction task for different prediction horizons. FM-MPUR performs worse than dNRI and NRI-NSI, due to the lack of explicit dynamic interaction modelling. To better illustrate the capability of our model, we consider two different scenarios for NRI, dNRI, and NRI-NSI. First, we use the models without the actions and then add the actions to see how this affect the results and how efficient is our modelling of the individualized features. dNRI outperform NRI for both scenarios, but both of models almost fail to take advantage of the additional information about the actions. On the other hand, NRI-NSI outperform all of the baselines for both with and without action scenarios and obtain a significant gain by adding the actions, meaning that our model can predict more accurate interactions given the actions of the ego-agent. Also our model outperforms dNRI even in the scenario with no actions, which can be associated to the better learning of the prior encoder, as described in section 4.3.2. Moreover, to see how this additional information changes the prediction of the agents with and without the individualized features, we report the results for the set of all agents, which includes the ego-agent, and the set of other agents which excludes the ego-agent. Given the action of the ego-agent, approximating the next state of this agent is easy. Therefore
see that results for all
7

---

Published as a conference paper at ICLR 2022
min DDE (all other)
Model
Is
2x
46
FM MAMU
0.34/0.84
0,38/0.94
0.41/1.06
0.45/1.19
0.54/1.44 0.61/1.68
0.64/1.75
0.7001.85
NRI no action
0. 0,39/0.9
0.50V
0.68/1.78
0.75/2.01
NRI with action
0.36/U888 0.38/0.91
0.44/105
0,49/1.19
0.62/1667 0.67/1183
0.70/1.93
0.7011.3-0..5/2..00
0..75/2.02
dNRI no
0.330.85
0.39/L01
0,49/1.24
0.57/1.5
dNRT with action
0. 0.27/0.666 0.32/0.72
0.34/0.95 0,38/1.03
0.43/1. 18 0.48/1.30
0.51/1.34
0.57/1 .54
NRI-NSI NSI no action
0. 0,28/0.61
0.33/0.90
0,42/1.03
0.49/1.52
NRI-NSI with action
0.20/0.51 0.230.58
0.26/0.63 0.29/0.99
0.34N0.75 0.37/0.83
0.42/1.02 0.45/1.17
Table 1: Performance of models on action-conditional task for different prediction horizons
minADE/minFDD other)
Model
Is
2x
4s
NRT
1.0 03-0.1200.16
0.14/0.20- 0. 16/0.24
0.24/0 38 -4.00.8--.2200.44 0.28/0.4
0.30/0.60 1..0060-00.5/06.6
GNRI
0.09/0.12 1 12-00.100000000000.00000011 0.100.114
0.12/0.17 11220.17-0.. 15/0.23
0.20/0.31 0.2550..35
0,27/0.50 3.3000030.....
Trajectron++
0.10/0. 13 1010..
0.12/0.18
0.19/0.33 0.2330.39
0.28/0.54 0.34/0.60
PRECOG
0.06/0.08 0.06/0.09
0,09/0,
0. 10/0.
0.16/0.25
-..00.5--0188)) 18/0/
0.25/0.48-
0.29/0.58
NRI-NSI
0.06/0.08 0.06/0.09
0.07/0.11
0.08/0.12
0.12/0.19 0.13/0.21
0.18/0.31 0.20/0.36
Table 2: Performance of models goal-conditional task for different prediction horizons the Basketball
dataset.
Model
minADDEmiiHHD (all-other)
3%
NRI
0.27/0.54 - 0.3189.60
0.38/0.80 - 0.42/0.88
0.35/1.24-066311
0.88/1.98 0.98/2.12 12
dNRI
0.23/0,52
0.27/0.56 54
0.34/0.77- 0.42/0.
0.51/1.08
0.59/1.22
0.78/1.62 0.86/1
Trajectron++
0.24/0.49- 11499-000 0.27/0.52
0.35/0.74 0,39/0.77
0,48/0.97
0,54/1.08
0,76/1.68 68 0.88//197
PRECOG
0.21/0.47- 0.24/0.49
0.33/0.68 0.37/0.74
0.44/0.85 0.49/0.96
0.73/1.55 0,82/1.73
NRI-NSI
0.18/0.37 10037700.0 0018/0.38
0,27/0.54 0,290.56
-.800.78-0.4000330 0.40/0.80
0.62/1.34 0.64/1.37
Table 3: Performance of models on goal-conditional task for different prediction horizons on the nuScenes dataset.
models gets worse by removing the ego-agent. Note that for the scenario without the actions these two results are the same.
5.2 GOAL- LOLLICONDIIIONAL PREDICTION
We consider the goal-conditional prediction on two different sets of data with different types of interactions, i.e. basketball dataset and an autonomous driving dataset. For this task we also consider two other baselines. Trajectron++ (Salzmann et al., 2020), which uses spatio-temporal information for relational reasoning. It employs dynamical models for the agents in the scene to produce feasible trajectories for the entities. For this model we also inject the goal information at the last level of decoder for each agent. We also consider PRECOG (Rhinchart al., 2019), which performs goal-conditional prediction using flow-based generative model. The goal-conditioned prediction done by optimizing the samples of the latent code of trained model.
5.2.1 BASKETBALL DATASET
We study basketball players trajectory dataset (Yue et al., 2014). The dataset consists of trajectories of 5 players in the offensive team. We use the same preprocessing as Graber & Schwing (2020). The trajectories at each time step consist of position and velocity of the players. The length of each trajectory is almost 8 seconds, which is sampled at 5Hz (almost 40 frames per trajectory). All models are trained given 20 frames as the input. The prediction horizon varies from 5 to 20 frames. For this experiment we randomly choose players and assign them their goal their individualized features. The game is very agile, therefore we update the goal every 3 time steps. For this dataset none of the models use any context information and only use the trajectory of the players. We consider 2 relation types for NRI, dNRI, and NRI-NSI.
Table 2 shows the results for this dataset. Again, we report the results for both all agents and other agents (set of agents with no individualized features). For Trajectron++ their dynamical model cannot fully capture the motion of the players, therefore we removed this module from the model for this dataset and directly predict the next state. We report the results for different prediction horizons and as we can see NRI-NSI outperforms the baselines significantly. PRECOG also performs well on this task as it also sy systematically optimize the latent codes for coal-conditional prediction during inference. However, our model outperforms PRECOG almost across all horizons because in NRI-NSI the condiiinnii prediction is done systematically during both training and inference time and also the interaction learning is done explicitly at each time step. Therefore, the performance gap between our model and other baselines enlarges as the prediction horizon increases.
8

---

Published as a conference paper at ICLR 2022
IRIADE JNCSIM I-0 datwwet)
MINDEE (Basiorbbal dertsoet]
minAEE (nuScenes datared)
1014)
0.85
12
0.75
0.65
joinett
0.25
0.85
0.7
0.15
minFOr (NGSIM 1-80 dataset)
RisFDE (Boskethall datase))
minFDE (nuSoenes dataset)
2.5
1.35
1.35
1.5
1.15
0.55
0.75
0.35
0.5
0.35
0.15
Figure 3: Result of action conditional (left column) and goal-conditional (middle and right columns) experiments on different datasets based on minADE and minFDE for different time horizons. We can see that our model outperforms other model across all tasks, Specially for the goal-conditional tasks the performance of our model drops less than other models from the of all agent to the other agents
5.2.2 NUSCENES DATASET
nuScenes dataset Caesar et al. (2020), is a real-world dataset for multi-agent trajectory prediction. consists of 850 episodes of 20 seconds of driving, recorded at 2Hz. The dataset includes the positions of all agents together with synchronized context information, e.g. map. We encode the context information using convolutional neural network (CNN) and use the encoded information as node in our graph that all entities have access to, similar to Li et al. (2020). This is also done for NRI and dNRL. We use 2 seconds of past trajectories and predict up to 4 seconds into the future. Individualized features are assigned to 2 agents in each scene at random. We set the goal as the last position of the agents for each horizon. Similar to the NGSIM I-80 dataset we use 4 relation types here.
Table 3 shows the results for this model. Similar to the Basketball dataset our model and PRECOG outperform other baselines. Another interesting observation is that our model performs very similarly on the set of all and other agents. This means that the additional information provided by the goal is effectively propagated to the whole set of agents each time step (through the observable features from the previous steps) and therefore the interactions for all agents are inferred more accurately. However, performance of other baselines is heavily biased in favor of the agents with the individualized features. Therefore, when we remove these agents the values of minADD/ for other agents drop more. This can be seen across all tasks Tables 1-3, Fig. 3 better demonstrates these results.
6 CONCLUSION
We considered a novel problem in the framework of relational inference for multi-agent systems in which each entity can potentially have access to set of individualized information that affects the relationships among the entities. We tackled this problem by assuming a graph structure for the system and proposed to represent the individualized information using private nodes on this graph, whereas the observable features of the agents are the public nodes. We then used variational inference to uncover the edges among the public nodes of the graph. Through experiments with real-world datasets, we showed that our model is capable of making predictions with high accuracy where the individualized information is the next action or the goal of the agents. Note that for all of the baselines the goal of all entities (last position of the entity) is given during the training of the model. However, we showed here that our model can efficiently and explicitly exploit such information to form better interaction among the entities, which is reflected in better prediction accuracy.

---

Published as a conference paper at ICLR 2022
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:/803.0/27/ 2018.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1661--116 1631, 2020.
Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. Learning discrete structures for graph neural networks. In International conference machine learning, pp- 1972-1982. PMLR, 2019. 2
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11525-11533 2020. 2
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. arXiv preprint arXiv: 2017. 2
Alberto Garcia Duran and Mathias Niepert. Learning graph representations with embedding prop- agation. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL https: http:p::/pppoeeeeeeee /ttp///ooceeccooooooooooccoccccc ngs neurips wwtppppp//0111111 /fiie/ e068881. 3 958a1 9e08 4555772222225555544--4--0-
pdf. 2
Colin Graber and Alexander Schwing. Dynamic neural relational inference. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8513-8522, 2020. 2, 4,
John Halkias and James Colyar. Ngsim interstate 80 freeway dataset. US Federal Highway Adminis- tration, FHWA- 1ETA--RT-06-1333 Washington, DC, USA, 2006. 7
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: /ttpp::/proceeeeee http:///ooooooeoeeeeeeeee wwwpiinnnnnneuurrree cc/ cc/papper/2117/E1ee 5dd9db5e 033 9owfbaaaaoaaaaaeeeeeaaea aa
pdf.2
Mikael Henaff, Alfireoo Canziani, and Yann LeCun. Model-predietive policy learning with uncertainty regularization for driving in dense traffic. arXiv preprint 1rXiv:1901.02705, 2019. 7, 13
Yedid Hoshen. Vain: Attentional multi-agent predictive modeling. arXiv preprint arXiv: arXiv:1706.000 1706.06122
2017. 2
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations (ICLR), 2017. 3
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pp. 2688- 2697. PMLR, 2018. 1,
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. 2017. 2
Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. Proceedings of the Neural Information Processing Systems (NeurIPS), 2020. 2,
10

---

Published as a conference paper at ICLR 2022
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference on Learning Representations, 2018.
Yaguang Li, Chuizheng Meng, Cyrus Shahabi, and Yan Liu. Structure-informet graph auto encoder for relational inference and simulation. In ICML Workshop on Learning and Reasoning with Gruph-Structured Dula, volume 8, 2019. 2
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In European Conference on Computer Vision, pp. 541-556. Springer, 2020. 2
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. 2017. 3
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1124-5124 2017. 2
Medhini Narasimhan, Svetlana Lazebnik, and Alexander G Schwing. Out of the box: Reasoning with graph convolution nets for factual visual question answering. 2018. 2
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances neural information processing systems, 31:7785-7794, 2018. 2
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of The 31st International Conference on Machine Learning, pp. 1278-1286, 1286, 2014.
Nicholas Rhinehart, Rowan McAllister, and Sergey Levine. Deep imitative models for flexible inference, planning, and control. arXiv preprint arXiv: arXiv: 1810.06544, 2018. 3
Nicholas Rhinchart, Rowan McAllister Kris Kitani, and Sergey Levine. Precog: Prediction condi- tioned on goals in visual multi-agent settings. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2821-2830, 2019.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290(5500):2333-23222 2000.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):
1181-1191, 2020.
Tim Salzmann, Boris Ivanovic. Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically- feasible trajectory forecasting with heterogeneous data. In Proceeding of Europe Conference on Computer Vision (ECCV), 2020. 2, 8
Rajat Sen, Hsiang- Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https::/ppocceedinnie newriisesssss ips.ipp.... cc/ paper/2011 file/ httpe///000044eeeeeeeeee www.:7/eeedd00335e11111
Paper. pdf.
Joshua Tenenbaum, Vin De Silva, and John C Langford. global geometric framework for nonlinear dimensionality reduction. science, 290(5500) 2000. 2
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In ICLR, 2017. 2
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR, 2017. 2

---

Published as a conference paper at ICLR 2022
Ezra Webb, Ben Day, Helena Andres- Terre, and Pietro Lió. Factorised neural relational inference for multi-interactonn systems. arXiv preprint 1rXiv:1905.08721, 2019.
Z Wu, S Pan, G Long, J Jiang, and C Zhang. Graph wavenet for deep spatial spatial-temporal graph modeling. In The 28th International Joint Conference on Artificial Intelligence (IJCAI). International Joint Conferences on Artificial Intelligence Organization, 2019.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: deep learning framework for traffic forecasting. In Proceedings of the 27th International Joint Conference Artificial Intelligence, pp. 3634-3640, 2018. 2
Yisong Yue, Patrick Lucey, Peter Carr, Alina Bialkowski, and lain Matthews. Learning fine-grained spatial models for dynamic sports play prediction. In 2014 IEEE international conference on data mining, pp. 770-679. IEEE, 2014. 8
12

---

Published as a conference paper at ICLR 2022
A
APPENDIX 1: IMPLEMENTATION DETAILS
A.1 ARCHITECTURE DETAILS:
Here we provide the architecture of the networks according to Eq. 8-19.
Jemb layers of MLP with 256 and 128 units and ELU activation
Jemb layers of MLP with 256 and 128 units and ELU activation
fo layers of MLP with 256 and 128 units and ELU activation
fe layers of MLP with 256 and 64 units and ELU activation
f2 layers of MLP with 256 and 64 units and ELU activation
f 2 layers of MLP with 256 and 256 units and ELU activation
LSTM of (z) c): hidden size 64
fp 3 layers of MLP with 256, 256, and (number of edge types) units and ELU activation
Bi-LSTM of q (z x, c): hidden size 64
fg :3 layers of MLP with 256, 256, and (number edge types) units and ELU activation
ft : layers of MLP with 256 and 128 units and ELU activation
8 layers of MI. with 256 and 128 units and ELU activation
f layers MLP with 256 and 256 units and ELU activation
LSTM of po (x|z, c): hidden size 64
In the case encode the context information: A CNN with 3 layers of is used with kernel size followed by flattening and MLP layers with 256 and 128 units.
We use batch normalization after the layers of our neural networks.
A.2 TRAINING DETAILS:
Data split:
For the NGSIM 1-80 the training and test data are split according to the preprocessing of Henaff al. (2019). The training data divided 80% training and 20% validation.
For the Basketball dataset the data is divided tof 65% training, 10% validation, and 25% test set.
For the nuScence dataset the training set is divided 80% training and 20% validation. The provided validation set used for test.
Hyperparameters (chosen by validation):
Parameter of the Gumbel distribution 7 0.5
Percentage of samples from the pe (z)x, x, c) model: a 10%
batch size: 128
Number of epochs:
50 for the NGSIM 1-80 dataset
500 for the Basketball dataset
50 for the nuScences dataset
Type of GPU: single TITAN X GPU.
B APPENDIX 2: MORE RESULTS
B.1 ERROR BOUNDS THE RESULTS:
Tables below show the results of the experiment with standard deviation. The results are based on running the models for times. We split the set of all and other agents into two tables.
13

---

Published as a conference paper at ICLR 2022
minADE/innDD (all)
Model
FM MPUR
0.14110.:220.8--111 0.024
0.4110015/1. 06 000029
0.5410.181.3
0.034
0.641001991.5 0.050
NRT no action
0.19:1:.:1220095--0000
0.50:10.0150 8/1. 07
0.68:10.0221..26 08
0.75:000282000 0.054
NRT with action
1.96::0.0120.88=00000
0,44+00018/1 0.000199
011+4000241167700000
0.70+:0.2001 93==0.045
dNRI no action
0.33:t0.0113085500.000
0.39:0.013/1.00 03:00.223
0.49::0.01611 24400.0399
017-00002//1633-000
UNRI action
0227:10.0090066-00000
0.34::0.02000.00
0.43-10.011P1.. 180000000111.
0,51::0.02/11
0.000
NRI-NSI no action
0.28:00.10006110000
155=000150090000000
00==0000111.03000000
0,49 52
NRI-NSI with action
0.20::0.1100.51:0::00
0.26::
0.34:0.01100.5000.0000
0.4::00011///0020000003
Table 4: Performance of models action-conditional task for different prediction horizons the NGSIM 1-80 datasct.
minAADE/minFDE (other)
Model
FM-MPUR
0.381:0:1 0.012
0.4510.02MLL 0900.030
0.1111001119166 0.006
0.70:10:028/1 005.:0022888880044444040 0540..04
NRI with action
0.38 00.01.0001120.91 0.030
0.4911..018/11 19 0.035
0.67 0.055
0.75 0.000
aNRI 0.12:40.1100722-00002
0.38.1..01
0.09
0.00
0.57::0.024/1 0.040.
NRI-NSI NSI with action
0.23:10.0550058 0.003
2.910001550.69 0.018
0.37 0.015/0.. 0.025
1.5-10..111/1170000000
Table 5: Performance of models on action-conditional task for different prediction horizons on the NGSIM 1-80 dataset.
minADDmmnnFDD (all)
Model
1s
4s
NRI dNRI
0.08000.0080,
03000
114400.00000000000
0.30:E0:01
00200.2
0.09=0000050.11 02+0.007
112==000110017+0000000
0.20=0001110311
0.27=0.01:00000--010 00006
Trajectron++
0.10:10.0060.13 1.000
0.12=
0.000
119=10..1180113-100000
0.28110.1155.44000011
PRECOG
0.0610.00660.05
0.09-0..0000.151..00000
0.6:1001200151100011
0.25
NRI-NS
0.06.10.0070000
1.0005
0.0=100.09/01111.0000
0.1::0..10009::::0:14
0.18100100.3100000
Table
Performance of models on goal-conditional task for different prediction horizons on Baskethall
dataset.
minADD/minFDD (nther)
Model
26
4s
NRI dNRI
0.12 0.000
1.16:00009002400001
0.35:00.0250.66
08-.000.255..88111111 68-0.036
0.10:10.006M.11 14..0000...44.0111
0.25::0.1100.15:00000
0.34:10.0200.55
Trajectron++
0.10:10.007AL. 15 0.009
0.23 19.00000
0.34 1 10.600
PRECOG
0.06:00.0:40.09220000.
0.10
18:0:00009
0.18::0.01100 291
0.29::0.01
58-0.015
NRL NSI
0.06 1.005.1.09 0.004
0.08 0.00770.122.0.006
0.13 0.2.0000000211000
0.2010.014N.33 0.013
Table Performance of models on goal-conditional task for different prediction horizons on the Basketball dataset.
minA.DEminHFE (all)
Model
2x
NRI ANRI
0.27
0.38-000000:000000000
0.35 0220
02
98-00052
0:23::0:1220.22000000
0.31:0.011..770002222
2.003
178::0001116620000
Trajectron++
122410.01880.491100
0.35_00.150.7410000
0.48 0.48 1.002
0.7611.003811 0.051
PRECOG
0211+0.0100:47-00001
0.3--0.0110.8:000000
04-2000198885+000000
305+-0.0291.55-00000
NRI-NST
0.18:000160077:0022
0271000330:5100006
0:8100166788::0077
Table 8: Performance of models on goal-conditional task for different prediction horizons on the nuScenes
datasct.
minADE/minFDD (other)
Model
NRI dNRI
0.31:00.011N.0000000111
0.42-0.0100XXX-00003
0.3::0.015/1 40000.11
0.98:20.0112 12=0.012
0.27:0.01150054.00000
0.22:0.1770:4110022
22.10.019
Trajectrone++
0227400.166052200001
0.39-00100.774.0.00
0.54:0.0190 0..
0.83:10.026/1 97.
PRECOG
0.24 1.0.000.911.
0.37:00180.7420.01
000-0000100994+00000
022+0.00441.73-00000
NRI-NS
0.15:10.1000388-00000
0.29-0016006::1000000
0:40:00.1122000.0..00
0.641:00111133-000000
Table 9: Performance of models on goal-conditional task for different prediction horizons on the nuScenes
dataset.
14

---

Published as a conference paper at ICLR 2022
B.2 RESULTS ON RELATIONAL INFERENCE
The results in the tables 1-3 show that NRI-NSI performs better than other baselines in terms of trajectory prediction performance. This might implicitly indicate better relational inference by NRI-NSI. Nevertheless, here we try to investigate the performance of NRI-NSI compared to NRI and dNRI terms of relational inference explicitly using toy dataset.
Here we use the charged particles dataset. We use 5 and 10 particles in a 2D box with positive and negative charges {=q} that are sampled uniformly. The force among them follow the Coulomb's law. We follow similar procedure by Kipf et al. (2018) to stabilize the generation process, i.e. soft clipping the force using the softplus function. Similarly, we generate 50k training examples, and 10k validation and test examples. Although the generated data might not exactly follow the physics rules, we will have explicil relations (force) among particles and can use this as the ground truth for our experiments. Our goal is to accurately infer the relation among the particles given that know the final position of of the particles for S-particle experiment and final position of 2 particles for 10-pa partitiicl experiment. For all models information is fed to the last layer their encoders. Table below shows the accuracy of different models in predicting the relations among the particles:
use 2 edge types. For both NRI and dNRI the goal
Toy dataset
NRI
dNRT
NRINSI
particles
82.3 0.5
83.1 + 0.6
88.5 + 0.4
10 particles
70.6 +
70.9 +
76.8 +
Table 10: Performance terms of relational inference accuracy (in %) for the charged particle dataset.
Note that for this results we only used the encoder of different models.
15


Reviews:
[{"review_id": "yhX4Kdzel-x", "reviewer": "Reviewer_hKEW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Weaknesses\n* The task assumption of the paper does not look straightforward to me. I am doubtful about the practical value of the task setting: \"the individualized information cannot be accessed by other entities but its effect will be observed in the future\". Is this a well-recognized setting? Are there practical scenarios of this assumption?\n* Given the existing work of NRI and the variational inference framework, the paper is not clear about the contributions. For instance, what are the new challenges that need to be solved after introducing the private entities? Besides the model architectures, what are the key innovations? \n\n## Strengths\n* The experimental results look strong and thorough. Different datasets are considered and the improvements are significant. ", "summary_of_the_paper": "**Summary**: This paper introduces a neural relational inference model that makes use of the hidden features of each node in a variational inference framework. Specifically, the hidden/individual information is modeled as private node in the graph. Importantly, the task assumption made by the authors is that these individualized features cannot be observed by other entities. \n\n**Contributions**: \n1. The authors claim be to the first to study the use of individualized information for each entity in this direction.\n2. The proposed approach achieve state-of-the-art results while only introducing minimum additional computational complexity. ", "summary_of_the_review": "Overall, this paper has demonstrated strong results on neural relational inference tasks. The concerns I have are about the assumption of the approach and unclear novelty points. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tPI8qOst7Fs", "reviewer": "Reviewer_NM6N", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a neural relational inference model with node-specific information. Experiments on real-world datasets validate the merit of the proposed method. \n\nStengths:\n1. This paper is well written and the idea is clear. \n2. The results are promising compared to various baslines. \n\nWeaknesses:\n1. The codes are not availabel, it is difficut for others to re-produce the results. ", "summary_of_the_paper": "This paper presents a neural relational inference model with node-specific information. Experiments on real-world datasets validate the merit of the proposed method.", "summary_of_the_review": "This paper presents a neural relational inference model with node-specific information. Experiments on real-world datasets validate the merit of the proposed method. \n\nStengths:\n1. This paper is well written and the idea is clear. \n2. The results are promising compared to various baslines. \n\nWeaknesses:\n1. The codes are not availabel, it is difficut for others to re-produce the results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QNo0EW2Eadx", "reviewer": "Reviewer_SFbP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The concept of node-specific information (NSI) seems to be novel and is interesting.\n- The paper is well-written. All components are explained in detail.\n- The paper demonstrate the effectiveness of the idea. The proposed approach performs better than reference systems that use the private information only at the decoder. Hence, I agree with the conclusion that the approach helps to form better interaction graphs.\n- The distinction to prior works, especially NRI, is clear. It is easy for the reader to understand what the contributions of the paper are.\n\nWeaknesses:\n- The paper argues that in many real-world examples, \"there is a set of hidden features that affect the way entities interact with each other.\" However, I am not sure how reasonable this is. For instance, in autonomous driving, it seems reasonable that considering the intention of individual entities can help to generate an interaction graph. However, it is not the hidden features that affect the interaction of the entities, but only the public nodes, which are affected by the hidden features.\n- The paper uses autonomous driving as an example. However, in autonomous driving, the intention of a car can be shared with other cars via car2car communication. However, sharing this information is limited to system that are capable of this form of communication. Pedestrians, for instance, also have private intentions that cannot be shared easily.\n- I am missing an analysis of the learned graphs, since the paper states that their method is better able to learn interaction graphs. Now I am wondering if this difference can be visualized in an example from one of datasets, or even better analyzed in a systematic way.\n\n\nQuestions:\n- Question 1: One perhaps crucial part is not yet entirely clear to me: Since the public nodes are connected, and each public node is connected to its private node, isn't it possible that the private information flows to the other nodes in only 2 hops, especially in the decoder? Then, the private information can be shared with other public nodes. The paper states that \"... however its [the private node's] effect might be observed in the future times steps ...\". But as far as I understand, it is not only the effect of the private information but the private information itself that can be shared. It would be great if the authors or other reviewers could clarify this point.\n- Question 2: Furthermore, I am wondering if the approach can actually be used in real-world situations, for instance for autonomous driving. In a real-world setting, each car, and therefore each model, has its own set of hidden information that is not shared with other cars, which means that each model only has access to the private information of a single car. However, the model assumes to have  access to all the private nodes (i.e. it has a global view of the problem, not only a local view), which seems to not fit to the envisioned application setup.", "summary_of_the_paper": "The paper introduces the concept of node-specific information (NSI) to model that nodes in a graph may have private information that other nodes cannot have access to. The paper uses Neural Relation Inference (NRI), a framework published in 2018 based on variational inference, to uncover the hidden relations of nodes in the graph. For instance, in a driving scenario, different cars can be nodes in a graph with their publicly visible trajectory and their private information about the intention (e.g. desired destination), which is not shared with other nodes. The encoder and decoder in NRI are modified such that NSI stays private and is not shared with other nodes. The paper considers problems that require uncovering the interactions of entities in a multi-agent dynamical system. The evaluation is based on the accuracy of future trajectory prediction. The paper demonstrates the effectiveness of the idea on three different datasets, one action-conditional dataset and two goal-conditional datasets.", "summary_of_the_review": "To summarize, the idea of introducing node-specific, private information into a graph of multiple interactive systems seems to be novel and interesting and improves the prediction performance. However, it is not entirely clear to me if the private information can be propagated to other nodes (question 1) and if the model can even be applied in the envisioned scenario (question 2). I think the answers to both questions are important to make a reasonable assessment of the paper. Hence, my recommendation is rather tentative and I will update it as soon as the application setup becomes more clear to me.\n\nAfter rebuttal:\nThanks a lot for the author's replies. I still think the applicability of the idea may be problematic, but I think the idea of NSI is sufficently interesting and novel to be accepted to this conference. Hence, I raise my score to accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends the Neural Relational Inference framework for probabilistic inference of interaction relations between entities, to a scenario where entities may have private features, which requires modifications of the standard graph encoders and decoders in NRI.

Reviewers appreciated both the model and the overall execution of the paper: the building blocks are clear, the evaluation does its job well. The main doubts are about the applicability of the setting, for which the authors don't provide too many examples. However, the construction is somewhat intuitive, and even in cases where private attributes aren't explicit, it may be valuable to disentangle the shareable attributes this way. We encourage the reviewers to discuss the applicability a bit further.

Typos: (not exhaustive, please doublecheck with a spell checker)
 - multiple occurrences of Gumble instead of Gumbel
 - bottom of pg 4, factorzied -> factorized
Decision: Accept (Poster)