Learning Heterogeneous Interaction Strengths by Trajectory Prediction with Graph Neural Network
Authors: Seungwoong Ha
Hawoong Jeong
Venue: ICLR 2023 poster (2023)
Keywords: relational learning
complex systems
dynamic systems
graph learning

Abstract: Dynamical systems with interacting agents are universal in nature, commonly modeled by a graph of relationships between their constituents. Recently, various works have been presented to tackle the problem of inferring those relationships from the system trajectories via deep neural networks, but most of the studies assume binary or discrete types of interactions for simplicity. In the real world, the interaction kernels often involve continuous interaction strengths, which cannot be accurately approximated by discrete relations. In this work, we propose the relational attentive inference network (RAIN) to infer continuously weighted interaction graphs without any ground-truth interaction strengths. Our model employs a novel pairwise attention (PA) mechanism to refine the trajectory representations and a graph transformer to extract heterogeneous interaction weights for each pair of agents. We show that our RAIN model with the PA mechanism accurately infers continuous interaction strengths for simulated physical systems in an unsupervised manner. Further, RAIN with PA successfully predicts trajectories from motion capture data with an interpretable interaction graph, demonstrating the virtue of modeling unknown dynamics with continuous weights.

Published as a conference paper at ICLR 2023
LEARNING HETEROGENEOUS INTERACTION STRENGTHS BY TRAJECTORY PREDICTION WITH
GRAPH NEURAL NETWORK
Seungwoong Ha Department of Physics
Hawoong Jeong Department of Physics
KAIST
KAIST
Daejeon 34141. South Korea
Daejeon 34141, South Korea {eongg @
(skyround2002) @maa com
ABSTRACT
Dynamical systems with interacting agents are universal in nature, commonly modeled by graph of relationships between their constituents. Recently, various works have been presented to tackle the problem of inferring those relationships from the system trajectories via deep neural networks, but most of the studies assume binary or discrete types of interactions for simplicity. In the real world, the interaction kernels often involve continuous interaction strengths, which can- not be accurately approximated by discrete relations. In this work, propose the relational attentive inference network (RAIN) to infer continuously weighted interaction graphs without any ground-truth interaction strengths. Our model em- ploys novel pairwise attention (PA) mechanism to refine the trajectory represen- tations and a graph transformer extract heterogeneous interaction weights for each pair of agents, We show that our RAIN model with the PA mechanism ac- curately infers continuous interaction strengths for simulated physical systems unsupervised manner. Further, RAIN with PA successfully predicts trajectories from motion capture data with an interpretable interaction graph, demonstrating the virtue of modeling unknown dynamics with continuous weights.
INTRODUCTION
Dynamical systems with interactions provide a fundamental model for myriad of academic fields, yet finding out the form and strength of interactions remains an open problem due to its inherent degeneracy and complexity. Although it is crucial to identify the interaction graph of complex system for understanding its dynamics, disentangling individual interactions from trajectory data without any ground-truth labels is notoriously hard inverse problem. Further, if the interactions are heterogeneous and coupled with continuous strength constants, the interaction graph is called weighted and the inference became much harder with increased level of degeneracies.
In this work, we assume the dynamical system with N objects (or agents), and their (discretized) tra- jectories X1, XN, from timestep = 0 T are given. If the system has interaction kernel Q(x,, X5) and the dynamics are governed by form of x; Ejzs ki,Q(X;, X;) with some variable ki,, which is prevalent in nature and physical system, we call kaj as an interaction strength between the object and j. With proper normalization, we can always regard S kij S 1. In general, kys may have continuous values and forms a weighted interaction graph, which can be expressed in the form of connectivity matrix K; a conventional adjacency matrix with continuous-vall kg. Hence, the problem is inferring continuous adjacency matrix K from trajectories x alone.
entries of
In the current work, we propose a neural network called Relational Attentive Inference Network (RAIN) to address the problem of inferring weighted interaction graphs from multivariate trajec- tory data in an unsupervised manner. RAIN infers the interaction strength between two agents from previous trajectories by learning the attentive weight while simultaneously learning the unknown dynamics of the system and thus is able to precisely predict the future trajectories. Our model em- ploys the attention mechanism twice: once for the construction of pairwise trajectory embedding

---

Published as a conference paper at ICLR 2023
(unknoww) Adjacency mark
iuubated trajectories
RAIN
Graph Attentive Module
Intoraction natwork
Pairwise Attonntoo
E Encoder
Attentive weight
Compar son
Minimize loss
(Aa((e. ()..)))))))
Spring system
Aggregated
LSTM
fax 7(h))
Decoder
Attentive weight matrix
Predittdd enjectories
Figure Overview system formulation and RAIN architecture. RAIN encodes each agent's trajectory with an LSTM encoder and applies pairwise attention (PA) to the hidden states for con- structing pair of embeddings for each agent pair. Then the graph attentive module extracts the interaction strength from a pair of embeddings in the form of attention weight with MLP, Ag- The decoder module finally predicts the future trajectories of each agent with an LSTM decoder, but here, each prediction only employ the weighted information from other agents. This restriction on information induces the attention weights in the learning process to properly reflect the strengths of the connections.
and once for the actual graph weight extraction. Differing from previous approaches such as the graph attention network (GAT) Veličković et al. (2017). RAIN aims infer the absolute interaction strength that governs the system dynamics by employing attention module with multilayer percep- tron (MLP) and sigmoid activation. By comparing the inferred interaction strengths of simulated physical systems with ground-trute values that are not provided at the training stage, we verify that RAIN capable of inferring both system dynamics and weighted interaction graphs solely from multivariate data. We further show that RAIN outperforms discrete baselines capture data, representing a system in which we cannot be certain whether a continuous form of interaction strengths even exists. In this way, demonstrate that the rich flexibility and express- ibility of the continuous modeling of interaction strengths are crucial for the accurate prediction of the future dynamics of an unknown empirical system.
real-world motion
2 RELATED STUDIES
There has been a long history and substantial amount of work on both inferring the network topology and the nonlinear correlation between interacting constituents from data Casadiego et al. (201
Ching & Tam (017)) ; Chang et al. (2018); Shi et al. (2020): ; (Ha & Jeong (021 Fujii et al. 2021
along with the development of various measures capture the relation between constituents (e.g.. Pearson correlations, mutual information, transfer entropy, Granger causality, and variants thereof
Schreiber 2200)). Many of these inferences focus on specific systems with the necessity for a model prior, such as domain knowledge of the agent characteristics, proper basis construction, and detailed assumptions on the system dynamics.
Recently, by phenomenal advances in machine learning, adopting a neural network as a key compo- nent of the interaction inference has gained attention from researchers Veličković al. (2017): Kipf et al. (2018); Zhang et al.) (2019). The key strength of these approaches comes from the fact that a neural network enables relatively free-form modeling of the system. One influential work in this di- rection, neural relational inference (NRI)Kiph) et al. (208)), explicitly infers edges by predicting the future trajectories of the given system. But previous studies for extracting interaction graphs with neural networks Kipf et al. 2018); Webb et al. (2019); Graber & Schwing (2020) mainly focused on inferring edges with discrete edge types, which means that they are incapable of distinguishing the edges of different interaction strengths with the same type. Considering the common occurrence of such heterogeneous interaction strengths throughout diverse systems, the assumption of discrete edge types severely limits the expressibility of the model.
2

---

Published as a conference paper at ICLR 2023
Ball
Encoded LSTM hidden states
aaair
Key
Query
Weight
Value
Weak interaction
X
0.0
X
Painwise Attention (PA) hidden states
0000000000
X
+
Strong interaction
t=48
X
Σ,
a Spai V (hht)
[49
X
-0.6
Ball
500
0.1
0.2
0.3
0.4
0.5
Inner product
Figure 2: Overview of the PA mechanism, which facilitates effective interaction inference by em- phasizing the critical part of the time series. Here, two balls weakly interacted first but showed strong interaction at the end. The circled points at timestep 2 and 49 are emphasized as exemplary timestep where the interaction was weak and strong, respectively. PA achieves refined trajectory representations by comparing encoded LSTM hidden states at the same time. Due to the asymmet- ric nature of the transformer architecture, the weighted hidden state for A to B and B A can be drastically different, which lets PA properly handle the directional asymmetry of interaction.
One recent study Li et al. 2022) tackles this problem of inferring the connectivity matrix by first training the neural network and then additionally applying 'graph translator' to extract continuous graph properties, which needs a ground-truth label to train. Several studies from physicists Zhang et al. (2015); Lai (2017) employed perturbation analysis and response dynamics infer continuous interaction strengths. However, these correlation-bas
methods usually require more than 10a to 107 data points for the inference, which is feasible size for entire dataset but not for a single instance, and thus the direct application to experimental data is difficult.
Also, many of the recent studies in relational inference Li al. (2020) 2021 Sun al. (2022) and trajectory prediction Xu et al. (2022b); Mo et al. (222)) Xu et al. (2022a) explicitly model the relationship between objects continuous value. But, for interaction inference, all these models rely on the traditional Graph Attention (GAT) Velicković et al. (017)) or slightly modified version of it, which becomes limitation for challenging inference problem. Further, most of these works are not focusing on correctly infer the true interaction strength, hence only tested on the system with unknown ground-truth interaction strength or didn't report any comparison with it. In this work, we show that even the advanced and strictly more expressive version of GAT, GATv2 Brody et al. 2022 is insufficient to fully capture the true interaction strength.
3 MODEL DESCRIPTION
Our RAIN model as shown in Fig. T comprises three parts trained jointly: an encoder that com- presses time series data, a graph extraction module that infers the interaction weight between every pair of agents, and decoder that predicts the future trajectories of each agent. Note that RAIN does not require a ground-truth interaction graph for training; instead, it produces an interaction graph as byproduct of future trajectory prediction. In the following, formalize our model and describe each component in detail.
3.1 ENCODER AND PAIRWISE ATTENTION
The long short-term memory (LSTM) encoder is composed of a single layer of gated recurrent unit (GRU) Cho et al. (0144) with hidden state dimension dism = 128. The encoder repeatedly takes the trajectory data of all agents from each timestep by receiving Tame steps of trajectories each consisting of state variables N agents at time T, and producing corre- sponding Tene hidden states, h' h2 ...., hT T.,. The last hidden state h? Tax could preserve the infor- mation from the entire trajectory of all agents in theory. But we found that a naive final hidden state is insufficient to fully capture the interaction strength in large-scale systems. In this study,
.x......x"" Tm..
3

---

Published as a conference paper at ICLR 2023
NRI(2)
RAIN
RAIN PA
Groundtruth
408
bal
bal12
-050
-0.0
-050
-050
+0.2
Figure 3: Visualization of the trajectory predictions (upper) and retrieved connectivity matrices (lower) for the spring-ball system. The results from left right are from NRI(2), RAIN without PA, RAIN with PA, and ground-truth. Here, 5 out of 10 balls are drawn for trajectory visualization, and semi-transparent paths indicate the first 50 steps of input trajectories while solid paths denote 50 steps of predicted future trajectories.
propose pairwise attention (PA) mechanism effectively infer the interaction strength between two trajectories. The intuition behind PA is straightforward: a single hidden state cannot be a suit- able choice for extracting the interaction strengths from every possible pair. Thus, calculate the attention between kame-time hidden states to assign weights to their contribution. as depicted in Fig,
2 Here, the term 'same-time' PA indicates that the attention values are calculated between h and hy (K i and Q technically) only for same time t, not between every possible = 0 T, as conventional full attention mechanism does. We use a slightly modified transformer Vaswani et al. 4 heads and head dimension of dn dism//m=2 to emphasize strong interaction. See Vaswani et al. (2017) for detailed description of the transformer architecture. Formally, the LSTT hidden states are processed into Key(K) (K), Query(Q), and Value(V) matrices as follows,
(217) with
h = fLSTM(( x()
X pair, = Spuir, (hl)
where X E (K,
(2)
xt X
x-1 pairs
+ X pair,
① ... + X
(3)
pair,
where the symbol @ indicates the concatenation of the matrices the last dimension, fiSTM the GRU layer, x are the state variables of the ith agent at time and Spair,X is stack of MLP layers for Kpair. pair and Vpair- The superscripts t and m on X indicate the time and head number, respectively. The attention- -wiehtt hidden state is expressed as follows,
f.m aj
softmax Khlm (Q (patt)))) ) // /dn)
(4)
T
6
m
Vim t.m pa
(5)
h h
+ hi
t.m. where af is an attentive weight between agents and j. and h is weighted hidden state for the math attention head. RAIN passes in to the graph extraction module.
By employing the PA mechanism, time interaction, which is unique to each agent pair. We highlight that comparing the same-time time hidden states only, rather than considering the full attention in conventional settings, significantly
refine the LSTM hidden states by focusing the specific
4

---

The outline of the graph extraction module of RAIN is similar to the GAT Veličković et al. (2017) The difference between ours and the original GAT is that RAIN has no prior knowledge of the underlying graph and thus the O'graph attention value should infer the presence of the edge itself as well its strength, while GAT aims to find the relative importance between fixed graph edges. Also, the conventional inner-product attention of GAT replaced with a 3-layer MLP Ag in RAIN to achieve much more flexible representations. We use attention- weightedd hidden states from the PA mechanism apply the transformer as O'graph. a (Aa (h, @ h.)). Note that instead of softmax which normalizes the attention weight, sigmoid activation ar is used for graph attention to obtain the Published as a conference paper at ICLR 2023
NR
RAIN
RAIN + PA
Groundtruth
-64
-52
AMMAAMA
666
-01
-02
62
Figure 4: Visualization of the trajectory predictions (upper) and retrieved connectivity matrices (lower) for coupled Kuramoto oscillators. The results from left to right are from NRI(2), RAIN without PA, RAIN with PA, and ground-truth. Here, 5 out of 10 oscillators are drawn for trajectory visualization, and semi-transpann tranppraaspr paths indicate the first 50 steps of input trajectories while solid paths denote 50 steps of predicted future trajectories.
dei dt
reduces the time complexity of the inference while achieving the goal of the extraction of temporal information. Also, since the transformer can handle asymmetric relationships by differentiating key and query, PA can capture directional connections with ease.
3.2 GRAPH EXTRACTION MODULE
absolute interaction strength. This is because our main goal is infer the ground-truth interaction strength, not the relative strength for single instance. Extracted graph attention Orgrahh. becomes a weight for the decoder module.
3.3 DECODER
The decoder shares the same GRU layer with the encoder module and the employ new value func- tion, Viec.i, for the message aggregation. For agent i, RAIN concatenates attention-weiig weightedd value vectors from other agents along with its own value vector as follows,
h Toutt fLSIM (hTactt-1
x Toutt)
Tantt dec,i
= Spair, v (h? Tactt)
(8)
h
Tantt dec.s
Σ
agraph V Taett dec
+ V Tantit dec,
where fLSTM and Fpuir, consists of MLP. RAIN finally produces a prediction of the state variables of
the next step using another stack of MI. layers saecs fps and fa by first
h Tactt
des ʃaee (I bee,
and
5

---

Published as a conference paper at ICLR 2023
then 14; -/==+ fz ( (( and o: for ()) Thee outputs of f. and fo are r-dimensional vectors, each representing the means and variances of the difference of state variables. RAIN sam- ples the next state from Gaussian distribution with fr and fo and adds the values to the previous state. The decoder iterates this for Tace steps to predict the future states. For training, we employ negative log-likel (NLL) loss for Gaussian distribution.
4 EXPERIMENTS
We demonstrate the capability of RAIN by performing inference tasks with various model systems ranging from simulated physical systems real motion capture data from a walking human. All of the models and baselines are implemented with PyTorch and optimized with Adam Kingma See the Supplementary
& Ba (2014). Data and models are available at ggihub..co//okpiilRe. Materials for details on the data generation, model architectures, and training details.
4.1 PHYSICAL SIMULATIONS
We simulated the trajectories of two physical systems: spring-ball system and phase-coupled Ku- ramoto oscillators Kuramoto (1975). Unlike previous studies Kipf et al. (2018); Webb et al. (2019) in which every interaction strength in the system discrete and constant (commonly w 1), we consider a more general setting where interaction strengths are drawn from a continuum and are thus heterogeneous. Assuming that two agents i and j are interacting, the interaction strength for each system would be spring constant kij for spring-ball system and coupling weight wi for a Kuramoto model. Although our model can handle asymmetric interaction wrennghh-hhanie take the symmetric form of connectivity matrix
to the asymmetric nature of the transformer- kiy = kyx for the simulated systems. For the spring-ball and Kuramoto systems, we first select the edges between nodes with probability p (excluding e construct an interaction graph, and then assign a randomly sampled interaction strength to each edge from a uniform distri- bution U|0,, 1] while expressing non-assigned edges with zero interaction strength. We generate 10k training samples and 2k validation samples for all simulated tasks. The state variables consisting of trajectories are 2, y, Us: By (positions and velocities) for the spring-ball system. For the Kuramoto oscillators, a concatenated vector of de sin øo and intrinsic frequency w are used to construct the trajectories, where é is the phase of oscillator.
For evaluation, we measure how accurately the model predicts future trajectories by mean squared error (MSE) and how well the model retrieves the original connectivity matrix by two forms of Pearson correlation. We first gather every retrieved interaction strength a and corresponding true interaction strength & from all 2k validation samples and calculate the correlation in total (pux)- Secondly, we independently calculate the correlation for each validation sample and take the average of the 2k samples (psample). One may interpret Prot as the overall performance of the model, while Psample indicates the expected correlation for single instance at a test time. We excluded diagonal trivial zeroes (due to no self-loop interaction) while calculating correlation, so the reported value is strictly lower compared to the case where every value in the connectivity matrix is used. Correlations from the baseline models with discrete edge types are obtained by assigning continuous weights to each edge type according to every possible permutation and choosing the best one. More precisely, we assign n weights of O, n-1 n to n edge types with n! different assignments and report the highest correlation.
4.1.1 RESULTS
Table shows the correlations between the true interaction graph and the predicted interaction strengths from various models. Following Kipf et al.) (2018), we measure the correlation between trajectory feature vectors as Corr. (Path) and the correlation between trained LSTM feature vectors as Corr. (LSTM). NRI(2) and NRI(4) indicate the NRI from Kipf et al. (2018) with 2 and 4 edges types, respectively. As baseline with continuous attention, we employ GATv2 Brody et al. (2021), which is an advanced version of GAT and analytically proven to be strictly more expressive than the original GAT. We check the effect of PA by testing the RAIN model both with and without the PA mechanism. In Table T we can clearly see that RAIN with PA significantly outperforms ev- ery other model at inferring interaction strengths. Here, GATv2 showed worse performance in the

---

Published as a conference paper at ICLR 2023
Table 1: Total Pearson correlation (Pra) and the sample Pearson correlation (Psample) between trieved interaction weights and true interaction weights for simulations with 5 and 10 interacting objects.
Model
Spring
Kuramoto
Spring
Kuramoto
Corr.
Poot
Prample
Pro
Prample
Piot
Psample
Psumple
5 objects
10 objects
Corr. (Path)
0.2917
0.2944
0.0243
-0.1206
0.2654
0.2732
0.1047
0.0682
Corr. (LSTM)
0.0979
0.0952
-0.1369
-0.0751
0.1196
0.1096
0.0024
-0.0012
NRI(2)
0.8482
0.8660
0.0027
-0.0048
0.8602
0.8593
0.0393
0.0413
NRI(4)
0.9250
0,9214
0.0230
0.0192
0.8921
0.8966
0.0438
0.0454
GATv2
0.8425
0.8643
0.6296
0.6466
0.7981
0.8060
0.5410
0.5481
RAIN
0.8411
0.8770
0.5213
0.4987
0.7787
0.8164
0.4560
0.4683
RAIN + PA
0.9400
0.9568
0.8731
0,8925
0.9117
0.9221
0.8265
0.8411
Table 2: Mcan squared error (MSE) in predicting future states for simulations with 5 and 10 inter- acting objects. Underlined entries show better results than those from RAIN with PA. Model
Spring
Kuramoto
Prediction steps
10
50
10
30
5 objects
Static
0.6665
1.3614
1.2340
1.0671
0.9901
1.0108
SingleLSTM
0.1969
0.5643
0.6048
0.5006
0.4957
0.5247
JointLSTM
0.0704
0.3913
0.6362
0.0391
0.1083
0.1749
NRI(2)
0,0171
0.2232
0.5263
0.0268
0.1615
0.3275
NRI(4)
0,0122
0.1395
0.3573
0.0308
0.1498
0.3050
GATv2
0.0230
0.2081
0.4252
0.0455
0.1901
0.3140
RAIN
0.0259
0.1322
0.3113
0.0388
0.1705
0.3675
RAIN PA
0.0084
0.0714
0.2193
0.0041
0.0645
0.2059
RAIN (true graph)
0.0062
0.0181
0.0732
0.0037
0.0068
0.0122
10 objects
Static
0.6070
1.1253
1.0779
1.0380
0.9935
0.9799
SingleLSTM
0.2028
0.4991
0.5189
0.5511
0.5138
0.5173
JointLSTM
0.1317
0.4823
0.5840
0.0953
0.2490
0.3832
NRI(2)
0,0078
0.1158
0.3169
0.0392
0.2341
0.4451
NRI(4)
0,0061
0.0866
0.2440
0.0372
0,2411
0,4385
GATv2
0.1309
0.2634
0.4192
0.0354
0.1665
0.3766
RAIN
0.1665
0.3109
0.4996
0.0307
0.1902
0.3942
RAIN PA
0.0069
0.0892
0.2351
0.0115
0.1586
0.4016
RAIN (truc graph)
0.0059
0.0163
0.0504
0.0009
0.0063
0.0301
spring-ball system and better performance in Kuramoto oscillators compared RAIN, but signifi- cantly worse than RAIN with PA model in both cases. Also, note that the high correlations from the NRI models for the spring-ball system can only achieved if and only we know the ground-truth interaction strength and choose the best permutation. Considering the huge performance difference between RAIN without and with PA, our model and mechanism are both crucial for accurate interaction strength retrieval. We show MSE results for predicting 50 future states in Table Here, again following Kipf et al.] (2018), SingleLSTM runs a single LSTM for each object separately, while JointL.TM concatenates all state vectors (thus may handle fixed number of agents only) and trains a single LSTM to jointly predict all future states. See the Supplementary Materials for further analysis, including full histograms of the correlation distributions.
Figures 3 and A show the results of future prediction and interaction strength inference for 10 inter- acting agents, where 5 trajectories drawn for visualization. In Fig. we can observe that all

---

Published as a conference paper at ICLR 2023
(A)
Groundtruth
(B)
RAIN + PA
(C)
NRI(2)
(D)
0-
0.85
08
08
0.8
6
04
G-
04
0.4 0,70
0.65
BANN PA
Undundracc
0.5
0.0
0.0
0.50
KRIZZ
Figure 5: (A) Ground-truth connectivity matrix of the given multilayer network with single inter- layer link. Inferred interaction weights from (B) RAIN with PA and (C) NRI(2) are shown. (D) Trajectory predictions (10 steps) for ball on the zy-plane. The inset depicts diagram of the given multilayer networks, where the inter-layer link is highlighted with the red dotted line.
models, including NRI(2), succeed in capturing the existence of interaction between agents. But is apparent that the NRI(2) model fails to capture the interactions with small weights, while RAIN without PA yields in numerous false positives with spurious weights. Such weakness each model is reflected in relatively large prediction errors compared to RAIN with PA, especially for ball (green) and ball 5 (purple). The power of the PA mechanism becomes more evident with the Ku- ramoto oscillators (Fig. 4). where only RAIN with PA succeeds in retrieving meaningful interaction weights. To sum up, we can conclude that the PA mechanism clearly helps the refinement of hid- den states and thus yields better results with greater correlation with the ground-truth interaction weights.
4.1.2 IMPACT OF WEAK INTERACTION
By inferring interaction with continuous strength as RAIN does, we capture entire spectrum of interactions with a single model. Particularly, we find that our model is able to detect weak interactions that are often ignored by the discrete NRI models due to their limited capacity. In Figure 5 we emphasize the significance of the inference of weak interaction by constructing the connectivity matrix in form of multilayer network Kivelä al. (2014) with 2 layers. Here, we prepare two densely connected layers of springs where their spring constants are sampled from [0.5, 1]. uniformly. Between the two layers, we set single inter-layer link (connecting ball 3 and ball 8) with a spring constant of 0.3, which is much smaller than the intra-layer interaction weights. Since the synchronization of multilayer network largely depends on the strength and structure of inter-layer coupling Leyva et al. (2017) Della Rossa et (2020), capturing this weak but significant connection between the two layers is critical for trajectory prediction. As Fig. 5 shows, RAIN with PA accurately captures this weak interaction, while NRI(2) misses it and thus its predicted trajectory considerably deviates from the true one.
4.2 MOTION CAPTURE DATA
Next, we test our model inferring interactions between the joints of a walking human data from the CMU motion database cmu (0003). Different from the physical simulations, this real-world system does not possess any well-known dynamics with a continuous interaction strength. Following Kipf al. (2018) and Graber & Schwing (2020), we use the data from subject #35 with 31 joints. We split the traming and validation data sets into 4 1 ratio. We use the same protocol as with the physical systems: provide 50 time steps of trajectories and let the model predict 50 unseen time steps afterward. For the motion data, we use raw value of the future state instead of the difference from the current state to calculate the Gaussian loss fast convergence.
Figure A shows the errors from each model's prediction, where RAIN with PA achieved the lowest MSE. An example visualization of the prediction is shown in Fig. GB. Without dynamically re- evaluating the interaction graph at every time step reported in Kipf et al. (2018) and Graber & Schwing 20200), our model precisely predicts the future states from static but continuous interaction weights. Also, shown in Fig. CC, model produces smooth and block-wise connectivity matrix which is much easier to interpret compared to that from the discrete NRI(2). This is because there is no restriction natural meaning for each edge type in the discrete model, and thus the
8

---

Published as a conference paper at ICLR 2023
(A)
(B)
(C)
1.0
SingleLSTM JointLSTM
10
0.0
NRI(2)
NRI(4)
rghthand
110
MSE
RAIN
RAIN PA
10-3
0.50
0.25
500 hadd
Prediction steps
Figure 6: (A) Validation MSE comparison for motion capture data. The prediction results of 50 time steps from SingleLSTM, JointLSTM, NRI(2), NRI(4), RAIN without PA, and RAIN with PA shown. (B) Sample predictions from NRI(2) (blue) and RAIN with PA (red) for validation tra- jectory of motion capture data with the ground-truth truth states (black). (C) Inferred interaction weights from NRI(2) (top) and RAIN with PA (bottom) are shown. Here, the diagonal entries from NRI(2) are marked with -1 to distinguish them from the two edge types, 0 and 1.
manual sparsity prior needed to handle the 'no interaction' edges if an abundance is expected. On the other hand, the attention value of RAIN is expected to convey the strength of interaction directly. For instance,
clear from RAIN's connectivity matrix that the dynamics of the feet are less correlated with the rest of the joints and also that the spine moves along with the right leg, arm, and hand, both of which agree with the qualitative movements of human while walking.
5 CONCLUSION
In this work, we introduced RAIN, model for inferring continuous weighted interaction graphs from trajectory data an unsupervised manner. With the PA mechanism that computes the attention between same-time LSTT hidden states between agents, we can sharply refine the representation for the interaction weight extraction. Our model successfully inferred the absolute interaction weights from simulated physical systems and further showed great prediction performance with an empirical system, demonstrating the advantage of continuous weight modeling in relational learning. Notably, RAIN needs only 50 time steps of data infer the interaction weights, and thus requires less data by several orders of magnitude than correlation- based theoretical methods Zhang et al. (2015 Lai (017).
Also, we found that the refinement of the LSTM hidden state with PA is critical for meaningful per- formance. Since the PA mechanism is generally applicable neural model where its trajectory is represented by a recurrent neural network, one may expect an increase in performance by employing PA in other relational models without increasing the inference time significantly. If the system size N is extremely large and ( time complexity of the attention mechanism becomes a bottleneck, one may adopt recently proposed attention mechanisms with linear time complexity and virtually no performance drop Wang et al. (2020); Shen et al. (2021) to calculate PA.
In the real world, interactions between agents in complex systems possess broad range of charac- teristics. For instance, they can be either positive (excitatory or encouraging) or negative (inhibitory or suppressing), time-delayed with heterogeneous time scales, and noisy both inherently and exter- nally. Complex systems in nature generally contain every aspect of these characteristics, such as neural signals human brain. Reinforcing the current RAIN architecture to handle data of such complex nature with single model would be promising future direction to explore.
DATA AVAILABILITY
All of the codes for data generations and models available at github.com/noopi//RRNNN

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGEMENT
This research was supported by the Basic Science Research Program through the National Research Foundation of Korea NRF-2022RI 12002RRIA2B58020001303 (HJ) and National Research Foundation of Ko- rea Grant funded by the Korean Govermment NRF-2018S A3A2075175 (SH).
REFERENCES
Cmu, carnegie-mellon motion capture database. http://mocca.cc.cmmu 2003.
Shaked Brody, Uni Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arX1v:2105..4491, 2021.
Jose Casadiego, Mor Nitzan, Sarah Hallerberg, and Marc Timme. Model-free inference of direct network interactions from nonlinear collective dynamics. Nature Communications, 8(1):1-10, 2017.
Yu Chang, Fan- Sun, Yueh-Hua Wu, and Shou-De Lin. A memooy-networkk based solution for multivariate time-series forecasting. arXiv preprint (rXiv:/809002/03 2018.
Emily SC Ching and HC Tam. Reconstructing links in directed networks from noisy dynamics. Physical Review E, 95(1):01030,, 2017.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using mnn encoder-decoder for statistical machine translation. arXiv preprint 1rXiv:1406.1078 2014.
Fabio Della Rossa, Louis Pecora, Karen Blaha, Afroza Shirin, Isaac Klickstein, and Francesco Sor- rentino. Symmetries and cluster synchronization in multilayer networks. Nature Communications, 11(1):1-17,
2020. 2020.
Keisuke Fujii, Naoya Takeishi, Kazushi Tsutsui, Emyo Fujioka, Nozomi Nishiumi, Ryoya Tanaka, Mika Fukushiro, Kaoru Ide, Hiroyoshi Kohno, Ken Yoda, et al. Learning interaction rules from multi-animal trajectories via augmented behavioral models. Advances in Neural Information Processing Systems. 34:1 108-1 122, 2021.
Colin Graber and Alexander Schwing. Dynamic neural relational inference for forecasting trajecto- ries. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 1018-1019, 2020.
Seungwoong Ha and Hawoong Jeong. Unraveling hidden interactions in complex systems with deep learning. Scientific Reports, 11(1):1-13, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.698,, 2014.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. arXiv preprint arXiv: 1802.0080004487 2018.
Mikko Kivelä, Alex Arenas, Mare Barthelemy, James P Gleeson, Yamir Moreno, and Mason A Porter. Multilayer networks. Journal of complex networks, 233):203-271, 2014.
Yoshiki Kuramoto. Self-entrainment population of coupled non-linear oscillators. Interna- lional symposium mathematical problems in theoretical physics, pp. 420-422, Springer, 1975.
Pik-Yin Yin Lai. Reconstructing network topology and coupling strengths in directed networks of discrete-time dynamics. Physical Review E, 95(2):022311, 2017.
Leyva, R synchronization in non-identical multi-layer networks. Scientific Reports, (1):1-9,22 2017.
NadaiiaaNadaal R Gutiérrez, JM Buldú, and S Boccaletti. Inter-layer
Jiachen Li, Fan Yang, Masayoshi Tomizuka. and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning, Advances in neural information processing systems, 33:19783-19794, 2020.
10

---

Published as a conference paper at ICLR 2023
Longyuan Li, Jian Yao, Li Wenliang, Tong He, Tianjun Xiao, Junchi Yan, David Wipf, and Zheng Zhang. Grin: Generative relation and intention network for multi-agent trajectory prediction. Advances in Neural Information Processing Systems, 34:27107-27118 2021.
Zhe Li, Andreas S Tolias, and Xaq Pitkow. Learning dynamics and structure of complex systems using graph neural networks. arXiv preprini arXiv:2202. 10996, 2022.
Xiaoyu Mo, Zhiyu Huang, Yang Xing, and Chen Lv. Multi-agent trajectory prediction with hetero- geneous edge-enhanced graph attention network. IEEE Transactions on Intelligent Transportation Systems, 2022.
Thomas Schreiber. Measuring information transfer. Physical Review Letters, 45(
Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities, In Proceedings of the IEEE/CVF winter conference on appli- cations of computer vision, pp. 3531-3539, 2021.
Rundong Shi, Weinuo Jiang, and Shihong Wang. Detecting network structures from measurable data produced by dynamics with hidden variables. Chuos: An Interdixciplinary Journal of Nonlinear Science, 30(1) 1)03133,,,, 2020.
Fan-Yun Sun, Isaac Kauvar, Ruohan Zhang, Jiachen Li, Mykel Kochenderfer, Jiajun Wu, and Nick Haber. Interaction modeling with multiplex attention. arXiv preprint arXiv:2208. 10660, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information 2017.
processing systems, pp.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua 2017.
Bengio. Graph attention networks. arXiv preprint
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 047666 2020.
Ezra Webb, Ben Day, Helena Andres-Tent Terre, and Pietro Lió. Factorised neural relational inference for multi-interaction systems. arXiv preprint arXiv:1905.08722 2019.
Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hyper- graph neural networks for trajectory prediction with relational reasoning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6498-6507, 2022a.
Yi Xu, Lichen Wang, Yizhou Wang, and Yun Fu. Adaptive trajectory prediction via transferable gnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6520-6531, 2022b.
Zhang Zhang, Yi Zhao, Jing Liu, Shuo Wang, Ruyi Tao, Ruyue Xin, and Jiang Zhang. A general deep learning framework for network reconstruction and dynamics learning. Applied Network Science, 4(1):1-17, 2019.
Zhaoyang Zhang, Zhigang Zheng, Haijing Niu, Yuanyuan Mi, Si Wu, and Gang Hu. Solving the inverse problem noise-driven dynamic networks. Physical Review E, 91(1)1112814, 2015.


Reviews:
[{"review_id": "_pXTcpM96_g", "reviewer": "Reviewer_jYRz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a model called RAIN for interaction and trajectory prediction. Given the history trajectories of multiple moving agents, the RAIN model predicts their interaction weights and future trajectories.\n\nThe RAIN model works as follows. 1) It first encodes the history of each agent with an LSTM encoder. 2) It then uses pairwise attention to enrich the per-agent feature vectors. 3) It then uses a graph attention network to further infer the interaction weights. 4) Finally, it uses an LSTM decoder to predict their future trajectories.\n\nThe authors evaluated the RAIN model on two physical systems (a spring-ball system and a phase-coupled Kuramoto oscillator system) and a human walking dataset. They used the NRI method as their baseline. The evaluation result shows that the RAIN model significantly increases the accuracy of correlation prediction and reduces the trajectory MSE.", "summary_of_the_review": "This paper presents a sound method for predicting agent interaction and future trajectory. The proposed method achieves better performance than the baseline method on public benchmarks. So I am giving it a weak accept.", "strength_and_weaknesses": "--- Strengths\n\n- This paper presents a sound method for predicting agent interaction and future trajectory. The proposed method achieves better performance than the baseline method on public benchmarks.\n\n- The result shows that adding pair-wise attention helps.\n\n\n--- Weaknesses\n\n- There is not sufficient comparison against the state-of-the-art baselines to demonstrate the advantage of the proposed method. In the evaluation section, NRI is the only public baseline that the authors compared their method against, and it's quite an old baseline. Also, there is not much discussion about the existing works in the paper.\n\n\n--- Other comments\n\n- I think there are some typos in Eq (5). It should be V_j^{t,m} instead of V_i^{t,m}. Also, I think there should be a sum over j. Otherwise, why does the subscript j disappear in the left-hand side of the equation?\n\n- I suggest the authors adjust the positions of the result tables and figures. For example, Table 1 and Table 2 are referenced in the same paragraph in the text, but one is on Page 6 and one is on Page 9.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M4eFAKfASIv", "reviewer": "Reviewer_cYy1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors propose a relational attentive inference network (RAIN), in which the interactions between agents are continuous. A pairwise attention (PA) is adopted to refine the representations and a transform structure is also employed to extract the interaction weights. The proposed method achieves good performance on interaction strength inference and promising results on motion prediction.", "summary_of_the_review": "I list my concerns in [Weaknesses], I am happy to discuss and increase the rating if my concerns are addressed.", "strength_and_weaknesses": "**Strengths**:  \n1. Well-organized and clearly written.\n2. Strong motivation with promising results.  \n\n\n**Weaknesses**:  \n1. In Section 3.1, when evaluating on the simulated physical systems, the procedure of obtaining the ground-truth of connectivity/interaction matrix is not clear, since the method in the unsupervised manner. Why use a single inter layer to get the \"ground truth\" matrix? More details and explanations should be included.\n\n\n2. In Section 3.2, when it comes to the motion prediction problem, especifically using CMU motion dataset, existing methods usually predict the difference from the previous time step, but authors predict the raw value here. What's the reasons/insights here?  Also, in Figure 6(c), are there any ground-truth interaction weights? or how to tell which one is better when comparing the interaction weight matrix? \n\n\n3. In Section 2.1, for the model complexity, for the pair-wise attention weights calculation, at each time step, it would be $O(N^{2})$, assuming there are $N$ objects. Also, when employing the transformer structure, the calculation cost would be very large. More explanations about reducing  time complexity should be included, some experiments would be better to illustrate.\n\n\n4. Experiments are not sufficient, only NRI is compared (2018), more existing SOTA baselines should be included.\n\n5. Missing related works, it is not a mandatory task, but I strong encourage authors to include a related work section to introduce some recent works on, i.e., interaction inference/learning[1,2], trajectory/motion prediction [3, 4], etc, and to highlight the differences between the proposed method and existing works.  \n\n\n6. Minor mistakes:  \n(1) In Figure 2, is it \"+\" or \"$\\bigoplus$\"?  \n(2) In Figure 5 (D), x/y axis label is missing.  \n(3) In References, the names of conferences or journals, i.e., format and first letter, are not consistent.\nI highly encourage authors to revise the submission to avoid such small mistakes, as well as some typos.\n\n[1] GRIN: Generative Relation and Intention Network for Multi-agent Trajectory Prediction. NeurIPS2021  \n[2] Evolvegraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning. NeurIPS2020  \n[3] Adaptive Trajectory Prediction via Transferable GNN. CVPR2022  \n[4] GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction With Relational Reasoning. CVPR2022  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FFbWlxXLdS", "reviewer": "Reviewer_zkVt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a RAIN to infer interactions among agents. It learns the attentive weight and unknown system dynamics, which is also helpful to predict future trajectories. The author(s) verified that RAIN can infer the system dynamic parameters and interaction graphs, and can outperform baseline discrete models.  \n", "summary_of_the_review": "The main contribution of the paper is the proposal of the RAIN + PA which achieves a surprisingly good result in Table 2. Probably the contribution of adding PA to the RAIN is more important (empirically) than the main model (RAIN).\n\nThe introduction of PA is not unheard of, but is new in the interaction/prediction setting. This is the main reason it is marginally above the acceptance threshold.\n\nThe reason it is not getting higher ratings is because the performance of the main model RAIN without PA is somewhat disappointing. Also, the reviewer would like to see the method tested on some more non-trivial datasets to verify its validity.\n\n", "strength_and_weaknesses": "Strength:\n\nThe triplet of (trajectory encoder + interaction graph + trajectory decoder) seem to work well, with the important added tricks such as modified RNN's pairwise attention (Eq 4-5), prior-less graph with transformer-like attention (Eq 7-8). Those methodologies seems sufficiently different from previous works and are novel.\n\nTable 2 indicate a significant performance gain by using RAIN. \n\nWeakness:\n\nThe paper draws similarity between the neural network and the physical dynamic system. However, the reviewer can't find any physical constraints are directly applied in the neural networks, nor at the neural network losses. The reviewer would like to see the actual dynamic system encoded in the NN architectures to generate more guaranteed realistic trajectories.\n\nThe experiment results include a simulated data (Table 2) and motion capture data of human walking (Table 3) seems toy-ish (although some previous works also uses those data). The reviewer would like to see how this method perform in a more complex scenario such as in https://waymo.com/open/", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DTco731D1Q_", "reviewer": "Reviewer_5vb5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces RAIN, a method to learn relational graphs between different nodes (\"agents\") of a system from time-series data describing the trajectories followed by said nodes. \n\nRAIN is composed of a) an encoder computing an embedding of the trajectories seen so far, b) a graph extractor that infers the interaction strength between nodes. The authors also combine RAIN with a \"pairwise attention\" (PA) mechanism, which greatly improves the reported performance of RAIN.\n\nThe authors describe their method and then evaluate it on spring-ball models, Kuramoto oscillators and motion capture data of a person walking.", "summary_of_the_review": "I believe the paper addresses a challenging and relevant problem, and proposes a novel architecture to do so. It also presents a range of strong empirical results, albeit one that raises a few questions about the reported baseline. Also, the paper suffers from a global lack of clarity and a missing background section, which is its main weakness in my opinion.\n\nGiven that, I do not believe the paper is ready for publication, but I am willing to reassess if the issues mentioned are resolved.", "strength_and_weaknesses": "**Strengths**\n\nThe main strength of the paper appears to be its empirical evaluation. Evaluating on three different very different kind of systems (plus a multi-level variant of the ball-spring system) provides a significant insight into the advantages of RAIN over NRI, which can only assign discrete interaction strengths to graph edges. \n\nThe results section is further strengthened by a varied analysis combining trajectory predictions, connectivity matrices and correlation coefficients. I found that this multi-modal approach helped my appreciation of the results despite being unfamiliar with previous works.\n\nFinally, the problem of learning continuous interaction graphs from data seems challenging and highly relevant to both future research and applications.\n\n**Weaknesses**\n\nThe first issue with the paper is the lack of a Background section clearly describing the problem and introducing the key variables and general approaches. Without a background section, the paper is hard to follow since it is unclear what assumptions can be made about the system or what constitutes success. \n\nSecondly, I found the Section 2 lacked clarity, partially due to the issue above but also due to writing and inconsistent notation. \nFor instance, it is not clearly stated before the method whether one trajectory describes the evolution of the entire system, or whether it is one trajectory per node. Based on the section 3, I believe it to be the latter, but sentences like the following at first let me believe it was the former, when trying to understand the method:\n\"The encoder receives $T_{enc}$ steps of trajectories $x_1, x_2, ... , x_{T_{enc}}$ , each consisting of $R$ state variables of $N$ agents, and produces corresponding $T_{enc}$ hidden states, $h_1, h_2, ... , h_{T_{enc}}$.\"\n\nRelatedly, the authors state \"we calculate the attention between _same-time hidden states_ to assign weights to their contribution\". The term \"same-time hidden states\" is never clearly defined, despite being key to PA. Additionally, Figure 2 clearly features a sum over different time-steps to compute the Pairwise Attention, which seems contradictory.\n\nAnother point of confusion is that variables used in figures 1 and 2 ($h'_{ij}, A_\\theta, w_t$) are absent from the main body of the paper, including equations. Given the two previous points, I found Figures 1 and 2 to be of little help to my understanding, overall.\n\nI also found it confusing that nodes (e.g. balls in the ball-spring system) be called \"agents\", given that they take no action. That said, feel free to ignore this comment if this is standard terminology in the field.\n\nFinally, on the experimental side, results reported lacked confidence intervals or other measures of uncertainty. I also found it slightly odd that the only baseline reported dates from 2018. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster