Interactive Sequential Generative Models
Authors: Dennis Fassmeyer
Pascal Fassmeyer
Ulf Brefeld
Venue: Submitted to ICLR 2023 (2023)
Keywords: Generative Models and Autoencoders
Graph Neural Networks
Recurrent Networks
Sequential Models
Multi-Agent

Abstract: Understanding spatiotemporal relationships among several agents is of considerable relevance for many domains. Team sports represent a particularly interesting real-world proving ground since modeling interacting athletes requires capturing highly dynamic and complex agent-agent dependencies in addition to temporal components. However, existing generative methods in this field either entangle all latent factors into a single variable and are thus constrained in practical applicability, or they focus on uncovering interaction structures, which restricts their generative ability. To address this gap, we propose a framework for multiagent trajectories that augments sequential generative models with latent social structures. First, we derive a novel objective via approximate inference using a disentangled latent space that accurately describes the data generating process in such systems. Based on the proposed training criterion, we then present a model architecture that unifies insights from neural interaction inference and graph-structured variational recurrent neural networks for generating collective movements while allocating latent information. We validate our model on data from professional soccer and basketball. Our framework not only improves upon existing state-of-the-art approaches in forecasting trajectories, but also infers semantically meaningful representations that can be used in downstream tasks.

Under review as a conference paper at ICLR 2023
INTERACTIVE SEQUENTIAL GENERATIVE MODELS
Anonymous authors Paper under Nouble-blind review
ABSTRACT
Understanding spatiotemporal relationships among several agents is of consider- able relevance for many domains. Team sports represent a particularly interesting real-world proving ground since modeling interacting athletes requires capturing highly dynamic and complex agent-agent dependencies in addition to temporal components. However, existing generative methods in this field either entangle all latent factors into single variable and are thus constrained in practical appli- cability, or they focus on uncovering interaction structures, which restricts their generative ability. To address this gap, we propose framework for multiagent trajectories that augments graph-structure sequential generative models with ex- plicit latent social dependencies. First, we derive novel objective within the variational autoencoder family using a disentangled latent space that aims to en- capsulate inherent data traits. Based the proposed training criterion, then present a model architecture that unifies insights from neural interaction inference and graph-structured variational recurrent neural networks for generating collec- tive movements while allocating latent information. We validate our model on data from professional soccer and basketball. Our framework not only improves upon existing state-of-the-ar approaches in forecasting trajectories, but also infers semantically meaningful representations that can be used downstream tasks.
INTRODUCTION
The study of agent behavior governed by temporal and spatial dependencies is of great importance in many different fields, such as autonomous driving (Brown et al. 20200 Rasouli & Tsotsos 2019), robot navigation (Rudenko et al., 2020), or sports analytics (Tuyls et al. 2021). In particular, aceu- rate detection of implicit causal social structures offers several advantages by removing confounding
factors for trajectory forecasting tasks and providing practitioners with interpretable dynamics that can in turn be integrated into downstream decision-making processes applications.
Modeling the dynamics of multiplayer sports games (Omidshafiei et al. 2022, Le et al. 2017;
Yue et al. 2014] Liu et al.) (0200 is particularly challenging since accurate trajectory generation in this environment requires capturing highly dynamic and complex underlying modular structures (Makansi et al. 2022). For example, the roles prescribed in team formation are poor indicator of the actual behavior observed in a given situation. Moreover, most of the interacting elements in- ject noise into the forecasting process because they are either irrelevant (c.g., goal keepers) or their influential nature changes as the situation evolves. However, existing methods for modeling sports data rely on graph encoding strategies (Kipf' & Welling, 2016 Vaswani et al., 2017) that aggregate social information into only single variables that need to capture all latent stochasticity (Zhan et al., 2019 Yeh al. 2019, Sun et al.) 2019: Omidshafiei et al. 2022).
In recent years, a considerable amount of methods have been proposed that aim to infer interactive components general multiagent systems via discrete latent variables. These methods are usu- ally formulated as some form of variational autoencoder (Kingma & Welling) 2013 Sohn et al. 2015) that learns latent edge categories of an assumed underlying graph structure Kipf et al. 2018 Graber & Schwing 2020 Löwe et al.) 2022). However, being the only causal factors specified, the proposed frameworks neglect other potential latent characteristics not originating in mere interactive categories but equally affecting multimodal agent behavior, which limits their generative capacity.
To address previous shortcomings, we propose a novel framework for modeling multiagent trajec- tory data that enhances existing graph-structured latent variable models by explicitly encoding social

---

Under review as a conference paper at ICLR 2023
structures in sports games. Since the contemplated spatiotemporal systems caused by dynamic dependencies among heterogeneous agents, we define this component as a causal graph compris- ing categorical agent roles and pairwise interactions. Based the specified generative setting, then introduce an objective function within the variational autoencoder family and instantiate a con- crete architecture for computing the derived training components. Empirically, our model exceeds existing state-of-the-art methods in forecasting trajectories on data from professional soccer and basketball. In addition, we report on extensive quantitative and qualitative analyses wrt. the learned latent variables that show informative properties in generative tasks and downstream applications.
2 BACKGROUND
Given data D = (x <r fi-1 consisting of N sequences æ<T = mate the underying data distribution via maximizing the likelihood of the collected evidence, i.c. max max-po(x-T)) In practice, pe (IxTT) is often highly multimodal, which complicates direct deploy- ment of MLE. A frequently used modeling paradigm for stochasticity in complex multimodal dis- tributions introducing latent variables and optimizing the variational lower bound on the maginal Rezende et al. 2014; Sohn et al.) 2015).
N
our goal is to esti-
log-like ogllikelihood (Kingma & Welling, 2013
Existing conditional variational models for generating highly-structured sequential data øgr usually associate a latent variable ....... 27 with each timestep of the segment to describe the generative process (Bayer Osendorfer 2014 Goyal al. 2017: Fraccaro al., 20T6). The variational RNN (VRNN, (Chung al.) 2015)) is renowned instantiation in this domain that, assuming specific dependency structures in the generative and inference parts, arrives at the following lower bound on log po (x)T):
Eg (Sggrllee (ggg)
Σ log pa (ar( (2g, x2) - KL =eI) |l Po Po( =ze1]]
(1)
where information Pet, Zet is captured via recurrent neural network h, = f'NNN(N, 2t, h--1). Given the temporal and multimodal notion of human movement, sequential generative models con- stitute good starting point for designing a framework tailored to multiagent trajectories. However, such approaches only account for the temporal aspect of the problem, but neglect potential social dependencies at each timestep.
Adding Graph Structure As a remedy, sequential data can be augmented by a social dimension (a)
EST { Va E A}. where x2(") Rd denotes d-dimensional feature representation of agent A at time (c.g., its 2D position). Permutation invariant models are prerequisite for processing sequential sets with potentially divergent cardinality, so direct adoption of Eq. [ in multiagent settings would implicitly impose the assumption of social independence across agent trajectories. This assumption is trivially inappropriate for interactive systems; thus, related work proposes sensitive solutions usually in the form of graph encoding strategies capture agent- agent interactions.
Yeh et al. (2019) introduce the graph VRNN (GVRNN), which operates within the VRNN framework with graph neural networks (GNNs, Battaglia et al. (2018) representing agents and their interactions as nodes and edges, respectively. More formally, the architecture for computing the components in Eq. [ ammuuts to the following structure:
Pe (z x Rez) = NZx: GNN (ht-1))
(2)
4s (2 Fgg, NZzx;; GNN enee [Iz,, h1-1l))
(3)
Po (a, Eect Zgt) = -N'(x;; GNN dee ([z,, 22,1h h1-1)))
(4)
where h4 is the set of recurrent agent states h, = fRNN( (0) (0(00 (a) (11)). We emphasize that,
although factorized, the latent space is not marginally independent across agents since each 2(()) is conditioned on information of all other entities via the (assumed) fully-connected graph. However, in many spatiotemporal patterns, most of the observed elements are irrelevant or even distracting, and the specific composition of relevant factors can change rapidly. Thus, a better strategy is to explicitly detect semantic classes that describe the underlying structural component before aggregating social information into entangled variables.
2

---

Under review as a conference paper at ICLR 2023
(n.)) iin
( Exdt
CVRNN Module
Interaction Module
LEGEND
XLP
Connal modge Strpping
Pummettriaaton
Figure 1: Left: Illustration the output layer an interaction module (encoder Jo or prior poy for target agent E A and an neighboring agent E A, where e and e are node and edge embeddings, respectively. Right: Illustration of an interaction network (decoder pé,, prior P02, encoder (o,) that is parameterized by realized elements from the discrete latent subspace: E picks from multiple node-to-edge MI.ss and KC())) picks from multiple edge-to-node MI.s run parallel. Thus, implicit knowledge about fundamentally differ- ent interaction patterns is encapsulated the parameters the edge MLPs associated with the corresponding
class. The encoded neighborhood communication ( contingent the predicted interaction type, has dif- ferent behavioral consequences based the specifie role the target agent is currently adopting.
METHOD
In the following, we propose novel line of thought on how to model the joint distribution of interactive sequential data po(=gr) using variational methods. To address shortcomings of existing methods, our formalization defines disentangled factors variation that explicitly include a discrete latent subspace describing causal factors that arise from social perspectives. Having conceptualized our modeling goal, we instantiate a concrete architecture termed DIA (Detecting Important Agents) that computes the inferred training components via imposing an inductive bias on the latent space. Appendix B.2 details how the model is trained in an end-to-end fashion.
3.1 DERIVING THE TRAINING OBJECTIVE
Latent Variables The interactive systems contemplated in this work are caused by implicit and dynamic structural dependencies that vary both in the type of interaction well as in the specific role of the interacting participants. For example, in a turnover sequence in soccer, both the role performed (e.g., attacker to defender) and the influential nature (e-g., on the ball agent from ball carrying to pressure exerting) of each player change within a few moments. We propose to explicitly account for such structures via introducing latent causal graphs G. (K,, Et}, where K., represents
the set of discrete agent roles at t (e.g., ball, attacker or defender) {x] a E A and &, encodes their (discrete) pairwise strategies the form of interaction types (E(*.)) (a. j) E Ax (e.g., ball handler, intended pass receiver, etc.). To enable the model to exclude potential social confounders, we externally define an interaction type & as class "no interaction" while the remaining categories are learned from the data.
However, the structural state captured in G, is not sufficiently informative to precisely account for factors that arise from both individual and social perspectives. Therefore, to improve the genera- tive capacity of our model, we inheret latent concepts z, that account for all remaining sources of uncertainty not represented by the inferred causal graphs, yet may vary function of G.-
Generative Process We formally express the generative process by incorporating the above de- scription of causal factors into the definition our modeling goal, i.e., the marginal likelihood po(=gr):
pe( = /
Jer zgr
T
= Σ
II
(2. Zer, Sst) (G. (G Zes, dZgT
Ger Egrtml
For only single agent type, we have Kr A.
3

---

Under review as a conference paper at ICLR 2023
TRNN
Mo (anbl
Ber)
7 RMx
LEGEND
(#i Gizl
Sanpllag
Dipoakreiis eff -
Interaction Graph Learning
Adapted GVRNN
Figure 2: Illustration of the computational dependencics timestep for system with two interacting agents when using one agent type, i.c., Kr A. We only show the first layer of the overall two-layer graph network the interaction part left side) to clarify the injection of the GRU states. Red connections denote the parts utilized only during training, orange connections used during inference. We use a diamond shape to indicate the value is deterministic, and use a circle to indicate the variable is stochastic.
where the model likelihood pe, is parameterized using a decoder, and pe, and pe, are the prior distributions over the specified latent factors.
Given the generative model, learning representations of data can be seen as learning variational approximation of the posterior using an encoder that constructs the distribution of latent values for a given observation Pgr. We propose the following factorization:
Inference Model
(s<zer 95T(Ts))
II 9s, D GcL)
Note that, as opposed to the VRNN objective, do impose restrictions this approximation when defining its dependency structure by including also future inputs 1+41-7 in the conditions of go, and 4d.- This has been shown be empirically beneficial in sequential settings (Goyal al. 2017
Fraccaro et al. 2016).
Objective Function We construct an objective function LDIA o, 0) for training the intro- duced concepts po and ga- by defining bower-bound on log- likelihood log po (x
G k 2o 2 Zer, Per (2 WT Zcr, Get)
po, (xx G
(G. Let. Zet GeL)
E (fgrrnrrrrrgg log
1=1
(5)
with Qe (9.lm Zzt; Ger)
Gon (Kg Z<c. Pee (Ell Zct, Bct)
and
pe, Eel Z21, Get) = Pe, Po, (Kel Zct, Gea (E, Zc,, Get) See Appendix A for
derivation of Eq. 5 and elaboration of the relationship Eq.
3.2 ARCHITECTURAL COMPONENTS
Given past realizations Bett Ret, Get, the introduced dependency structure in Eq. infer the structural component G. before subsequently generating variables and St. Hence, we divide the training procedure at each timestep into two distinct phases, with the first phase addressing the calculation of components components over 2, and 1, (po,, Po,, and qa.). ). The overall architecture is illustrated in Figure 2 We describe the computations in more detail below.
5 requires to first
Gi (pe, and 4s. ). while the subsequent second phase infers
4

---

Under review as a conference paper at ICLR 2023
Interaction Graph Learning First, we uncover the (dynamic) latent graph space by constructing
the encoder (and prior) modules estimating Gr, Vt E (I)....)) Since we define latent concepts
G. (K2, } in discrete terms (cf. Section 3.1, it is natural to model the encoder (and prior) as multivariate distribution of independent Categoricals:
g 2 Rcc,
Ret, Rct, = II
Cat ( (aen acr, 9etiir))))))
(6)
aEA
9 (E) 2 921)
Cat( (u(E,(t)))) fint (agr, Zcc (ec))))))))))))) (L.DEAA
II
where function fene an encoder neural net. Since the interaction space contains the defined "no interaction" label, the derived probability values over &, in a Bernoulli context can be interpreted as importance values that encode the level of influence interacting neighbors on target agents.
While elements in Kr are anchored to the elements of the agent set, interactive patterns & however, are defined non-smmmetrically over all agent pairs. Thus, we impose a function class on Sene that enables learning single-agent as well pairwise representations based which we classify over both node and relation variables. Inspired by Kipl et al. (2018), we employ an adapted variant of GNNs similar interaction networks (Gilmer et al., 2017) that operates by learning alternating node and edge representations through an iterative process where the learned edge embeddings are used messages for neighborhood aggregation See Appendix B.1 formalization of the core computations of this GNN variant.
We model the input to (ane(ar-T,, Zet; G<t)) via multiple GRU cells (Chung et al. 2014 defined over quantities related to the individual agents. More formally, fene is given by
foint eint] = fént (TST, Z Get) GNNimt ene (h t-1, h.), 1i
(8)
where [] denotes concatenation, hi is set of forward RNN states (RNN( (a) h (1) (-1))
h is set of hidden states defined along the edges (N (g(a5) _ ((.j))mmt) of fully-connected graph structure, and Pazes eitt are the node and edge embeddings of the last/output GNN layer, respectively (ef. Figure left). The
h ; is a set of backward RNN states {RNN( (a) (h()))
concatenated agent sets h h h: refer to the initial node representations of the network and the hidden interaction set h; concatenated with the edge embeddings of the first GNN layer (cf. Figure 2 left). The usage of multiple message passing rounds (2 2) in conjunction with fully- connected adjacency matrix allows the model to accomodate structural information from the full input when estimating agent and interaction types.
Finally, we model the distributions comprising Eq. 6 and 7 as Go (G, (TTT Zet, Get) softmax softmax(Linear( fint (T.TT, Zet, Get, (1)r;)))); see Figure Po, (Gr) Feet Pet,, Get) is derived analogously to the interaction encoder, omitting the summa- rized future h as GNN input.
The prior distribution
Adapted GVRNN Once G. Io. is generated, the model needs to reason the remaining components of Eq. (5) To achieve this, we observe that the functionality of G, is grounded in uncovering latent social dependencies at every timestep t within multiagent segment. Thus, the generated G. implicitly characterizes the computational graph at specific junctures and, since pre- dictions carried out autoregressively, encompasses all information required for computing agent states H, and zg. Motivated by the previous considerations, we assume G, is Markov and thus dis- card Get from the conditions of objects pe, Pe, and q remaining computations of Eq. (5) thereby comply with the structure of G.-conditional VRNN components (cf. Eq. 4
In general, we can use any graph encoding strategy performing forecasting on the inferred latent subspace G. to model the distributions over I; and 2,. An instantiation with interaction networks constitutes a reasonable design choice in the present context, as there is already empirical evidence
'Other commonly used graph encoding strategies in the multiagent regime such as graph attention networks Veličković et al. 2017)) or transformers (Vaswani et al. 2017) update node embeddings via learned attention weights and consequently do not explicitly learn vectorial edge representations, rendering them inappropriate for the problem at-hand.
5

---

Under review as a conference paper at ICLR 2023
of their effectiveness for modeling sports tracking data (Yeh et al. 2019 Dick et al. 2022). For ease of exposition, we assume an interactive system consisting of two agents and define computations for graph i f- j. where i A and j E A as follows.
To accomodate Gr, we parameterize the node and edge functions inherent in each graph network dependent on the predicted classes e(i.)) E {I, K} and K.( (1,...,M). More specifically, we pick corresponding parameters for the edge MLP fc from the set (ei..... b with 1 = denoting the manually defined "no interaction" label and parameters for the node MLP fo from the set {?,,.. }M The message passing operations for target agent amount to
( Sc((1o 00 thx)
u fo(e(t.);
(10)
Intuitively, this inductive bias encourages the model to learn semantically meaningful concepts de- scribing distinct movement and interaction patterns and governs attention towards important pieces of data when predicting collective movements. See Figure [ detailed visual depiction of the logic. An overvíew over the computational dependencies is given Figure 5b)
4 EXPERIMENTS
In this section, we empirically validate our proposed model on two challenging real-world datasets: basketball and soccer datz") The dataset details, model architectures, and hyperparameters of both our model and the baseline approaches are discussed in Appendix C.T
4.1 SETUP
Metries We adopt the commonly used Average Displacement Error (ADE) and Final Displace- ment Error (FDE) metrics to assess the generative performance of future predictions. ADE refers to the 1a error between the predicted locations and the ground truth averaged over the entire trajectory, while FDE is the l2 error for the last predicted point (Alahi et al. 2014). Following prior works, we report the minimum over 20 generated samples.
Baselines We compare method with variety of SOTA baselines for modeling (i) sports track- ing data: Weak-Sup (Zhan et al. (211)) GVRNN et 2019). dNR/ (Graber & Schwing 2020). DAG-Net (Monti et al. (021). GRIN (Li al. 202Ib); and (ii) for modeling urban dynamics: Joint- 3-cVAE (Bhattacharyya et al. 2021) and GRIN (Li et al. 2021b). To enable direct comparison against this diverse set methods, we benchmark in two experimental configurations.
Firstly, DAG-Net and Weak-Sup are inherently advantageous in generating agent trajectories by in- corporating future locations in form of heuristically generated labels prediction time. This renders direct comparison to fully unsupervised generative methods (like ours) biased. To minimize this bias, we report quantitative results on rather long prediction horizon of Tobs 10, Tpred 40 for both methods. Other baselines such GRIN and dNRI cannot be trivially tested on these long- term predictions, so we generate results on configuration Tok. = 40, Tpeed 10. Finally, GVRNN, Joint-B-cVAE and our framework (DIA) can be applied to the full range of prediction scenarios.
4.2 QUANTITATIVE EVALUATION
Baseline Comparisons In the first set of experiments, we benchmark our model against recent generative methods for the task of trajectory forecasting, Our empirical findings are summarized in Table I and Table (left.. DIA emerges as the best generative tool across all tested tasks improving the best (unsupervised) competitor performance by at least 17.3% minADE and 25.5% minFDE for basketball. Remarkably, despite their inherent advantages, DIA also outperforms recent supervised generative methods Weak-Sup and DAG-Net) by least 8.3% minADE and 16.8% minFDE, Fur- thermore, we observe that the methods that learn global latent variables (GRIN and Joint-3-cVAF) perform significantly worse than the remaining models. Though it may be premature to arrive at a definite conclusion, however the comparatively dismal performance of such baselines in our results 'The code will be made publicly available upon acceptance of this manuscript.

---

INTERACTION- WELNCTIONRNN--EEE RNN-FULL Under review as a conference paper at ICLR 2023
Joint-B-cVAE
GVRNN
dNRI
GRIN
DIA (Ours)
min ADE
4.03
2.60
2.77
3.00
2.20
min FDE
6.56
5.66
5.52
6.12
4.51
Joint-B-cVAE
GVRNN
Weak-Sup
DAG-Ner
DIA (Ours)
min ADE 10.64
9.73
9.47
8.98
8.29
min FDE
14.47
15.80
16.98
14.08
12.05
Table I: Quantitative results basketball data (in meters) modeling offensive players for short-term predic-
tions with Tpred 10 and Tob. 40 (on top) and long-term predictions with T Firee 40 Toke 10 (below).
MODEL
MINADE
MINFDE
MODEL
ACCURACY
F1
JOINT-3-CVA
8.10
10.91
GVRNN
7.48
10.72
INTERATTION- RNN- DIAG
0.82
0.85
DNRI
7.60
10.88
0.83
0.84
GRIN
7.88
10.54
INTERACTION-RNN
0.88
0.91
DIA (OURS)
7.02
9.68
Table Results on soccer data. Left: Quantitative results for trajectory prediction (in meters) modeling all players with Tobs = 10 and Treed = 10. Right: Results for the auxiliary classification task.
tend to suggest limited transferability of such models for sports data. We hypothesize that this is due to lower inherent social interaction patterns in urban environments in relation to the heterogencous and dynamic causal nature of multiagent systems in the sports domain (Makansi et al. 2022).
Investigating Latent Structure To quantify the benefit of augmenting the latent space with an explicit causal graph Gt, we compare performance metrics of the GVRNN and an adapted DIA version at varying observation and prediction lengths. For this, we modify the DIA architecture such that the second part of the training procedure is identical to the GVRNN computations. In this way, the resulting performance differences capture only influences originating from the proposed graph mechanism. The left part of figure visualizes the resulting numbers. As can be seen, DIA version yields substantial performance gains, with the performance difference increasing with task complexity, i.e., prediction horizon. We note that increase in GVRNN complexity is accompanied by a further increase performance difference, see Appendix C.3 The results thus highlight that modeling relations via attention-baan based aggregation strategies is insufficient capture decisive social signals in non-trivial multiagent systems.
The right part of Figure 3 shows predictive results for different agent subgroups when varying the representational capacity of the latent interaction graph & E G. Since separate function approxima- tors are parameterized for each selected interaction class, enlarging the interaction space dimension- ality causes overfitting issues, despite higher model expressivity. Thus, the best results are realized by the model configuration with the lowest dimensionality, which still sufficiently encapsulates the underlying system dynamics. Accordingly, this experimental setup can shed light on whether the proposed interaction mechanism &, behaves according to our theoretical considerations. For ex- ample, in modeling defensive players, we would expect reactive patterns based on the behavior of attacking players, resulting in little intra-group dependencies. Indeed, the results in Figure 3 indi- cate that our model inherits the expected social behaviors, as the best parameter configuration is in line with the structural complexity of the different player subgroups. Appendix CC.4 provides an analogous discussion for agent types K,, E G..
Auxiliary Task Finally, examine the extent to which learned latent categories Gr serve mean- ingful semantic contributions beyond trajectory generation tasks. To this end, we report quantitative metrics classification task using soccer data. In the first step, we assemble dataset in which the social context at each time step is defined by the estimated agent- ball influences of a DIA training run. That is, agent locations annotated "no interaction" when considering influences on the ball

---

Under review as a conference paper at ICLR 2023
16
0.50
+
GVRAM DIA (FDE)
0.25
12
OVRRN (FOe)
6.00
ALL DAT
5.75
1.50
9.25
5.00
4.7
4.50
dbc.ess/ped_
Figure 3: Left: Quantitative comparison between GVRNN and an adapted version our method at different observation and prediction horizons modeling offensive players. Right: Representational capacity of latent G. and performance for modeling different agent subgroups. In particular, we fix Kie and report quantitative results by changing the dimensionality of interaction space &s.
MA
d
Figure 4: Left: Trajectories of the offensive players and the ball (blue dots) from data point in the test set of the basketball data. Top right: Influence g(hr)) the red player r on the ball Bottom right: Influence of the ball on the red player.
node are discarded. We then train a spatiotemporal classification network, termed Interaction-NNS for event detection Y' on the updated data (more details given in Appendix [B.3). We compare our method quantitatively against a diagonal (i.e., using ball trajectories only) and fully-connected (i.e., using all agents for neighborhood aggregation) version of Interaction-RNN in Table 2 (right). The model configuration trained on the data previously extracted by DIA outperforms both baseline configurations. The results indicate potential advantages of injecting interaction structures induced by DIA in downstream applications, as they facilitate detecting key information for learning spa- tiotemporal classification tasks.
4.3 QUALITATIVE EVALUATION
To provide more insights in the learned sequential graphs GST of DIA, we visually depict a multi- agent segment from the (and one "no interaction" category) in Figure The upper right image shows the influencing factors of the red player the ball. This interaction structure is mainly characterized by frequent alterna- tions between the two interaction types with scarce instants of low aggregated influence. Since the interaction values are determined accurately reflect the input data, the fluctuating pattern observed illustrates highly dynamic social structures in our tested setting.
and estimated probability values within two interaction categories
The figure below encodes the influential structure of the ball as the sending agent on the movements of the red player. Compared to the previous figure, probability values are consistent with one inter- action type being active over most of the observed time period. The second interaction type becomes the dominant influential force from the moment the ball is passed to the spatially closest neighbor (green player), and also remains active when the ball reaches the target agent (red player). Thus, the first interaction type encodes more holistic influences, while the second type focuses more immediate social motion effects. Notably, there are no periods time in which either the two in- teraction modes is inactive. This distinguishes the ball the central element baskethall and thus
8

---

Under review as a conference paper at ICLR 2023
affects the movement of all offensive players at any time. The qualitative results provide further evidence for the ability of our model to learn a semantically meaningful discrete latent subspace. More qualitative results are given in Appendix C.6
RELATED WORK
A conceptual line of work adresses human trajectory forecasting deterministic regression prob- lem by minimizing the negative log-likelihood and assuming bi-variate Gaussian output distributions Mohamed al.) (2020); Rudenko al. (2020). While assuming unimodal Gaussians may be suffi- cient for pedestrian datasets with rather linear trajectories Makansi et al. (2022), they fall short for multimodal trajectory distributions like in team sports [Brefeld et al. (2019); so do not consider these models in this work. Instead, recent methods mostly propose some form of latent variable model, generally formulated as a conditional VAE (Sohn et al. 2013), capture the stochasticity inherent in future trajectories. At a higher level, the concrete approaches can be broadly categorized by how they model temporal and social dependencies, as well as by their latent space characteriza- tions.
For example, Yeh et al. (2019); Sun et al. (2019); Zhan et al. (2019); Monti et al. (2021 extend the VRNN to handle agent-agent relations Via estimating components of Eq. using graph neural networks. Casas et al. (2020b); Salzmann et al. (200);; Bhattacharyya et al. (2021) choose a sim- ilar task formulation
well interaction encoding strategy (Casas et al. 2020a Anderson et al. 2018). but generate future trajectories non- -on-autoregrressvveyy by flattening the time axis into a single dimension. Other conceputally similar approaches (e.g.. Girgis et al. (2021); Yuan et al. (202) pro- (e.g..Girris
pose transformer-bast
architectures to encode spatial and temporal relations. There a plethora of work that aims improve trajectory forecasting by extending latent information with specific types Zhao al. 2021 Monti et al., 202T Zhan et al. 2019 Fan et al. 2021] Choi al.) 2021 Girase al.
of agent long-term goals extracted from the trajectory data (Mangalam et al. 2020 2021
2021 However, all these approaches model interactions only implicitly by aggregating messages
along the social dimension into spatiotemporal representations. Interestingly, Rudolph et al.] (2020) conclude that graph conditional variational methods for predicting multiagent trajectories are often too powerful and simplified methods (e.g., MDNs) should preferred.
Alternatively, our method can be seen an novel contribution to the line of research that aims to explicitly infer agent interactions while executing trajectory prediction task. This task formulation was originally proposed by Kipf et al. (2018). who introduce the NRI framework, variational autoencoder, where the discrete latent code represents edge predictions in a causal graph. Since the original formulation is limited learning single graph for the entire multiagent sequence, several works propose dynamic extensions where the interaction graph can adapt to the conditions per time step (Graber & Schwing 2020 Li al. 2020 Gong al. 2021 Li al. 2021a). Recently, Löwe et al. (2022) propose to infer causal relations across data points with different underlying interaction graphs but shared dynamics. Other work (Li et al. 2021b) builds upon the NRI framework but leverage a continuous latent space that aims to separate interactive factors from agent intentions.
CONCLUSION
In this paper, we presented novel framework for modeling the joint distribution of agent trajectories using latent variables. To alleviate shortcomings of previous works, we described the data genera- tion process using disentangled factors of variation that explicitly include discrete latent subspace reflecting social structures in sports games. We demonstrated that the emerging architecture per- forms better in predicting trajectories compared to existing strategies and learns informative latent variables. An interesting future line research is modeling the underlying dynamics at a group level to identify unique team strategies (Raman et al. 2021).
AUTHOR CONTRIBUTIONS
If you'd 'd like to, you may include a section for author contributions as is done in many journals. This is optional and at the discretion of the authors.

---

Under review as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
Use unnumbered third level headings for the acknowledgments. All acknowledgments, including those to funding agencies, go at the end of the paper.
REFERENCES
Alexandre Alahi, Vignesh Ramanathan, and Li Fei-Fei. Socially-aware large-scale crowd forecast- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 2203-2210, 2014.
Peter Anderson, Xiaodong Hc, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp-
6077-6086, 2018.
Peter W Battaglia, Jessica B Hamrick. Victor Bapst, Alvaro Sanchez-Gonzaler Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:/806.000 2018.
Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. arXiv preprint 1rXiv:1411.777 2014.
Apratim Bhattacharyya, Daniel Olmeda Reino, Mario Fritz, and Bernt Schiele. Euro-pvi: Pedestrian vehicle interactions in dense urban centers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. 20th SIGNLL Conference on Compu- tational Natural Language Learning, CoNLL 2016, pp. 10-21. Association for Computational Linguistics (ACL), 2016.
Ulf Brefeld, Jan Lasek, and Sebastian Mair. Probabilistic movement models and zones of control. Machine Learning. 10871):1271147, 2019.
Kyle Brown, Katherine Driggs-Camphell, and Mykel J Kochenderfer. A taxonomy and review of algorithms for modeling and predicting human driver behavior. arXiv preprint 2rXXi:2000.088.2... 2020.
Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urtasun. Spagnn: Spatially-aware graph neu- ral networks for relational behavior forecasting from sensor data. 2020 IEEE International Conference Robotics and Automation (ICRA), pp. 9491-9497. IEEE, 2020a.
Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forecasting. European Conference on Computer Vision, pp. 624-641. Springer, 2020b.
Chiho Choi, Srikanth Malla, Abhishek Patil, and Joon Hee Choi. Drogon: A trajectory prediction model based on intention-conditioned behavior reasoning. In Conference on Robot Learning, pp- 49-63. PMLR, 2021.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 4rXiv:1412.3555, 2014.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. Advances in neural information pro- cessing systems, 28, 2015.
Djórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07288 2015.
Uwe Dick, Daniel Link, and Ulf Brefeld. Who can receive the pass? a computational model for quantifying availability in soccer. Data Mining and Knowledge Discovery, 36(3):987-1014, 2022.
10

---

Under review as a conference paper at ICLR 2023
Shiwei Fan, Xiangxu Li, and Fei Li. Intention-driven trajectory prediction for autonomous driving. In 2021 IEEE Intelligent Vehicles Symposium (IV).p. 107-113. IEEE, 2021.
Marco Fraccaro, Spren Kaae Sønderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. Advances in neural information processing systems, 29, 2016.
Justin Gilmer, Samuel Schoenholz, Patrick F Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, PP- 1263-1272. PMLR, 2017.
Harshayu Girase, Haiming Gang, Srikanth Malla, Jiachen Li, Akira Kanehara, Karttikeya Man- galam, and Chiho Choi. Loki: Long term and key intentions for trajectory prediction. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9803-9812, 2021.
Roger
Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D' D'Suza, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Latent variable sequential set trans- formers for joint multi-agent motion prediction. In International Conference on Learning Repre- sentations, 2021.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Dong Gong, Frederic Z Zhang, Javen Qinfeng Shi, and Anton Van Den Hengel. Memory- dynamic neural relational inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1 11843-11852, 2021.
Anirudh Goyal, Alessandro Sordoni, Marc-Alexanee Côté, Nan Rosemary Ke, and Yoshua Bengio. Z--frcing: Training stochastic recurrent networks. arXiv preprint 1711v:/711.05//11 2017.
Colin Graber and Alexander Schwing. Dynamic neural relational inference for forecasting trajecto- ries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 1018-1019, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456. PMLR, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with gumble-softmax. In International Conference Learning Representations (ICLR 2017). OpenReview. net, 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint
arXiv:1412.6980 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. orXiv:1312.611 14, 2013.
arXiv preprint
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference Machine Learning, pp. 2688 2697. PMLR, 2018.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint 1xXiv:1609.029000 2016.
Hoang M Le, Yisong Yue, Peter Carr. and Patrick Lucey. Coordinated agent imitation learning. In International Conference on Machine Learning, pp. 1995-2003, PMLR, 2017.
Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory
prediction with dynamic relational reasoning. Advances neural information processing systems, 33:19783-19794, 2020.
Jiachen Li. Fan Yang. Hengbo Ma, Srikanth Malla, Masayoshi Tomizuka, and Chiho Choi. Rain: Reinforced hybrid attention inference network for motion forecasting. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 16096-16106, 2021a.

---

Under review as a conference paper at ICLR 2023
Longyuan Li, Jian Yao, Li Wenliang, Tong He, Tianjun Xiao, Junchi Yan, David Wipf, and Zheng Zhang. Grin: Generative relation and intention network for multi-agent trajectory prediction. Advances in Neural Information Processing Systems, 34:27107 01-27107-27111 271128 2021b.
Guiliang Liu, Oliver Schulte, Pascal Poupart, Mike Rudd, and Mehrsan Javan. Learning agent representations for ice hockey. Advances in Neural Information Processing Systems, 18715, 2020.
Sindy Löwe, David Madras, Richard Zemel, and Max Welling. Amortized causal discovery: Learn- ing to infer causal graphs from time-series data. In Conference Causal Learning and Reason- ing, pp. 509-525. PMLR, 2022.
C Maddison, A Mnih, and Y Teh. The concrete distribution: A continuous relaxation of discrete random variables. In Proceedings of the international conference on learning Representations. International Conference on Learning Representations, 2017.
Osama Makansi, J von Kügelgen, F Locatello, P Gehler, D Janzing, Thomas Brox, and B Schölkopf. You mostly walk alone: Analyzing feature attribution in trajectory prediction. In International Conference on Learning Representations (ICLR), 2022.
Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajec- tory prediction. In European conference on computer vision, pp. 759-776. Springer, 2020.
Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals, waypoints & paths long term human trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15233-15242, 2021.
Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgenn: A so- cial spatio-temporal graph convolutional neural network for human trajectory prediction. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14424- 14432, 2020.
Alessio Monti, Alessia Bertugli, Simone Calderara, and Rita Cucchiara. Dag-net: Double attentive graph neural network for trajectory forecasting. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 2551-2558. IEEE, 2021.
Shayegan Omidshafiei, Daniel Hennes, Marta Garnelo, Zhe Wang. Adria Recasens, Eugene Tarassov, Yi Yang. Romuald Elie, Jerome T Connor, Paul Muller, et al. Multiagent off-screen behavior prediction in football. Scientific reports, 12(1):1-13, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances neural information processing systems, 32, 2019.
Chirag Raman, Hayley Hung, and Marco Loog. Social processes: Self-supervised meta- learning over conversational groups for forecasting nonverbal social cues. arXiv preprint arXiv:2107.13576. 2021.
Amir Rasouli and John K Tsotsos. Autonomous vehicles that interact with pedestrians: A survey of theory and practice. IEEE transactions on intelligent transportation systems, 21(3)9900-918, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. The International Journal of Robotics Research, 39(8):855-935, 2020.
Yannick Rudolph, Ulf Brefeld, and Uwe Dick. Graph conditional variational models: too complex for multiagent trajectories? 2020.
12

---

Under review as a conference paper at ICLR 2023
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasibe trajectory forecasting with heterogeneous data. European Conference on Computer Vision, pp. 683-700. Springer, 2020.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances neural information processing systems, 29, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
Chen Sun, Per Karlsson, Jiajun Wu, Joshua B Tenenbaum, and Kevin Murphy. Stochastic prediction of multi-agent interactions from partial observations. arXiv preprint arXiv:1902.099 1902.09641, 2019.
Karl Tuyls, Shayegan Omidshafiei, Paul Muller, Zhe Wang, Jerome Connor, Daniel Hennes, Ian
Graham, William Spearman, Tim Waskett, Dafydd Steel, et al. Game plan: What ai can do for football, and what football can for ai. Journal of Artificial Intelligence Research, 71:41-85 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 2017.
Bing Xu, Naiyan Wang. Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint 1rXiv:1505.0085: 00853, 2015.
Raymond A Yeh, Alexander G Schwing, Jonathan Huang, and Kevin Murphy. Diverse generation for multi-agent sports games. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4610-4619, 2019.
Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transform- ers for socio-temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1813-9823, 2021.
Yisong Yue, Patrick Lucey, Peter Carr, Alina Bialkowski. and lain Matthews. Learning fine- fine-grained spatial models for dynamic sports play prediction. In 2014 IEEE international conference on data mining, pp. 670 6799. IEEE, 2014.
Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, and Patrick Lucey. Generating multi-agent trajectories using programmatic weak supervision. In International Conference on Learning Rep- resentations, 2019.
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. In Con- ference on Robot Learning, pp. 895-904, PMLR, 2021.
13

---

Under review as a conference paper at ICLR 2023
A
ALTERNATIVE PERSPECTIVE ON THE OBJECTIVE FUNCTION
JnE
h
h
h
G1
GVRNN
21
h
(c)
(a) GVRNN
(b) DIA
Figure 5: Depicting computational dependencies of (a) GVRNN; (b) the proposed interactive sequential latent variable model (DIA); and (c) DIA in relation to GVRNN, Dashed lines indicate the encoding procedures, solid lines the generation process.
Alternatively, Eq. 5 can be written as
log pe Rec XCClg, (2 Ren, (2, 9ct)]
(G1 Fec, Gel) |l pe, Zel, ((()
T
Σ (GVRNN(T Go) CC h h pe, (Gil (G
1=1
where GgT, zgr 2gT (0(2gr, (erle,rrrrrraar.. (gTT) See also Figure
This can be shown follows:
/
9ce) Q Gee)
po xec
uo(=gr,9rr (x)) log
dg grgddzgr
20 Be 9
T
Po 2 / 2
Zet Ger)
Σ (= G5T Gerlorrrrrrrr)llel log
dG dG<rdzer
as Z 0
zgn Get)
Σ
2 (00 2o (z, x Zcn G /
pa xe
Get)
dz d2
Ges)
ʃʃ [ 00022211201222) - - mg KC
Σ
Zec 95T (0 zee, Gc.)]
KCCgg Zcc, Ged) Celleenneeeeeee Gee)l dzzidG<t
T BeolEgr
- -KCCgalGGltsss Zen
Zen, Ren) pe po zen. Sclll
14

---

Under review as a conference paper at ICLR 2023
B MODEL DETAILS
B.1 GRAPH NEURAL. NETWORKS
The core operations of the GNN modules comprising the DIA can be formally expressed as follows:
ceii (l.))) fe((vs,zz)
(11)
e e- o = fol Σ e(iJ)),
(12)
JEN()
where v; is the input feature representation of agent/node o, is the respective node embedding after 1 GNN layer, (i) denotes the set of agents/nodes that interact with target agent i, e!f.) is the edge embedding of the first GNN layer, and Se and fe are the MLPs described above. For the GNNs of the interaction encoding phase,
define 2 rounds of message passing and utilize additional linear layers operating on the node and edge embeddings of the second GNN layer to output Ir, and &,, respectivell. For the adapted GVRNN modules,
1 layer as described in the main text with additional MLPs computing the mean and variance vectors of the Gaussians.
B.2 MODEL TRAINING & TESTING
Instead of absolute positions, model predicts movements Ad; at each timestep. Consequently, we estimate agent locations via क; = 2;-1+ Aū. For simplicity, 2, refers to both relative and abso- lute positions. For training, the model uses the entire T = Toba pred timesteps from ground-truth sequences Ø<T. To enable gradient flow through stochastic operations, we use reparametrization and gumbel-softmax trick (Maddison al. 2017 Jang et al.) 2017) for sampling from the encoders over the continuous Z! and the discrete G. latent subspace, respectively.
At test time, we divide the trajectories into an observation and prediction period, where the model only observes the first portion of the ground-truth trajectory {æ1...., VTos, } and predicts the remain- ing pred timesteps autoregressively: TTi E Z. Latent variables are sampled from the prior distributions.
B.3 CLASSIFICATION BASELINE
The proposed classification framework for the spatiotemporal multi aaleti regime (Interaction-Rl) is essentially geared towards the functionality of an RNN for capturing temporal dependencies of player trajectories. Additionally, to encode interactive agent patterns, we update the inferred RNN states each time step using attention based graph neural network architecture (Veličković al. 2017). Thus, the updated feature vectors jointly encode social and temporal information. softmax layer is then used for classification and the model is optimized by minimizing the cross- entropy loss. Because the labels denote ball-centric events, we use the output of the ball node for loss computation and evaluation.
final
C EXPERIMENTS DETAILS
C.I DATA
The soccer data contains trajectories of soccer players and the ball extracted from 16 professional soccer matches sampled at 25 frames per second. We assemble a dataset consisting of game excerpts such that the center frames of the sequences timestep t T//2 correspond to an on-ball event from the set y {pass, other ball action, shot, none). We choose sequence length of 2s and downsample the data to 10H?, i.e., T= = 20. This extraction process yields a total of roughly 34000 multiagent segments divided into 70% training, 15% validation, and 15% test data. We center and normalize the trajectories onto the range [-1, 1] and transform them so that the team in possession of the ball always plays from left to right.
The STATS VU basketball data consists of tracking data recorded from the 2016 NBA regular season. Every game sequence has two-dimensional positions of 10 players and the ball sampled at 5 frames per second. The data is split into 60% training, 20% validation, and 20% test sets. All data is translated that the origin of the underlying coordinate system mapped onto the top- corner.
15

---

Under review as a conference paper at ICLR 2023
C.2
IMPLEMENTATION DETAILS
DIA All model variants are implemented using PyTorch (Paszke et al. 2019). Training is carried using Adam (Kingma & Ba 2014) with default parameters and learning of 0.001 and teacher forcing. We select the best performing model using minADE on the validation set. The message passing operations within our graph networks fe and fy are 2-layer MLPs with batch normalization [Tofe & Szegedy, 2013), dropout (Srivastava et al., 2014). and ELU activations (Clevert et al. 2015). In addition, we leverage 2-layer MLPs with LeakyReLUU (Xu al., 2015) activations as Teature extractors operating on 2,, Zg, and 9r. All fully-connected layers are initialized using Xavier initialization (Glorot & Bengio 2010). For recurrence, we use 2-layer GRU networks (Chung et al.) 2014).
Baselines We used the code from the official repositories for baselines (Zhan et al. 2019 Graber & Schwing 2020 Monti et al. 2021 ei al. 2021b] Bhattacharyya al. 2021) and chose the best perfoming hyperparameters from their experiments. For (Yeh et al. 2019)) we re-implemented the model by faithfully following the descriptions in their paper. The overall architecture of GVRNN was initially designed such that it is comparable in parameter number to DIA. However, we found that reducing model expressivity led improved results. If not explicitly stated, the reported num- bers correspond to a GVRNN version with approx 84% parameters compared to DIA.
C.3 DATA & MODEL EFFICIENCY
SOTA (MinACE)
SOTA (MMFFDE)
DIA (mirAAE)
OVINN (MaSADE)
DIA (minFDE)
14
OVANN (PRSADE)
minADE/FDE
12
10
8
20
Training data (%)
Figure 6: DIA versus GVRNN model performance on long-term prediction for varying data fractions of basketball data. The same data was used across the models. The vertical lines show error bars; the horizontal lines the previously best reported results for this task.
minADE
minFDE
Parameters
GVRNN (best)
9.73
15.80
355588
GVRNN (x2 width)
9.91
16.12
733700
GVRNN (x3 width)
9.94
16.20
1647364
DIA (no structure)
10.27
15.17
400836
DIA
8.29
12.05
421768
Table 3: Importance structure versus latent capacity.
To demonstrate the importance of our proposed structure, we perform two sets of additional ex- periments. First, we directly compare the importance of structure versus larger latent capacity by making adjustments to the GVRNN baseline and our proposed model. To achieve this, we (i) enlarge the encoder/prior and decoder by increasing latent, hidden and mnn dimensions of the GVRNN mod- ules and (ii) remove the proposed graph mechanism from our (adjusted) model so that it essentially resembles deeper GVRNN. Secondly, at different data percentages of 5%. 10%, 20%, and 50% of the training data. The training data randomly selected and the same data is used to train the different models. Hence, the purpose
compare generative capabilities of DIA and GVRNN
16

---

Under review as a conference paper at ICLR 2023
here is to investigate whether the integrated structure translates into data efficiency by, for example, governing attention to important pieces of data.
Table 3 shows results to investigate the effect of model size generative capacities. Clearly, larger decodders/enccoder/piioo yield degenerate results compared to the initial (best) GVRNN configura- tion. Furthermore, the DIA comparisons highlight the significant importance of enforcing graph structure since the only difference between both model configurations is our interaction learning strategy. Figure [ shows the data learning efficiency experiments results with mean and error. Im- pressively, our model still significantly outperforms the full GVRNN baseline on both erro met- ries using only 5% (!) of the data. Our model discovers the underlying patterns much faster than GVRNN, whose min ADE numbers improve significantly between 5% and 100% data partitions. Although the results may not be conclusive, they do provide further evidence that our realized im- provements do not originate from increased model complexity, but rather because G, accurately captures the latent ground-truth factors that arise from social perspectives.
Parameters
Sampling Speed
DAG-Net
184040
1.4581
GVRNN
355588
0.0764
DIA (Ours)
421768
0.0931
Table Parameter size and sampling speed for SOTA models. Models were benchmarked on a Nvidia V100 GPU. The sampling speed is the average over mini-batch long-term predictions (Tyred = 40) offensive players.
Table shows parameter sizes and sampling speeds SOTA methods expressed as averaging over mini-batched for modeling offensive players in baskethall. While our model is slower than GVRNN, we emphasize that DIA can reduce runtime by at least 10x compared to GVRNN while maintaining superior results (cf. Figure 6
C.4 AGENT TYPES
# Agent Types
minADE
minFDE
9.18
12.87
9.06
12.74
8.94
11.64
9.00
12.71
Table 5: Quantitative results on basketball data (in meters) modeling all players with Tpred 40 and Take 10 for varying dimensionality of Kr and dimensionality of 1 for the interaction space &
To provide further evidence for potential semantic interpretations of the discrete latent space, examine the effect of capacity change in agent types K, E G, on the generative performance. Table 5 summarizes the results for modeling the dynamics of all players on basketball data. Intuitively, we would expect 3 fundamentally different agent types causing the observed movements since the data consists of attacking and defensive players, and the ball. Similar to Section 4.2 the empirical results confirm this intuition.
17

---

Under review as a conference paper at ICLR 2023
C.5 CONTINUOUS LATENT VARIABLE
An appropriate local solution our learning problem accurately describes the data while collecting implicit information into latent space z1) As part of the main paper, we quali- tatively and quantitatively investigated the dis- covered relational structure Gr. finding seman- tically significant patterns, and demonstrated an accurate approximation of the underlying multimodal data distributions through baseline comparisons. Thus, this section is devoted to quantify the information captured in the contin- uous subspace Zz.
loss
train loss
mmm
A potential metric to measure the degree of dependency between a and 2 i.e., to decide whether the continuous latent variable encodes useful information, is by monitoring the KL di- vergence between the variational posterior and prior. When naively training VAE architectures that consist of decoder exhibiting sufficiently powerful function approximators with autore-
40
80
100
120
140
opoch
Figure 7: The full training loss LDIA and KL loss between prior and approximate posterior on zgr during a training run on soccer data.
gressive dependencies (e.g., RNNs), existing sequential latent variable models frequently report a phenomenon named posterior collapse (Chung et al.] 2013), where the model tends to converge to regions of the loss surface that contain bad local minima (or saddle points) at KL = Since the latent information the decoder receives is then essentially equivalent to Gaussian noise, the model remits to a standard unconditional RNN, i.e., it learns to generate the input solely based on the autoregressive properties independent of the latent information
Figure 7 displays (average) values for losses CDIA and XCC(ge, (2) Zct, G5t)
Po, (ze [C Zect, Gs()] during training run on basketball data. Here we observe a gradual in- crease KI. loss values that is accompanied by a steady decrease in training loss. Thus, the results indicate effective exploitation of the continuous subspace 2, generate future agent movements without the necessity to rely on common optimization strategies such as cost annealing (Bowman
al. 2016 Sønderby al. 2016).
C.6
QUALITATIVE EVALUATION
1.00
probaility
0.75
type
0.50
type
20
30
time
1.0
0.0
0.6
type
type
10
20
50
time
Figure 8: Top: Influence of the ball on the green player. Bottom: Influence of the green player on the ball.
18

---

Under review as a conference paper at ICLR 2023
Figure shows another example of agent-ball interactions for the multi-agent segment from section When considering influences on the ball (upper image), we again see a high frequency with respect to the dominant interaction type. When reversing the influence direction, again observe more constant patterns, where the second interaction type becomes dominant as soon as the ball arrives at the spatially closest agent.
1.00
probabiity
0.50
3
0.00
50
Figure 9: Left: Trajectories of two agents. The numbers indicate times at which the probability values of the right graph undergo rapid change. Right:
Figure 9 depicts an exemplary timeline for player-player interactions in Bernoulli setting, Again, our model learns an interpretable semantic structure, which we describe below. The sharp drop in probability values at around = 10 (right plot) is accompanied with the direction change of the blue player (indicated by symbol 1) in the left plot. The probability spike between
20 and 30 caused by the change of direction of the orange player (symbol 2) at the right comnrr on the pitch. The spike at around 40 is caused by the blue player turning towards the orange player (symbol 3); the last increase in probability occurs when the blue player accelerates towards the orange player (symbol 4).
19


Reviews:
[{"review_id": "uZipHhIcoae", "reviewer": "Reviewer_qCyi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to use a specific structure in the latent space of conditional VAEs used for multi-agent time-series prediction. In particular, the paper argues that the existing use of graph neural nets to capture social dependence between agents for predicting future states contains a lot of redundancies due to the fully-connected nature of the assumed underlying graph. To that end, a latent structure is proposed where at each time step first a node/edge labeling of the fully-connected graph of agents is attained, and then the labels are used to select the parameters of the decoder/prior/encoder that are conditioned on the graph, thereby selecting a particular choice among a discrete set of latent graph structures when computing the final state predictions at each time step. The experiments show improvements in time-series prediction of two team sports (soccer and basketball dynamics) over existing methods, as well as some evidence for learning a sensible latent graph structure.", "summary_of_the_review": "The paper targets an interesting problem: placing a structure on the latent graph for multi-agent time-series prediction; but it lacks clarity in its writing, and its experiments are not convincing as detailed above.\n\n---\n### Post-rebuttal update\nI thank the authors for addressing my concerns. I've raised my score to 5 since the paper now reads better and includes experiments clarifying the importance of the added latent structure. However, since results still lack error bars (all except the new figure), and also it is not explained how the error bars are obtained, I cannot be sure of the significance of the contributions. Nonetheless, I think the paper's main idea is valuable and interesting, and I encourage the authors to rewrite the paper with more focus on statistically robust and well-explained comparisons with state-of-the-art methods.\n", "strength_and_weaknesses": "**Strengths**\n\nThe paper aims to solve an interesting problem with clear applications, and sets up the problem well. The design choices are sufficiently explained (though an ablation study would be nice to verify some of the intuitions).\n\n**Weaknesses**\n\nI have the following concerns regarding the validity of the contributions:\n\n1) The method uses the fully-connected graph structure to infer the discrete latents (agents and type interactions), which are then used to infer the graph structure used for time-series prediction at each step. However, if you are going to use the fully-connected graph as a latent to your latent, why not directly use it as the latent as in GVRNN? To elaborate more, the main contribution of this work in my view is to place a structure on the fully-connected latent graph (since existing works like GVRNN already incorporate the fully-connected latent graph), but if you are going to use the fully-connected graph to learn this additional structure, this seems to essentially defeat the purpose: you are using the fully-connected graph as part of your latent anyways!\n\n2) Since the proposed model is larger (more complex) compared to baselines due to having a set of graphs to choose from instead of a single one (more concretely, the functions involved in encoding/decoding use a discrete set of parameters), how can we be sure that any improvement is not merely due to this increase in complexity? In other words, does simply using “larger” decoders/encoder/priors achieve the same results as your proposed model?\n\n3) I don’t see how the results in Tables 1 and 2 support this statement on page 7: “The empirical results suggest that our proposed modeling strategy effectively both eliminates interfering social signals and discriminates between different agent and interaction types so that future agent movements can be captured more accurately.”\n\n4) None of the reported results show error bars, so it is not clear if the average improvements upon baselines are statistically significant.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lfWH_GA7cA", "reviewer": "Reviewer_Gms6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper claims to incorporate latent factors such as agent intents for the task of multiagent trajectory prediction. It claims to do so by explicitly encoding social structures in sports games to enhance graph-structured latent variable models. Specifically, the method models categorical agent roles and pairwise interactions through what the authors claim is a causal graph. The method is evaluated on professional soccer and basketball game data and performs better than the current state of the art.", "summary_of_the_review": "The paper fails to define and model agent intentions or causality in trajectory forecasting in a principled manner as they claim to do in this work. This, combined with the lack of fair coverage of relevant literature that represents the concepts and modeling approaches dealt with in this paper prevents me from suggesting an accept. ", "strength_and_weaknesses": "## Strengths\n\n1. The quantitative evaluation is extensive, including baseline comparisons, investigating the latent structure, and an auxiliary task. \n2. The alternate perspective on the objective function in the Appendix is insightful \n\n## Weaknesses\n\n### 1\\. Literature Review\n\nThe paper regrettably fails to acknowledge a vast body of related literature, on (i) intention-conditioned trajectory prediction, (ii) variational graph methods for trajectory prediction, and (iii) models that explicitly model social interactions for forecasting. At the very least, these references ought to be mentioned and discussed for a diligent representation of the research space, even if the methods are not directly compared against. \n\n(i) **Intention-Conditioned Trajectory Prediction**:\n\n[R1, R2, R3] talk about intention-conditioned trajectory prediction for autonomous vehicles. Apart from the data the methods are applied to, the architectures can be applicable to, and are relevant for, the problem being addressed here. Crucially, the DROGON paper defines intention explicitly (more on this in Weakness 2. below). \n\n(ii) **Variational Graph Methods**:\n\n[R4] from the Neurips I Can't Believe It's Not Better Workshop explicitly deals with graph conditional variational methods for multi-agent trajectory prediction. The results in that paper are very relevant for this research area and should be included. \n\n(iii) **Encoding Social Interactions**:\n\nGraph and other stochastic methods that encode social interactions between agents have been long applied to trajectory and behavior forecasating problems. [R5] explicitly incorporates a spatiotemporal graph for incorporating social interactions between agents. [R6] more recently explicitly takes a meta-learning approach for modeling the dynamics unique to a group for probabilistic forecasting. A sports team is a group, and if each team is viewed as having unique social dynamics resulting from the team's strategy then [R6]'s core modeling idea is directly applicable. The cue in [R6] terms is simply player location here. Their modeling of social influence of other agents is also permutation invariant, a limitation this paper claims about existing methods. \n\n#### References:\n\n[R1] DROGON: A Trajectory Prediction Model based on Intention-Conditioned Behavior Reasoning - Choi et al. \n\n[R2] Intention-Driven Trajectory Prediction for Autonomous Driving - Fan et al. \n\n[R3] LOKI: Long Term and Key Intentions for Trajectory Prediction - Girase et al. \n\n[R4] Graph Conditional Variational Models: Too Complex for Multiagent Trajectories? - Rudolph et al. \n\n[R5] Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction - Mohamed et al. \n\n[R6] Social Processes: Self-Supervised Meta-Learning over Conversational Groups for Forecasting Nonverbal Social Cues - Raman et al.\n\n### 2\\. Unsupported claims and definitions\n\nThe paper doesn't actually define agent intentions and causality in the specific setting, so there is no reasonable way to evaluate whether the proposed method actually models intentions. The intention-conditioned trajectory works I've mentioned talk about intention over long- and short- time horizons, where e.g. the former is in terms of goal destinations. Here the paper is talking about team sports with player intentions but simply states that this results from communication. What does intention mean here? Also, the paper claims to model causal relationships, but I can't see any explicit causal factors modeled of learned in the graph structure. There might be other exogenous variables explaining trajectory behavior.\n\n### 3\\. Notation\n\nThere are a few notational errors. For instance, the variable used for the sequence cannot be the same as the individual elements: $x_{<t} = [x_1, ...]$. See [R4] for this. In many places there exist grammatical errors and incomplete sentences. Please do a pass to fix these. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hSeLpCE91U", "reviewer": "Reviewer_JaLg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for trajectory prediction in a social context. The proposed model is a variational auto-encoder with a graph based latent space. The graph based latent space is used to capture the effect of social interactions. Evaluation is performed on soccer data which contains trajectories of soccer players and the ball.", "summary_of_the_review": "Although the paper is interesting and novel, a detailed comparison with methods from the very related area of pedestrian trajectory prediction is desirable. Moreover, a detailed analysis of sampling speeds would be welcome.", "strength_and_weaknesses": "Strengths,\n* The proposed method is interesting and novel, in particular the graph based latent space.\n* The proposed method performs well on soccer data outperforming several state of the art methods.\n* The paper is well written and easy to understand.\n\nWeakness,\n* Recent works on trajectory prediction in a social context are not discussed: \"Euro-pvi: Pedestrian vehicle interactions in dense urban centers, CVPR 2021\" which also develops a  variational auto-encoder based joint inference model that learns an expressive multi-modal shared latent space across agents. A discussion and quantitative comparison is desirable.\n* Methods developed for pedestrian trajectory prediction are in general applicable to the target soccer data as these methods also aim to predict trajectories while modelling the effect of social interactions. e.g. Trajectron++, ECCV 2020. A quantitative comparison to such pedestrian trajectory prediction methods are desirable, including a discussion on the advantages/disadvantages of such models for sports data.\n* Sampling speeds: Inference with a complex graph based latent space would be slow. A quantitative comparison of sampling speeds across state of the art models would be desirable.\n* Datasets: It would also be interesting to benchmark the proposed method on pedestrian trajectory prediction datasets e.g. Stanford Drone, Waymo Open, nuScenes to illustrate the differences and challenges involved i sports data vs pedestrian data.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7Tud_1MMMzc", "reviewer": "Reviewer_yCKo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a framework for multiagent trajectories that augments sequential generative models with latent social structures. The contributions are as follows:\n\n1. The authors modeled dynamic dependencies among heterogeneous agents as a graph comprising categorical agent roles and pairwise interactions\n2. The authors proposed the neural network architecture combined with interaction graph learning and adaptive GVRNN (graph variational RNN) based on GVRNN ( Yeh et al. 2019)\n3. They validated their model on data from professional soccer and basketball and show not only improving upon existing state-of-the-art approaches in forecasting trajectories, but also inferring semantically meaningful representations.\n", "summary_of_the_review": "Based on the comment above, I consider that the strength of this paper outperformed the weakness, but I had some concerns at this stage, so I did not give a higher rating.\n\n(After seeing other reviews)\nAfter seeing other reviewers' comments, I consider my rating was too high because I almost agree with these opinions. Therefore, I decreased my rating.\n", "strength_and_weaknesses": "Strength: \n* The first two above contributions may have novelties.\n* The experimental results clearly show the superiority of the proposed approach.\n\nWeakness:\n* The presentation of the motivation and what the authors did may be clear, but why they did so compared to the existing methods and specific technical contributions may be unclear (e.g., specific formulations and network architectures). Moreover, there may be no ablation study, so the empirical (quantitative) contributions were also unclear. \n* For other unclear points, I described the following specific comments.\n\nSpecific comments\n* Where is the DIA explanation (may be the proposed method)? \n* What was the soccer dataset the authors used? I did not find the details.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject