Semi-Supervised Generative Models for Multiagent Trajectories
Authors: Dennis Fassmeyer
Pascal Fassmeyer
Ulf Brefeld
Venue: NeurIPS 2022 Accept (2022)
Keywords: Generative Models and Autoencoders
Graph Neural Networks
Recurrent Networks
Sequential Models
Semi-Supervised
Multi-Agent

Abstract: Analyzing the spatiotemporal behavior of multiple agents is of great interest to many communities. Existing probabilistic models in this realm are formalized either in an unsupervised framework, where the latent space is described by discrete or continuous variables, or in a supervised framework, where weakly preserved labels add explicit information to continuous latent representations. To overcome inherent limitations, we propose a novel objective function for processing multi-agent trajectories based on semi-supervised variational autoencoders, where equivariance and interaction of agents are captured via customized graph networks. The resulting architecture disentangles discrete and continuous latent effects and provides a natural solution for injecting expensive domain knowledge into interactive sequential systems. Empirically, our model not only outperforms various state-of-the-art baselines in trajectory forecasting, but also learns to effectively leverage unsupervised multi-agent sequences for classification tasks on interactive real-world sports datasets.

Semi-Supervised Generative Models for Multi-Agent
Trajectories
Dennis Fassmeyer Leuphana University of Lüneburg dennis fassmeyer@louphhna...m
Pascal Fassmeyer
Leuphana University of Lüneburg
pascal www.meeyrrrleuphan..... de
Ulf Brefeld
Luephana University of Lüneburg
brefel.@@leuphana.dd
Abstract
Analyzing the spatiotemporal behavior of multiple agents is of great interest to many communities. Existing probabilistic models in this realm formalized either in an unsupervised framework, where the latent space described by discrete continuous variables, in supervised framework, where weakly preserved labels add explicit information to continuous latent representations. To over- come inherent limitations, we propose a novel objective function for processing multi-agent trajectories based on semi-supervised variational autoencoders, where equivariance and interaction of agents are captured via customized graph networks. The resulting architecture disentangles discrete and continuous latent effects and provides natural solution for injecting expensive domain knowledge into inter- active sequential systems. Empirically, our model not only outperforms various state-of-the-ar baselines in trajectory forecasting, but also learns to effectively leverage unsupervised multi-agent sequences for classification tasks on interactive real-world sports datasets.
Introduction
Analyzing the spatiotemporal behavior of multiple agents bears great deal of value in many domains
such as autonomous driving [45] [1]. public transportation [29 54], migration [37 23 or sports
analyties (50 M... Particularly, the detection of collective patterns across time and space is of interest
to many communities but non-trivial inter-dependencies between agents render estimating the inherent
multi-modal distributions often difficult.
Existing generative approaches to modeling sequential multi-agent data thus capitalize on variants from graph networks (52 [4] and (sequential) variational autoencoders 225 g being purely unsuper-
vised, their functionality is grounded in uncovering the implicit modular structure in agent-wise latent
variables directly from the observed trajectories via approximate inference 2 48
[55] [7] 16 32).
Although a generative view on the problem is very appealing, approaches this regime either lack interpretable and controllable latent factors (e.g., (55)), thus being limited in practical settings, or put a strong focus on relationship discovery (e.g., (27). which limits their generative capacity.
An alternative is offered by supervised approaches that aim counteract these issues by incorporating discrete behavioral indicators into the generation process (60] 61 (4). Ideally, these labels encode implicit expert knowledge that would be learned by the model and substantially expand the scope of information captured in latent space. Instead of arguing in favor of expensive and tedious manual annotations of spatiotemporal data, however, these approaches propose incorporate heuristic
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

surrogates as makeshift labels into the problem 66 60 38). While reporting impressive predictive accuracies, they require the existence of label sequences for all training instances and cannot include inexpensive and possibly abundant unlabeled data due to their purely supervised nature.
To close this gap, propose semi-supervised variational autoencoders for spatiotemporal multi- agent problems to process (weakly) labeled as well as unlabeled data. Our contribution generalizes the class of semi-supervised variational autoencoders [26] to spatiotemporal domains by using ideas from variational recurrent neural networks g At a high level, the derived objective function subsumes previous approaches into a unified framework via a disentangled latent space that can incorporate supervision signals in arbitrary quantities for structuring the discrete latent subspace. Additional adaptive graph network layers provide order in- and equivariance of agents and render the proposed approach appropriate for multi-agent scenarios. The resulting semi-supervised approach is applicable to wide range of problems including the generation of collective movements as well as classifying situations of interest. Empirically, our contribution significantly advances recent baselines in generation and classification tasks on interactive real-world data.
2 Related Work
Semi-supervised generative models Semi-supervised variational autoencoders have been origi- nally studied by [26] who propose incorporate additional labels in latent space. Related approaches introduce auxiliary variables that leave the original model unchanged but increase the flexibility of the variational posterior [33]. 24 propose to explicitly capture label characteristics in latent space instead of the label values themselves and [39] study generalizations that allow for learning more complex dependency Sependencyystrme structures. However, all the above approaches are introduced only for static domains. Our contribution constitutes a generalization of the above approaches to spatiotemporal domains.
Sequential generative models Approaches designed for predicting agent trajectories are frequently based on ideas from sequential generative models to encode temporal dependencies and address the stochasticity inherent future predictions. Sequential extensions of variational autoencoders Particularly, the VRNN [ is related to our contribution, as both deploy a VAE instance per time step conditioned on a recurrent state. However, their work lacks modeling of a social dimension as well as means to integrate (potentially expensive) discrete behavioral indicators. Other approaches associate single global latent variable with each
with dynamic latent variables include 4 g 13 I.
sequence 55
1 10 9]
graph-based based (GNNs) approaches [46] are natural methodological choice for applications requiring modeling datasets composed of sets. GNNs operate by learning a chain of hidden representations for each node through an iterative process that relies on aggregating messages along edges in each network layer. Prominent instantiations such as GCN (28], GraphS AGE
Graph neural networks Deep graph-
mainly differ in their notion of message passing and neighborhood aggregation [56]. GNN layers can be stacked and allow for context- and
119. GAT (52). and others [1] 51 21 3 59 53
adaptive neighborhoods for each node by incorporating particularly designed skip connections (53). See 7 for a discussion on the function approximation capabilities of this neural network family.
Generative models for sequential multi-agent data Given the potential benefit across various domains, a great deal of recent publications focus on movements pedestrians or self-driving cars
34 however, show that these standard benchmark datasets generally exhibit weak social interaction patterns, so in the remainder, we focus on methods using team sport data. Related approaches show that learning and computing realistic rollouts significantly benefit from the incorporation of makeshift annotations and heuristic surrogates into the problem propose architecture based conditional VAEs [47] to model distributions of future movements of basketball players. [55] 48 combine ideas from graph networks [4] and VRNNs 8 into unified framework aiming modeling data from basketball and soccer. Other [6] aim to explicitly infer discrete latent variables to represent
35
45
15
36
57
44, 2 22 58, 31.
[62 60 38
12 32
related approaches 30 27
17
interaction types while executing trajectory prediction task.
Hence, existing generative models for spatiotemporal pattern detection focus on multi-agent trajectory prediction in (ully-(un)superviser settings. Our contribution represents the first semi-supervised generalization of variational autoencoders for spatiotemporal domains, which allows addressing tasks beyond representational or generative modeling involving expensive annotation processes.

---

3 Methods
3.1 Preliminaries
Variational autoencoders (VAEs)
25 40 allow to capture intrinsic conditional dependencies and
are often used in structured problems. Essentially, they consist of 0-parameterized generative model pe (xr, 2) p(==)pp(() (<)=)) and an associated @-parameterized variational distribution go (z)))))) Optimal parameters {0, } are obtained via maximizing the variational lower bound on the marginal likelihood, given by
(x)) 2 (=)0)) [ogpo(<))>) (Z(gg(z)))) || p(z)]-
It often helpful to augment partially observed discrete labels into the model to guide the generating
process [26]. The resulting model p(æ, 2, p(a|=z v)p()))(() then acts on labeled and unlabeled instances. Consequentially, the variational distribution (3) z z| = qo go (o(2 (zly, æ)) needs to be
defined over both quantities (z and y). Semi-supervised VAËs (SSVAE) are trained by maximizing
Σ (L(x,)) (o)lz)))) + [(x))
(a.y)
log) + logp(z) log ge x) and U(x) Ey ( ()) v)) + (y)(()) denote (variational) lower bounds on labeled and unlabeled instances, respectively, and À balancing term. The original motivation for this framework was inspired by semi-supervised classification tasks. However, SSVAEs are also frequently employed for learning meaningful representations and generating new data, thus rendering it suitable method- ological foundation for performing diverse sets of tasks. The dependency structures of qs and po are displayed visually as part of Figure
where [(x,y) (pu(=|m.w) pog..
3.2 Problem Formulation
We are interested in modeling spatiotemporal multi-agent scenarios and focus on positions x.(a) R22 of an agent a E AA over time s t < T, denoted by agy.. We assume that a non-empty subset of agents A is present in any observed segment and collect their trajectories (in random order) to form collective movements ØST {x gT E A). We will make use of velocities dx/dt and linearized (a)
motion Az, Er Z., for t' of agents in the remainder.
In addition, we introduce discrete labels ((a) Y associated with agent E A time The collection of sequences ver == [yet E A} may be best thought of as discriminative causes of variation in the corresponding trajectories Ø<r- In practice, Vşr may contain anything that aids in generating multi-agent rollouts, including behavioral indicators, long-term goals, or current tasks. Finally, observations #&T accompanied by labels Уşт assemble the labeled share of the IiagN, while unlabeled observations are collected in Du = (eesrmmiggmsse We refer to all data by D = D2 Du.
data, denoted by Dr = {( UgT) n (
We aim at estimating the underlying distribution that generated the observations in D. A straight forward solution maximize the log likelihood given by
log (DD)
1og
Σ Do
Dr
However, human movement is inherently non-determenistic and multi-modal. Thus, is common to model the stochasticity inherent future agent behaviors via latent representations of VAEs. We propose to lower-bound Eq. ass follows.
3.3 SSVAEs for Modeling Multi-Agent Trajectories
We associate disentangled latent realizations {z,, yr with each time step / of the segment while injecting the dependency structures introduced in Section 3.T for the generative and inference parts. To model temporal dependencies, additionally condition the model components on the past

---

SSYAE
SSYAR
SSVAE
SSVAB
SSVAR
Prior
Decoding
Spatiotemporal
Encodling
Full
Figure 1: Illustrative graphical model Dashed lines indicate the encoding procedure, and solid lines generative model. We use a supervupee VAE (SSVAE, Scction per time stop and additionally introduce variables h; and o.. The representations in set o, jointly encapsulate both temporal and spatial patterns. The proposed model accommodate all contingent proportions of supervision for the discrete latent subspace ye-
observations wet and latent variables zct and you (cmp.
The joint distribution factorizes into
II (2 R5t U ye #<c Wew.:ee.eoolllleeeeeeoooo Fctt Wee)
P MgT)
with generating distribution po (a ( zgt, Uet)) and prior distributions po (2 Zec Vet) and Po(ye <<t. Mc)) for latents 2, and yı- Note that in our sequential setting, the priors are included the optimization rather than being represented by fixed distributions to account for latent dynamics and effective sampling inference time.
To derive the variational distributions, we differentiate between observed labels y and latent variables in the remainder. Using the factorization introduced in Section 3. yields
go (o(EsTlecT.VsT II de (z Ue) and 4o ( = II
ao se: Fet. ve)
Labeled instances render reasoning over latents j-t unnecessary since the true ST is already known. Henee, the right-hand side in Eqn (2) can ignored when sampling from Dz. We arrive at the following results and refer to Appendix ?? for the proofs.
Theorem 1. lower bound on log po (# UgT) in Eqn is given by
log po VcT)
[og #5, ver)
Σ -Js-MAr(an, v.)- (3)
Correspondingly, for unlabeled instances drawn from Du. sequential label information need to be estimated as shown in the following theorem.
Theorem 2. Let H() be the entropy of quantity B. A lower bound on log (x(gr) in Eqn 1 is given by
log (x( Σ
-eg.(u.lees.teei.eeee [Js ( w.)l
Σ -J -JU-MAAT(E)) (4)
Eqn 44) encodes the behavior of object go (WVer/ezT)) when dealing with unsupervised data. Since data likelihood log polaoo ( #itt, VsS. tends to comprise the largest factor in L(w,,,), taking the expectation wrt. go (ū, Pigt, zet,, Ver) encourages the model to realize the highest probability mass for the latent label valuel candidates in label space. This is desirable property that reflects our assumption of data generation.
that incurs the smallest reconstruction loss compared to other

---

2o1 Ser Berr
pulas Ric. Ror.
Encoder;
frna
Priorg
Samplins
Decoder
Fran
Prior;
A
Sumpling
Frnn
Encoder,
[[-1
h1-1
GNN
0p-1
ual Megd)
22
Figure 2: The full computational logic when processing unobserved multi-agent segments. Given temporal context encoded using an RNN module, the model leverages customized GNN architecture based on [22 53 (Section(3.5) to infer posteriors (and priors) over the discrete (y,) and continuous (z:) latent factors. The realized latent values. in conjunction with the GNN output, serve input for the decoding module, which generates distribution over agent movements.
The full evidence lower bound (EL (ELBO) is obtained by combining the lower bounds for all data D,
T
P(D 2 Σ E
-J--MAT(EE v.) + DL
ΣΣ
-U- (UU-MAT(E;))
(5)
DU ==1
This formalization focuses on either fully labeled or unlabeled observations. However, more general formulation can be obtained to also allow for partially labeled data points.
3.4 Extension to Classification Settings
Intrinsic our contribution is an encoding module that argues over the label space, qs (o( and hence can be used as a classification network annotating new observations. However, the ELBO in Eqn (5) is oblivious to classification and simply ignores relevant gradient updates for JS-MAT(C (Iz, Vr) by discarding Eqn ( This is clearly inappropriate for semi-supervised learning where the overall goal is to learn an unknown mapping æ y. Following [26], we circumvent this by heuristically incorporating Eqn (2) into the label-dependent objective. Hence, the full training criterion to learn {0, Φ is given by
JMAT(0, (; =
ΣΣ Ju JU-MAT(R:) Du
ΣΣ (Js-MAr((n Ai log go
where Åι balances generative and discriminative learning and Xn the contribution of labeled and unlabeled data, respectively. Thus, JMAT contains fully (un)supervised modeling tasks as special cases via adjusting '1, X2 accordingly. Note that the resulting supervised part in Eqn 6 (right part) differs from its unsupervised counterpart only in observing y, instead of factorizing over }, which results in computing the log- likelihoooo (auxiliary loss) log qo p Z <ts ye)) instead of the entropy H (y, [0 <<t, ).
3.5 Design Choices
As described in Section 3.2 we aim to learn distribution over possible sequences of agent sets that may differ in share of discrete annotations (and cardinality). However, the previously derived objective functions only account for temporal dependencies, that independence of the agent dimension required to realize cermutaaion-invariii interactive systems where future agent movements need to be coordinated with other agents. Thus, it common to propose some form of graph encoding strategy [46) to account for both, equivariance and agent interactions, respectively.
models. This assumption is trivially inappropriate for
Intuitively, optimal interaction modeling allows adaptively accommodate structural information from different levels of granularity for each agent. In team sports, for example, agent nodes should

---

The be able to capture immediate player influences as well as holistic team strategies dependent on the interaction structure best suited for the task at-hand. Motivated by the previous considerations, we customize an GNN architecture that aims to mimic the desired behavior. Specifically, we stack attention-based GNN layers [52] with particularly designed skip-connections [53] and construct the assumed graph structure using the & (spatially) closest agents.
There are various possibilities incorporate graph-based approaches into the overall scheme I An efficient way to model the distributions defining JMA is via hidden agent states h (a) conditioned on representations (a)
(a) Frna ( (a) (n)
(0)
(a) -0-1) with O. = GNN(
where fron denotes an RNN transition function, GNN is the described GAT-based GNN and o {o! a E A} refers to the set of updated agent representations? Since the elements in set O, aggregate neighboring information of the RNN outputs h, via graph networks, the inferred feature vectors describe the entire interactive past of individual agents. While it is reasonable to additionally enforce intra-timestep dependencies the latent variables using encoding and decoding GNN modules, we argue that capturing past spatiotemporal patterns is sufficiently informative for high-frequene data 43]. Hence, assuming conditional independence within t, the joint movement distribution factorizes across the agent dimension and is given by
po (e RCT, yer) = po (a (wh)) = II aEA
po ( (aaa))
where 1/2 = (v)(a)) A} are the distribution parameters and vo(()
(22) y 0
(a)
(a) (a)
computational logic of the full architecture schematically depicted in Fig. and
4 Empirical Evaluation
For evaluation, we focus on team sports as the coordination of players renders these tasks more difficult than other domains 34 Hence, we experiment STATS SportVU NBA for comparison, and tracking data from soccer games of the german national team. As detailed in Section 3.2 we use agent velocities as input to all models and assume linear motion between consecutive observations. For NBA, we adopt the experimental setup and processing strategy from 38
The STATS VU NBA data comprises tracking positions of offensive plays from the 2016 NBA regular season covering more than 1200 different games where game segment are given by sequences of length 50 and contain two-dimensional positions of all agents (10 players and ball) sampled at frames per second. The data is split into 60% training, 20% validation, and 20% test sets. All data is translated so that the origin of the underlying coordinate system mapped onto the top-left corner.
The soccer data consists of 12 matches where positions of players and ball are sampled at 25 frames per second. The tracking data is accompanied by manual event annotations that will also make use of in the remainder. Models are trained on eight matches, the remaining four are distributed evenly into validation and test data (two matches each).
4.1 Baselines
We evaluate versus several baselines on two distinct tasks: trajectory forecasting (Section 4.2 and 4.4) and spatiotemporal classification (Section 4.3 and [4.4.. Since our approach constitutes the first semi- supervised model for label predictions in multi-agent scenarios, we need to rely on self-constructed supervised baselines for scenarios where only few labels are available, which we discuss more detail in the relevant sections.
We empirically evaluate different configurations in Appendix ??.
2 EEppricallly, observe performance gains when exclusively processing discriminative information Vşr using separate recurrent and graph networks. Appendix ?? outlines the implementation details.
We report results on Stanford Drone Data (SDD) 41 in the Appendix.
battpa://gitaan httrpe://gthhabb..m//iooo//2 Pelyeeeeeeeeaaeeeee-eeeeeete
"The source code is available at https: bectps://gtie h/t::://gtttttttt how.a///.ooeaeeaeeeeeeeee////aat MMAeeeMMM/AATMMMM Neur IPS22 IPS22

---

Table 1: Error for NBA in meters for prediction interval of 10 timesteps with an observation period of 40 timesteps. Bold is the lowest avg Le & final Le for the supervised and unsupervised generative models, respectively.
GVRNN
dNRI
GRIN
DAG-Net.
U-MAT
S-MAT
avg Lg
2.60
2.77
3.00
2.10
2.31
1.94
final L2
5.66
5.52
6.12
4.28
4.80
3.99
By contrast, trajectory forecasting is an active topic in different fields (see, e.g., 42 for an overview), which allows comparisons with wide range of baseline models. is difficult determine the current state-of-the-arg in generative methods for sports tracking data because there is no consistent setting in terms of forecasting horizon or data (see e.g.. [32 38) While some approaches are restricted to making predictions for predetermined horizon, our method Iearns spatiotemporal representations autoregressively and thus can used for the full range of prediction scenarios. To include a variety of methods, we conduct two sets of experiments over short (10 timesteps) and long (40 timesteps) prediction horizons. We benchmark against the most recent and methodologically important work described below.
use weak labels in the form of agent objectives (heuristically inferred prior to model training) to improve trajectory reach the next area where the observed movement speed falls below predefined threshold. Adopting the experimental framework
Fully supervised generative models such DAG-Net (38 and Weak-Sup 660
prediction. In these settings, an agent's objective each timestep
of (38] allows us to make further comparisons with Social Social-Ways 2 and STGAT [22] without additional experiments. STGAT [22] augments a standard GAT with an LSTM [20] to capture both spatial and temporal dependencies of social interactions while Social-Waye uses Info-GAN 6 to
learn multimodal predictive distributions.
The majority of generative models for sequential multiagent systems operate in a fully-unsupervised fashion. GVRNN (55) is graph extension of the VRNN for explicitly modeling joint agent move- ments for basketball and soccer data, dNR/ [17 dynamically extracts interaction types that determine the parametrization of the decoder. Finally, GRIN 32 recovers (static) interactions via attention
using a disentangled latent space.
4.2 Exploring Boundary Cases for Trajectory Prediction
Methods We study the purely (un)supervised generation of trajectories by maximizing the respec- tive lower bounds on NBA. For the fully-sueervised variant, the weak labels described in Section
4.1 can be naturally integrated into the overall scheme via treating them as semantic concepts (a) These target areas naturally vary over time so that the sequential label yęr always denotes the next desired position for every involved agent at time l. Following related work [38, 660] we obtain 90
different areas/class labels. The underlying data distribution is estimated by maximizing Equ (3).
The supervised variant of our contribution denoted by S-MAT.
Although done in 60 direct comparison of unsupervised and supervised generative models is inappropriate inasmuch as the latter use ground-trun -tuth labels over the entire observation period (which include future information) at prediction time. Thus, we additionally benchmark against fully unsupervised instantiation of our proposed framework lacking ground-truit information for structuring the discrete latent subspace. To encourage the model to learn semantically meaningful concepts describing distinct movement patterns, we parametrize the decoder dependent on the
predicted agent class ((()) In this context, y(()) may be best thought of dynamically assigned agent roles that accurately explain the observed trajectories. We maximize Eqn (4) using 3 agent types and refer to the unsupervised variant as U-MAT. Further details for S-MAT and U-MAT are given in Appendix ??.
Metrics We measure the quality of the learned multimodal distribution using common standard
metrics: the minimum over 20 generated samples of the average and final Lg distance between predicted a (a)
<7 and observed positions xz7-
(a)
7

---

Table 2: Error for NBA in meters for prediction interval of 40 timesteps with an observation period of 10 timesteps. Top show results for offense, bottom the defense. Bold the lowest avg La & final L2 for the supervised and unsupervised generative models, respectively.
STGAT
Social-Ways
GVRNN
Weak-Sup.
DAG-Net
U-MAT
S-MAT
avg L2
9.94
9.91
9.73
9.47
8.98
9.01
8.11
final L2
15.80
15.19
15.89
16.98
14.08
13.28
12.52
avg L2
7.26
7.31
7.29
7.05
6.87
6.88
6.21
final L
11.28
10.21
10.62
10.56
9.76
9.04
8.45
pACt
0.73
0.3
0.5
070
Figure 3: Left: Prediction probabilities for different events on soccer data. The dashed lines correspond to the human annotations. Right: Exemplary rollouts for offensive players basketball data. The figure shows observed (light color) and generated player trajectories (intense color) well label information indicated by colored boxes, where color intensity corresponds to ground -ruth frequency of the (weakly obtained) location-based based labels.
Results Tables and and show the results. The proposed approaches clearly outperform their peers and realize the lowest average L2 and final Lg distances. The MAT variants effectively accommodate not only mutual influences across agents, but also discrete generative factors, leading to better approximations of the underlying multi-modal distributions. Unsurprisingly, Table [ also shows that defending players exhibit more structure and are easier predict than offensive players. The results of an ablation study are contained in Appendix ??. The right part of Fig. 3 displays exemplary rollouts that underline models' ability to capture highly complex changes in movement directions. The results indicate that the proposed framework denotes valuable contribution to the large body of research that explicitly addresses modeling multi-agent data accurately.
4.3 Semi-Supervised Classification
We now extend the experimental protocol by targeting accurate label discovery in semi-supervised scenarios. We refer to the full model simply as MAT where λo. A1 > O and incorporate the fully su- pervised S-MAT' (Ao 0) as an additional baseline. We also compare an Las-- classification network that addresses inter-agent dependencies by inferring hidden states of agents by the GNN architecture described in Section 3.5) final softmax layer used for classification, and the model optimized by minimizing the negative log-likelihood. We refer to this baseline MA-RNN.
Figure shows the performance of the models on NBA when using 50% labeled and 50% unlabeled examples (left side) and when using varying portions of labeled data (right side), while the supervised baselines have access only to the labeled part. The semi-supervised MAT clearly emerges as the strongest classifier as it converges to the lowest generalization error during training run and the performance benefit increases via reduction of labeled data. Comparing the numbers to supervised peer (S-MAT) highlights the value of incorporating unlabeled data that is effectively utilized by MAT inform classification decisions. Comparing the importance of an effective regularization mechanism becomes obvious; MA-RNN quickly overfits while the lower-bound S-MAT' acts as a regularizer and guarantees gradually increasing predictive accuracies validation sets.
fully supervised approaches S-MAT and MA-RNN,

---

0.4
MA-RNN
0.4
Sup
0.3
0.3
Val. acc
accuracy
0.2
0.2
0.1
0.1
MAT
100
200
0.0
epochs
% of supervised data
Figure 4: Semi-supervised results NBA. Right: Progression of accuracy during training run. Left: Test accuracies for varying amount laheled data.
Table Results on soccer data. Left: supersuue data generation for a prediction interval of timesteps with an observation period of 40 timesteps (errors in meters). Right: Semi-supervise event detection.
NAME
AVG La
FINAL L2
NAME
ACCURACY
F1
VRNN
2.91
7.10
VRNN GNN
2.87
6.98
MA-RNN-DIAG
0.74
0.80
GVRNN
2.89
6.97
MA-RNN
0.85
0,88
S-MAT
2.87
6.91
S-MAT
0.82
0.85
MAT-DIAG
2.84
6.90
MAT
0.88
0.92
MAT
2.81
6.88
4.4 Combining Generation and Classification in Semi-Supervised Settings
In our last of experiments, we study the combination of trajectory generation and classification and resort to tracking data from elite soccer. The reason for favoring this data over NBA is that the soccer tracking data is manually annotated by experts from the data provider. We focus on labels y = {pass, other ball action, shot, none), where class none denotes the absence of all other labels in frame and use 20% annotated data. Every label propagated to the previous five and the subsequent 30 frames. We generate balanced training set where half of the segments carry label none.
The main purpose of this set of experiments is to quantitatively validate the benefits of including expensive human annotations in the proposed modeling framework for structuring the discrete latent subspace. This approach has low additional costs because the method operates semi-supervised, so only a small subset of multi-agent segments needs to be annotated. Given the higher sampling rate and less interactive nature of soccer dynamics, we also aim evaluate the extent which social dependency modeling modules achieve empirical benefits. We design our baselines accordingly.
Generation For trajectory prediction, we compare MAT VRNN 8 an interactive version of thereof with GNNs on the hidden states (VRNN+GNN) GNN), GVRNN (5), S-MAT with long-term goals (introduced in Section 4.2 . and a diagonal MAT version, MAT-Diag. Note that the VRNV constitutes strong competitor for the task at-hand, as it effectively corresponds to a diagonal GVRNN, which has been shown to produce competitive results on soccer data (55]
The left part of Table 3 shows the results. Although the results are generally more similar compared the basketball experiments, the semi-supervised MAT generates trajectories that closest in both metrics to the observed ones. We hypothesize that decreasing the frequency of the data and extracting multi-agent segments that merely consist of highly interactive plays yields an significant increase performance gap.
Classification MAT learns model over the label space y simultaneously to generative model. To evaluate classification performance, we turn the output of the corresponding softmax for given frame into class label whenever exceeded a predefined threshold. We compare the prediction performance to the classification baselines introduced in Section

---

The resulting accuracies and FI scores are summarized in Table 3 Once more, MAT clearly beats the baselines and stands out by the highest accuracy and F1 scores. The result impressively demonstrates the benefit of including unlabeled data for the task. Surprisingly, MA-RNN performs better than S-MAT. This finding in contrast to the results on NBA shown Figure 4 However, soccer played much larger pitch and movements are likely more linear compared to basketball, which leads to less multimodality of the distributions involved and a consequently decreased benefit of variational methods. Together with significantly reduced label space and sufficient amounts of data this may lead to simpler learning task where MA-RNN less prone to overfitting.
The left part of Fig. shows exemplary prediction probabilities for two segments and three possible events. both segments, the algorithm highly confident no shot-action will occur in the near future. The segment on top contains an event of the class other ball action followed by an event of class pass. Both are correctly identified by MAT. The former clearly indicated by a peak the corresponding probability chart, which then decreases to give rise to the following pass action. The segment on the bottom shows solely a single pass event that is also clearly identifiable by a peak the correct frame.
Conclusion
We presented semi-supervised variational autoencoders for spatiotemporal multi-agent scenarios. The proposed approach effectively combined ideas from semi-supervised variational autoencoders, varia- tional recurrent neural networks, and graph neural networks. Empirically, approach clearly out- performed previous state-of-the-an models in interactive sequential generation and s classification tasks in all experiments. The performance underlines the benefit of including unlabeled data in spatiotemporal problems where labeled sequences are either scarce or assembled from weak makeshift signals.
Potential societal impact We have clear focus on team sports. Though dual use of the proposed technique is certainly possible (e.g., military domains), it may be bit far fetched as have no expertise in these domains and would only be able to deliver uneducated guesses about whether this realistic not. Generally, the data and tasks at-hand are privacy-sensitie as the targets are human individuals. While the soccer and basketball players are probably identifiable in the data, they/their contract/their clubs agreed in recording the data.
Acknowledgments
We would like to thank the German Football Association (DFB) for providing data for this study.
10

---

References
[1] Anirudh Goyal ALIAS PARTH GOYAL,, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rose- mary Ke, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks. Advances in neural information processing systems, 30, 2017.
[2] Javad Amirian, Jean-Bernard Hayet, and Julien Pettré. Social ways: Learning multi-modal distributions of pedestrian trajectories with gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[3] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physies. Advances in neural information processing systems, 29, 2016.
[4] Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. arXiv preprint 111161141..7610, 2014.
[5] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz. and Samy Bengio. Generating sentences from continuous space. In 20th SIGNLL Conference on Compu- tational Natural Language Learning, CoNLL 2016, pages 10-21. Association for Computational Linguisties (ACL), 2016.
[6] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
[7] Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. Advances neural information processing systems, 32, 2019.
[8] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data, Advances in neural information processing systems, 28:2980-2999 298800029988810 2015.
[9] Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. Advances neural information processing systems, 28:3079-3087, 2015.
[10] Outo Fabius and Joost Van Amersfoort. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.6588 2014.
[11] Dennis Fassmeyer, Gabriel Anzer, Pascal Bauer, and Ulf Brefeld. Toward automatically labeling situations in soccer. Frontiers in Sports and Active Living, 3, 2021.
[12) Panna Felsen, Patrick Lucey, and Sujoy Ganguly. Where will they go? predicting fine-grained adversarial multi-agent motion using conditional variational autoencoders. In Proceedings of the European conference on computer vision (ECCV), pages 732-747, 2018.
[13] Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. Advances in neural information processing systems, 29, 2016.
[14] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. International conference on machine learning. pages 1272-1272. PMLR, 2017.
[15] Roger Girgis, Florian Golemo, Felipe Codevilla, Jim Aldon D' D'Souza, Martin Weiss, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Autobots: Latent variable se- quential set transformers. arXiv preprint arXiv:2104.0056 2021.
[16] Dong Gong, Frederic Z Zhang, Javen Qinfeng Shi, and Anton van den Hengel. Memory- augmented dynamic neural relational inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11843-11852. 2021.
[17] Colin Graber and Alexander Schwing. Dynamic neural relational inference for forecasting trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1018-1019, 2020.
[18] David Ha and Douglas Eck. A neural representation of sketch drawings. In International Conference on Learning Representations, 2018
[19] William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 1035, 2017.
11

---

[20] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9())
1735-1780, 1997.
[21] Yedid Hoshen. Vain: Attentional multi-agent predictive modeling, Advances in Neural Informa- lion Processing Systems, 30, 2017.
[22] Yingfan Huang, HuiKun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. Stgat: Modeling spatial-temporal interactions for human trajectory prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2272-6281, 2019.
[23] Franziska Hübl, Sreten Cvetojevic, Hartwig Hochmair, and Gernot Paulus. Analyzing refugee migration patterns using geo-tagged tweets. ISPRS International Journal of Geo-Information, (10):302, 2017.
[24] Tom Joy, Sebastian Schmon, Philip Torr, N Siddharth, and Tom Rainforth. Capturing label characteristics in vaes. In International Conference on Learning Representations, 2020.
[25] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.611 14, 2013.
[26] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi- supervised learning with deep generative models. In Advances in neural information processing systems, pages 3581-3589 2014.
[27] Thomas Kipf. Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pages 2688 28987 PMLR, 2018.
[28] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint wrXiv:1609.02077, 2016.
[29] John Krumm and Eric Horvitz. Predestination: Inferring destinations from partial trajectories.
In International Conference on Ubiquitous Computing, pages 243-260. Springer, 2006.
[30] Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. Advances in neural information processing systems, 19119883-1994,, 2020.
[31] Jiachen Li, Fan Yang. Hengbo Ma, Srikanth Malla, Masayoshi Tomizuka, and Chiho Choi. Rain: Reinforced hybrid attention inference network for motion forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16096-16106, 2021.
[32] Longyuan Li, Jian Yao, Li Wenliang, Tong He, Tianjun Xiao, Junchi Yan, David Wipf, and Zheng Zhang. Grin: Generative relation and intention network for multi-agent trajectory prediction. Advances in Neural Information Processing Systems, 34, 2021.
[33] Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. In International conference on machine learning, pages 1445-1453. PMLR, 2016.
[34] Osama Makansi, J von Kügelgen, F Locatello, P Gehler, D Janzing, Thomas Brox, and B Schölkopf. You mostly walk alone: Analyzing feature attribution in trajectory prediction. In International Conference on Learning Representations (ICLR), 2022.
[35] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In European Conference on Computer Vision, pages 759-776. Springer, 2020.
[36] Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals, waypoints & paths to long term human trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15233-15242, 2021.
[37] Patrick L McDermott, Christopher K Wikle, and Joshua Millspaugh. Hierarchical nonlinear spatio-temporal agent-based models for collective animal movement. Journal of Agricultural, Biological and Environmental Statistics, 22(3):294-312, 2017.
[38] Alessio Monti, Alessia Bertugli, Simone Calderara, and Rita Cucchiara. Dag-net: Double attentive graph neural network for trajectory forecasting. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 2551-2558. IEEE, 2021.
12

---

[39] Siddharth Narayanaswamy, Timothy Paige, Jan- Willem Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. 2018.
[40] Danilo Jimenez Rezende. Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[41] Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In European conference on computer vision, pages 549-565. Springer, 2016.
[42] Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. The International Journal of Robotics Research, 39(8):895 935, 2020.
[43] Yannick Rudolph and Ulf Brefeld. Modeling conditional dependencies in multiagent trajectories. In International Conference on Artificial Intelligence and Statistics, pages 10518-10533 PMLR, 2022.
[44] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 1349-1358, 2019.
[45] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 683-700. Springer, 2020.
[46]
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.
[47] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28: 3483-3491, 2015.
[48] Chen Sun, Per Karlsson, Jiajun Wu, Joshua B Tenenbaum, and Kevin Murphy, Stochastic prediction of multi-agent interactions from partial observations. In International Conference on Learning Representations, 2019.
[49] Jennifer Sun, Ann Kennedy, Eric Zhan, David J Anderson, Yisong Yue, and Pietro Perona. Task programming: Learning data efficient behavior representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paternn Recognition, pages 2876-2885, 2021.
[50] Karl Tuyls, Shayegan Omidshafiei, Paul Muller, Zhe Wang, Jerome Connor, Daniel Hennes, Ian Graham, William Spearman, Tim Waskett, Dafydd Steel, et al. Game plan: What ai can do for football, and what football can do for ai. Journal of Artificial Intelligence Research, 71:41-88, 2021.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[52] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[53] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Machine Learning, pages 5453-5462. PMLR, 2018.
[54] Andy Yuan Xue, Rui Zhang, Yu Zheng, Xing Xie, Jin Huang, and Zhenghua Xu. Destination prediction by sub-trajectory synthesis and privacy protection against such prediction. In Data Engineering (ICDE), 2013 IEEE 29th International Conference on, pages 254-265. IEEE, 2013.
[55] Raymond A Yeh, Alexander G Schwing, Jonathan Huang, and Kevin Murphy, Diverse genera- tion for multi-agent sports games. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 4610-4619, 2019.
13

---

[56] Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33, 2020.
[57] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In European Conference on Computer Vision, pages 507-523. Springer, 2020.
[58] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transform- ers for socio-temporal Conference Computer Vision, pages 9813-9823, 2021.
aggent forecasting. In Proceedings of the IEEE/CVF International
[59] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[60] Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, and Patrick Lucey. Generating multi-agent trajectories using programmatic weak supervision. In International Conference on Learning Representations, 2019.
[61] Eric Zhan, Albert Tseng, Yisong Yue, Adith Swaminathan, and Matthew Hausknecht. Learning calibratable policies using programmatic style-consistency. In International Conference on Machine Learning, pages 11001-11011 PMLR, 2020.
[62] Stephan Zheng, Yisong Yue, and Jennifer Hobbs. Generating long-term trajectories using deep hierarchical networks. Advances Neural Information Processing Systems, 29, 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Since this work constitutes the first semi- comi-supervised approach for spatiotemporal multi-agent problems, it is difficult discuss techical limitations this stage. Application- wise, there are clear limitations e.g.. by using weak makeshift labels. We discuss these issues across the paper (e.g.. Section 4.)
(c) Did you discuss any potential negative societal impacts of your work? [No] We have clear focus team sports. Though dual use of the proposed technique is certainly possible (e.g., military domains), it may be a bit far fetched as we have no expertise in these domains and would only be able to deliver uneducated guesses about whether this is realistic or not.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 4.1. - 4.3.
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs contained in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL.? [Yes] The NBA data set is publicly available and provide URL in the paper. The soccer data is private and must not be shared with third parties. We will release the code and include a URL in the paper as well after the paper has been accepted.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4, more details are provided in the Appendixl.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix
14

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See supplemental material
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you' 're using/curating? [Yes] The NBA data is publicly available and free, for soccer we have the permission of the data provider.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] The soccer and basketball players are probably identifiable in the data, but they/their contract/their clubs agreed in recording the data. The soccer data is private and cannot be shared.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "gxAQwBTbRhV", "reviewer": "Reviewer_DZJp", "summary": "This paper presents a latent variable model for modeling multi-agent trajectories. The authors present a generative framework for modeling and annotating trajectories of multiple agents over time and space. Their approach leverages semi-supervised learning for variational autoencoders where equivariance and interaction of agents are appropriately represented by graph networks. The proposed approach is generally applicable to a wide range of problems, including generation, classification, and pattern detection. Empirically, the proposed approach outperforms all competitors on various interactive spatiotemporal problems.\n", "questions": "The authors are suggested to address the comments and concerns in the \"weaknesses\" and \"limitations\" sections in their rebuttal.", "limitations": "There seems no discussion on the limitations and potential negative societal impact of this work. It would be better to provide some discussions on these aspects in the rebuttal and revised version.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is generally well written and easy to follow. The structure and presentation of this paper are clear.\n2. Generative modeling of multi-agent trajectories is an interesting direction to investigate. \n\nWeaknesses:\n1. The technical novelty of this paper needs to be clarified more clearly. The proposed framework in Figure 1 includes two encoders to obtain both discrete and continuous latent variables, which was already proposed in [27]. It would be better to explicitly summarize the major novelty of this work and how it differs from existing methods.\n2. For trajectory generation tasks, how do you obtain the labels for latent variable $y$? Is it labeled by humans or automatically based on heuristics? The generation task in semi-supervised settings should be introduced more clearly.\n3. Since manual human annotations of events are used in this work, it would be better to also conduct an ablation study on using different portions of the annotated data for training with the purpose of validating how well the proposed method can leverage the small set of annotated data.\n4. In the references section, it would be better to replace arXiv versions with the actual publication conferences/journals. Another suggested related paper for relational inference is listed below:\n[A] RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecasting, ICCV 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "P8PJqCASWv2", "reviewer": "Reviewer_KKnc", "summary": "This work presents a semi-supervised approach for modeling multi-agent trajectories in the conditional variational autoencoder (CVAE) framework. The main contribution is the formalization of the combined losses of (weakly) labeled and unlabeled data points in spatiotemporal multi-agent settings. Experiments are performed on the NBA dataset, Stanford Drone Dataset and an Elite soccer dataset not publicly available (as far as I can tell). The proposed approaches (S-MAT and U-MAT) show strong performance compared to a variety of baselines.", "questions": "- Why is the citation on line 114 anonymized? There are many prior works that have stated and showed that human movement is inherently multi-modal. This is why most work in trajectory prediction employs latent variable models and evaluations are typically done minimum over K predictions, where K is the number of modes.\n- The sentence on line 143 needs to be proofread: \n“… labeled or unlabeled observations, however, … obtained to also allowing …” → “… labeled or unlabeled observations. However, … obtained to also allow …”\n\n- Line 165: “Arbitrary and possible random order” → “arbitrary order”?\n- Can the authors clarify what they mean by: “order agnostic methods rely on the independence of agent trajectories”? Or possibly provide a reference that qualifies this statement. There are many prior approaches in multi-agent motion prediction that are order agnostic and do not rely on the independence of agent trajectories (for example, [2] [3] [4]).\n- What is meant by “(un)supervised”? It is present throughout the text and it is unclear to me. \n- Line 243: “Following related work” what’s the citation?\n- Out of curiosity, is the elite soccer dataset a contribution that will be released as part of this work?\n- Figure 2 seems to be based on the NBA dataset. However, it is referenced in the elite soccer experiments on line 314. Can the author please clarify this point?\n\nReferences:\n\n[2] Ngiam, Jiquan, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs et al. \"Scene transformer: A unified multi-task model for behavior prediction and planning.\" arXiv e-prints (2021): arXiv-2106.\n\n[3] Casas, Sergio, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. \"Implicit latent variable model for scene-consistent motion forecasting.\" In European Conference on Computer Vision, pp. 624-641. Springer, Cham, 2020.\n\n[4] Girgis, Roger, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D'Souza, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. \"Latent Variable Sequential Set Transformers for Joint Multi-Agent Motion Prediction.\" In International Conference on Learning Representations. 2021.\n\n", "limitations": "Limitations are addressed throughout the text. Societal impact is somewhat addressed in the Checklist; I believe this can be moved to the conclusion.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- I found the Related work section to be well-written and provided good coverage of the related fields.\n- As far as I know, the formalization of semi-supervision in the context of CVAE for multi-agent trajectory prediction is novel.\n\nWeaknesses\n- The title of the paper is inaccurate. There is a plethora of latent variable models proposed for multi-agent trajectories. And the main part that sets this work apart from prior work is the use of semi-supervision in the CVAE framework.\n- I found it difficult to follow section 3, starting from 3.3 onwards. For instance: \n    - y_tilde appears on line 125 without any reference in the equations above. \n    - Figure 1 is referenced once at the end of section 3.5, and its caption is very short and not descriptive.\n    - I believe the second part of Equation 2 seems to be missing z_{<t} on the left-hand side.\n    - I found the last paragraph of section 3.4 (line 158) unclear. I think this explanation can be improved.\n- Having read section 4, I still found it unclear what the labels y correspond to in the supervised and unsupervised cases. For instance, what are the 3 agent types in the U-MAT? Also, my understanding is that S-MAT uses y_t as the area the agent should be in a timestep t. Is this correct? If so, this seems in line with goal-conditioned motion forecasting which has been proposed in prior work (e.g., [1]).\n- I’m not sure I follow the argument starting at line 247. If the work in question (e.g., DAG-Net) is using goals that are predicted by the model, why does that render it inappropriate for comparison with unsupervised methods? If everything is predicted, i.e., no ground-truth goal information is used, then it seems fair to me. I would like to also confirm with the authors that S-MAT does not use ground-truth $y_{\\leq T}$ and only uses the observed ground-truth labels. \n- There are newer methods that evaluated on the NBA sportVU dataset using a different a different configuration (5 observation timesteps, 10 prediction timesteps). I refer the authors to Table 3 of the EvolveGraph paper ([31] in manuscript). The reason I mention this is because EvolveGraph is one of the stronger methods in the field and it would help to compare the proposed approach to this method.\n- The experiments in Section 4.4 on the elite soccer dataset seem to present a marginal improvement over the VRNN baseline. \n\nReferences:\n\n[1] Zhao, Hang, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen et al. \"Tnt: Target-driven trajectory prediction.\" arXiv preprint arXiv:2008.08294 (2020).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ibv2hnt4jkC", "reviewer": "Reviewer_CLhb", "summary": "This work presents a new semi-supervised approach for multi-agent trajectory forecasting and classification. In particular, a new factorization and loss formulation is presented that handles both unsupervised and supervised data simultaneously in the same framework. The resulting model was evaluated on two sports trajectory datasets (NBA and soccer), demonstrating improvements over a variety of baselines.", "questions": "- What are the core contributions of this work? Is it Equation (1)? Is it an architectural improvement upon an existing neural network architecture? How do the contributions build upon existing works?\n\n- How does the method's performance compare to others on physically-grounded units such as meters?\n\n- Do any of the other baseline methods provide probabilistic outputs? If so, it would be best to compare to them on probabilistic metrics such as negative log-likelihood, since minADE/minFDE are only approximations of probabilistic metrics. If it is not possible to directly compute likelihoods, what about comparing to other methods using a metric like the KDE-based NLL from https://arxiv.org/abs/1810.05993 which more closely approximates a probabilistic metric?\n", "limitations": "This work does not state any limitations or potential negative societal impacts of the method.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\nThe method tackles an important problem and presents a sensible solution combining weakly-labelled data with unsupervised data.\n\nIt is difficult to judge if the method's formulation is novel, but if it is, then this method presents a novel formulation which combines two otherwise-disparate approach types.\n\nThe method is evaluated on real-world data comprised of interesting multi-human interaction scenarios.\n\n### Weaknesses:\n\nThe abstract is quite light on grounded examples whose inclusion may better allow readers to appreciate this work and its applicability. For instance, what are some spatiotemporal patterns that users wish to identify? What kinds of annotations are required for situations? Are they the positions of agents or text descriptions of events occurring in a scene? These are stated later in the paper, but it would be helpful to state the examples upfront in the abstract as well.\n\nIt is quite difficult to identify concretely the core contributions of this work. Section 3 (Main Contribution) spans 3+ pages and interweaves background material (e.g., Section 3.1) with the description of the method at hand. Is the core contribution Equation (1) and its subsequent term derivations?\n\nIn Section 3.3, the authors frequently refer to the dependency structures/factorization introduced in Section 3.1, however it is difficult to quickly parse exactly which dependency structures were introduced from text (presumably this is referring to the factorization of p(x, z, y) and q(y, z | x)). Perhaps a figure in Section 3.1 illustrating the probabilistic graphical model would help readers more?\n\nAssuming that the number of agents A remains constant across time (Line 174-175) limits the applicability of this work to many real-world domains such as autonomous driving or social navigation, where the number of detected agents can change as they drift in and out of sensor range or visibility.\n\nLines 212-215, the soccer dataset split has an error somewhere: \"The soccer data consists of 12 matches ... Models are trained on 8 matches, the remaining 6 are ...\". Are there 12 total matches or 14?\n\nLine 243: which related work was followed?\n\nTable 1 and 2 seem strange to include together, one shows a prediction horizon of 10 timesteps vs the other's 40 timesteps, with no other changes apparent (other than the methods compared against). Accordingly, it does not appear that Table 1 provides much value compared to Table 2 which focuses on a longer time horizon as well as compares offensive vs defensive player prediction accuracy.\n\nLine 260: Why are all metrics reported on un-normalized scales? Why not meters?\n\nIn Section 4.2, do any of the other methods provide probabilistic predictions? It would be best to compare to them on metrics such as negative log-likelihood if so, since minADE/minFDE are only approximations of probabilistic metrics. If not possible to directly compute likelihoods, why not compare using something like the KDE-based NLL from https://arxiv.org/abs/1810.05993?\n\nThe left part of Table 3 is hard to interpret, since all methods lie within ~0.10 units of performance of each other. Is this a lot or a little? Without physically-grounded units, it is difficult to tell.\n\n### Minor Comments\n\nThe title does not match the OpenReview title.\n\nFig. 1 is referenced 2 pages away from where it is graphed, perhaps moving it closer to Line 200 would be better?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HLHKxgb00Y9", "reviewer": "Reviewer_iUsD", "summary": "The work aims at learning temporal dynamics in a multi-agent setting from both labeled and unlabelled data. It combines the framework of VAE with semi-supervised learning and use a graph neural networks. Experiment results on two sports dataset shows superior performance of the proposed framework.", "questions": "1. Why is the U-MAT model in Table 1 called unsupervised? A predicted label is provided as input to the model. Calling it unsupervised may not be accurate.\n2. Why different experiments settings/datasets are compared with different baseline models? For example in Table 1 and Table 2, different baseline models are compared against U-MAT and S-MAT but the experiment settings are very similar. In Table 3, VRNN does not appear in classification results. Even though VRNN is a sequential generative model, it’s hidden state can also be used for classification.\n", "limitations": "The work addresses the limitations of the work in terms of application in Section 4 but does not discusses potential negative social impact of the work. In the reviewer's opinion, discussing potential negative social impact may not be necessary for this particular paper as it is primarily a methodology work and the social impacts can depend on specific implication. However, it would definitely be welcome if the author can discuss the paper's limitations from broader social perspectives, in the rebuttal.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThere are three major components in the proposed framework: VAE, semi-supervised learning and graph neural networks (GNNs). Each component is strongly motivated and they form a natural combination for the proposed problem setting of multi-agent trajectory modelling. I especially like the part of the application of semi-supervised learning to sequential data. It is of great practical potential. It can not only applied to sequential data without the need to labelling all of them which can be expensive but also provide the possibility of training on multiple sequential datasets including labeled ones and unlabeled ones. The paper is well structured with detailed introduction to the background knowledge and the work is easy to follow. The experiments are thorough. The work also compared against a rich set of baseline models on different tasks on NBA, soccer and Stanford drone dataset.\n\nWeakness: \nThe novelty of the work is limited. The proposed approach is a combination of three existing approaches, sequential latent variable models, semi-supervised VAE and graph neural networks. Despite being a natural combination with strong motivation, the novelty of the work is  incremental. I would also encourage the author to include a separate paragraph that summarize the work’s contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept