Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a latent variable model for modeling multi-agent trajectories. The authors present a generative framework for modeling and annotating trajectories of multiple agents over time and space. Their approach leverages semi-supervised learning for variational autoencoders where equivariance and interaction of agents are appropriately represented by graph networks. The proposed approach is generally applicable to a wide range of problems, including generation, classification, and pattern detection. Empirically, the proposed approach outperforms all competitors on various interactive spatiotemporal problems.

strengths_and_weaknesses: Strengths:
1. This paper is generally well written and easy to follow. The structure and presentation of this paper are clear.
2. Generative modeling of multi-agent trajectories is an interesting direction to investigate. 

Weaknesses:
1. The technical novelty of this paper needs to be clarified more clearly. The proposed framework in Figure 1 includes two encoders to obtain both discrete and continuous latent variables, which was already proposed in [27]. It would be better to explicitly summarize the major novelty of this work and how it differs from existing methods.
2. For trajectory generation tasks, how do you obtain the labels for latent variable $y$? Is it labeled by humans or automatically based on heuristics? The generation task in semi-supervised settings should be introduced more clearly.
3. Since manual human annotations of events are used in this work, it would be better to also conduct an ablation study on using different portions of the annotated data for training with the purpose of validating how well the proposed method can leverage the small set of annotated data.
4. In the references section, it would be better to replace arXiv versions with the actual publication conferences/journals. Another suggested related paper for relational inference is listed below:
[A] RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecasting, ICCV 2021.
questions: The authors are suggested to address the comments and concerns in the "weaknesses" and "limitations" sections in their rebuttal.
limitations: There seems no discussion on the limitations and potential negative societal impact of this work. It would be better to provide some discussions on these aspects in the rebuttal and revised version.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work presents a semi-supervised approach for modeling multi-agent trajectories in the conditional variational autoencoder (CVAE) framework. The main contribution is the formalization of the combined losses of (weakly) labeled and unlabeled data points in spatiotemporal multi-agent settings. Experiments are performed on the NBA dataset, Stanford Drone Dataset and an Elite soccer dataset not publicly available (as far as I can tell). The proposed approaches (S-MAT and U-MAT) show strong performance compared to a variety of baselines.
strengths_and_weaknesses: Strengths

- I found the Related work section to be well-written and provided good coverage of the related fields.
- As far as I know, the formalization of semi-supervision in the context of CVAE for multi-agent trajectory prediction is novel.

Weaknesses
- The title of the paper is inaccurate. There is a plethora of latent variable models proposed for multi-agent trajectories. And the main part that sets this work apart from prior work is the use of semi-supervision in the CVAE framework.
- I found it difficult to follow section 3, starting from 3.3 onwards. For instance: 
    - y_tilde appears on line 125 without any reference in the equations above. 
    - Figure 1 is referenced once at the end of section 3.5, and its caption is very short and not descriptive.
    - I believe the second part of Equation 2 seems to be missing z_{<t} on the left-hand side.
    - I found the last paragraph of section 3.4 (line 158) unclear. I think this explanation can be improved.
- Having read section 4, I still found it unclear what the labels y correspond to in the supervised and unsupervised cases. For instance, what are the 3 agent types in the U-MAT? Also, my understanding is that S-MAT uses y_t as the area the agent should be in a timestep t. Is this correct? If so, this seems in line with goal-conditioned motion forecasting which has been proposed in prior work (e.g., [1]).
- I’m not sure I follow the argument starting at line 247. If the work in question (e.g., DAG-Net) is using goals that are predicted by the model, why does that render it inappropriate for comparison with unsupervised methods? If everything is predicted, i.e., no ground-truth goal information is used, then it seems fair to me. I would like to also confirm with the authors that S-MAT does not use ground-truth $y_{\leq T}$ and only uses the observed ground-truth labels. 
- There are newer methods that evaluated on the NBA sportVU dataset using a different a different configuration (5 observation timesteps, 10 prediction timesteps). I refer the authors to Table 3 of the EvolveGraph paper ([31] in manuscript). The reason I mention this is because EvolveGraph is one of the stronger methods in the field and it would help to compare the proposed approach to this method.
- The experiments in Section 4.4 on the elite soccer dataset seem to present a marginal improvement over the VRNN baseline. 

References:

[1] Zhao, Hang, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen et al. "Tnt: Target-driven trajectory prediction." arXiv preprint arXiv:2008.08294 (2020).
questions: - Why is the citation on line 114 anonymized? There are many prior works that have stated and showed that human movement is inherently multi-modal. This is why most work in trajectory prediction employs latent variable models and evaluations are typically done minimum over K predictions, where K is the number of modes.
- The sentence on line 143 needs to be proofread: 
“… labeled or unlabeled observations, however, … obtained to also allowing …” → “… labeled or unlabeled observations. However, … obtained to also allow …”

- Line 165: “Arbitrary and possible random order” → “arbitrary order”?
- Can the authors clarify what they mean by: “order agnostic methods rely on the independence of agent trajectories”? Or possibly provide a reference that qualifies this statement. There are many prior approaches in multi-agent motion prediction that are order agnostic and do not rely on the independence of agent trajectories (for example, [2] [3] [4]).
- What is meant by “(un)supervised”? It is present throughout the text and it is unclear to me. 
- Line 243: “Following related work” what’s the citation?
- Out of curiosity, is the elite soccer dataset a contribution that will be released as part of this work?
- Figure 2 seems to be based on the NBA dataset. However, it is referenced in the elite soccer experiments on line 314. Can the author please clarify this point?

References:

[2] Ngiam, Jiquan, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs et al. "Scene transformer: A unified multi-task model for behavior prediction and planning." arXiv e-prints (2021): arXiv-2106.

[3] Casas, Sergio, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. "Implicit latent variable model for scene-consistent motion forecasting." In European Conference on Computer Vision, pp. 624-641. Springer, Cham, 2020.

[4] Girgis, Roger, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D'Souza, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. "Latent Variable Sequential Set Transformers for Joint Multi-Agent Motion Prediction." In International Conference on Learning Representations. 2021.


limitations: Limitations are addressed throughout the text. Societal impact is somewhat addressed in the Checklist; I believe this can be moved to the conclusion.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The work aims at learning temporal dynamics in a multi-agent setting from both labeled and unlabelled data. It combines the framework of VAE with semi-supervised learning and use a graph neural networks. Experiment results on two sports dataset shows superior performance of the proposed framework.
strengths_and_weaknesses: Strength:
There are three major components in the proposed framework: VAE, semi-supervised learning and graph neural networks (GNNs). Each component is strongly motivated and they form a natural combination for the proposed problem setting of multi-agent trajectory modelling. I especially like the part of the application of semi-supervised learning to sequential data. It is of great practical potential. It can not only applied to sequential data without the need to labelling all of them which can be expensive but also provide the possibility of training on multiple sequential datasets including labeled ones and unlabeled ones. The paper is well structured with detailed introduction to the background knowledge and the work is easy to follow. The experiments are thorough. The work also compared against a rich set of baseline models on different tasks on NBA, soccer and Stanford drone dataset.

Weakness: 
The novelty of the work is limited. The proposed approach is a combination of three existing approaches, sequential latent variable models, semi-supervised VAE and graph neural networks. Despite being a natural combination with strong motivation, the novelty of the work is  incremental. I would also encourage the author to include a separate paragraph that summarize the work’s contribution.
questions: 1. Why is the U-MAT model in Table 1 called unsupervised? A predicted label is provided as input to the model. Calling it unsupervised may not be accurate.
2. Why different experiments settings/datasets are compared with different baseline models? For example in Table 1 and Table 2, different baseline models are compared against U-MAT and S-MAT but the experiment settings are very similar. In Table 3, VRNN does not appear in classification results. Even though VRNN is a sequential generative model, it’s hidden state can also be used for classification.

limitations: The work addresses the limitations of the work in terms of application in Section 4 but does not discusses potential negative social impact of the work. In the reviewer's opinion, discussing potential negative social impact may not be necessary for this particular paper as it is primarily a methodology work and the social impacts can depend on specific implication. However, it would definitely be welcome if the author can discuss the paper's limitations from broader social perspectives, in the rebuttal.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work presents a new semi-supervised approach for multi-agent trajectory forecasting and classification. In particular, a new factorization and loss formulation is presented that handles both unsupervised and supervised data simultaneously in the same framework. The resulting model was evaluated on two sports trajectory datasets (NBA and soccer), demonstrating improvements over a variety of baselines.
strengths_and_weaknesses: ### Strengths:

The method tackles an important problem and presents a sensible solution combining weakly-labelled data with unsupervised data.

It is difficult to judge if the method's formulation is novel, but if it is, then this method presents a novel formulation which combines two otherwise-disparate approach types.

The method is evaluated on real-world data comprised of interesting multi-human interaction scenarios.

### Weaknesses:

The abstract is quite light on grounded examples whose inclusion may better allow readers to appreciate this work and its applicability. For instance, what are some spatiotemporal patterns that users wish to identify? What kinds of annotations are required for situations? Are they the positions of agents or text descriptions of events occurring in a scene? These are stated later in the paper, but it would be helpful to state the examples upfront in the abstract as well.

It is quite difficult to identify concretely the core contributions of this work. Section 3 (Main Contribution) spans 3+ pages and interweaves background material (e.g., Section 3.1) with the description of the method at hand. Is the core contribution Equation (1) and its subsequent term derivations?

In Section 3.3, the authors frequently refer to the dependency structures/factorization introduced in Section 3.1, however it is difficult to quickly parse exactly which dependency structures were introduced from text (presumably this is referring to the factorization of p(x, z, y) and q(y, z | x)). Perhaps a figure in Section 3.1 illustrating the probabilistic graphical model would help readers more?

Assuming that the number of agents A remains constant across time (Line 174-175) limits the applicability of this work to many real-world domains such as autonomous driving or social navigation, where the number of detected agents can change as they drift in and out of sensor range or visibility.

Lines 212-215, the soccer dataset split has an error somewhere: "The soccer data consists of 12 matches ... Models are trained on 8 matches, the remaining 6 are ...". Are there 12 total matches or 14?

Line 243: which related work was followed?

Table 1 and 2 seem strange to include together, one shows a prediction horizon of 10 timesteps vs the other's 40 timesteps, with no other changes apparent (other than the methods compared against). Accordingly, it does not appear that Table 1 provides much value compared to Table 2 which focuses on a longer time horizon as well as compares offensive vs defensive player prediction accuracy.

Line 260: Why are all metrics reported on un-normalized scales? Why not meters?

In Section 4.2, do any of the other methods provide probabilistic predictions? It would be best to compare to them on metrics such as negative log-likelihood if so, since minADE/minFDE are only approximations of probabilistic metrics. If not possible to directly compute likelihoods, why not compare using something like the KDE-based NLL from https://arxiv.org/abs/1810.05993?

The left part of Table 3 is hard to interpret, since all methods lie within ~0.10 units of performance of each other. Is this a lot or a little? Without physically-grounded units, it is difficult to tell.

### Minor Comments

The title does not match the OpenReview title.

Fig. 1 is referenced 2 pages away from where it is graphed, perhaps moving it closer to Line 200 would be better?
questions: - What are the core contributions of this work? Is it Equation (1)? Is it an architectural improvement upon an existing neural network architecture? How do the contributions build upon existing works?

- How does the method's performance compare to others on physically-grounded units such as meters?

- Do any of the other baseline methods provide probabilistic outputs? If so, it would be best to compare to them on probabilistic metrics such as negative log-likelihood, since minADE/minFDE are only approximations of probabilistic metrics. If it is not possible to directly compute likelihoods, what about comparing to other methods using a metric like the KDE-based NLL from https://arxiv.org/abs/1810.05993 which more closely approximates a probabilistic metric?

limitations: This work does not state any limitations or potential negative societal impacts of the method.
review_title: null

Meta-review: metareview: This paper studies an interesting problem, and overall the reviewers agreed the exposition and validation are sufficient. We encourage the authors to consider the issues raised by the reviewers and further improve the work in the final version.
confidence: Certain
Decision: Accept