Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for trajectory prediction in a social context. The proposed model is a variational auto-encoder with a graph based latent space. The graph based latent space is used to capture the effect of social interactions. Evaluation is performed on soccer data which contains trajectories of soccer players and the ball.
strength_and_weaknesses: Strengths,
* The proposed method is interesting and novel, in particular the graph based latent space.
* The proposed method performs well on soccer data outperforming several state of the art methods.
* The paper is well written and easy to understand.

Weakness,
* Recent works on trajectory prediction in a social context are not discussed: "Euro-pvi: Pedestrian vehicle interactions in dense urban centers, CVPR 2021" which also develops a  variational auto-encoder based joint inference model that learns an expressive multi-modal shared latent space across agents. A discussion and quantitative comparison is desirable.
* Methods developed for pedestrian trajectory prediction are in general applicable to the target soccer data as these methods also aim to predict trajectories while modelling the effect of social interactions. e.g. Trajectron++, ECCV 2020. A quantitative comparison to such pedestrian trajectory prediction methods are desirable, including a discussion on the advantages/disadvantages of such models for sports data.
* Sampling speeds: Inference with a complex graph based latent space would be slow. A quantitative comparison of sampling speeds across state of the art models would be desirable.
* Datasets: It would also be interesting to benchmark the proposed method on pedestrian trajectory prediction datasets e.g. Stanford Drone, Waymo Open, nuScenes to illustrate the differences and challenges involved i sports data vs pedestrian data.
clarity,_quality,_novelty_and_reproducibility: Quality: The paper is interesting and novel.

Clarity: the paper is clear and well written.

Originality: To the best of my knowledge, the proposed idea is novel.
summary_of_the_review: Although the paper is interesting and novel, a detailed comparison with methods from the very related area of pedestrian trajectory prediction is desirable. Moreover, a detailed analysis of sampling speeds would be welcome.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper claims to incorporate latent factors such as agent intents for the task of multiagent trajectory prediction. It claims to do so by explicitly encoding social structures in sports games to enhance graph-structured latent variable models. Specifically, the method models categorical agent roles and pairwise interactions through what the authors claim is a causal graph. The method is evaluated on professional soccer and basketball game data and performs better than the current state of the art.
strength_and_weaknesses: ## Strengths

1. The quantitative evaluation is extensive, including baseline comparisons, investigating the latent structure, and an auxiliary task. 
2. The alternate perspective on the objective function in the Appendix is insightful 

## Weaknesses

### 1\. Literature Review

The paper regrettably fails to acknowledge a vast body of related literature, on (i) intention-conditioned trajectory prediction, (ii) variational graph methods for trajectory prediction, and (iii) models that explicitly model social interactions for forecasting. At the very least, these references ought to be mentioned and discussed for a diligent representation of the research space, even if the methods are not directly compared against. 

(i) **Intention-Conditioned Trajectory Prediction**:

[R1, R2, R3] talk about intention-conditioned trajectory prediction for autonomous vehicles. Apart from the data the methods are applied to, the architectures can be applicable to, and are relevant for, the problem being addressed here. Crucially, the DROGON paper defines intention explicitly (more on this in Weakness 2. below). 

(ii) **Variational Graph Methods**:

[R4] from the Neurips I Can't Believe It's Not Better Workshop explicitly deals with graph conditional variational methods for multi-agent trajectory prediction. The results in that paper are very relevant for this research area and should be included. 

(iii) **Encoding Social Interactions**:

Graph and other stochastic methods that encode social interactions between agents have been long applied to trajectory and behavior forecasating problems. [R5] explicitly incorporates a spatiotemporal graph for incorporating social interactions between agents. [R6] more recently explicitly takes a meta-learning approach for modeling the dynamics unique to a group for probabilistic forecasting. A sports team is a group, and if each team is viewed as having unique social dynamics resulting from the team's strategy then [R6]'s core modeling idea is directly applicable. The cue in [R6] terms is simply player location here. Their modeling of social influence of other agents is also permutation invariant, a limitation this paper claims about existing methods. 

#### References:

[R1] DROGON: A Trajectory Prediction Model based on Intention-Conditioned Behavior Reasoning - Choi et al. 

[R2] Intention-Driven Trajectory Prediction for Autonomous Driving - Fan et al. 

[R3] LOKI: Long Term and Key Intentions for Trajectory Prediction - Girase et al. 

[R4] Graph Conditional Variational Models: Too Complex for Multiagent Trajectories? - Rudolph et al. 

[R5] Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction - Mohamed et al. 

[R6] Social Processes: Self-Supervised Meta-Learning over Conversational Groups for Forecasting Nonverbal Social Cues - Raman et al.

### 2\. Unsupported claims and definitions

The paper doesn't actually define agent intentions and causality in the specific setting, so there is no reasonable way to evaluate whether the proposed method actually models intentions. The intention-conditioned trajectory works I've mentioned talk about intention over long- and short- time horizons, where e.g. the former is in terms of goal destinations. Here the paper is talking about team sports with player intentions but simply states that this results from communication. What does intention mean here? Also, the paper claims to model causal relationships, but I can't see any explicit causal factors modeled of learned in the graph structure. There might be other exogenous variables explaining trajectory behavior.

### 3\. Notation

There are a few notational errors. For instance, the variable used for the sequence cannot be the same as the individual elements: $x_{<t} = [x_1, ...]$. See [R4] for this. In many places there exist grammatical errors and incomplete sentences. Please do a pass to fix these. 
clarity,_quality,_novelty_and_reproducibility: * Clarity: The paper is largely clear in writing. There are some broad overclaims in terms of modeling intention or causality, neither of which are clearly defined in context, which is a crucial problem 

* Quality: The quality can significantly be improved through a  principled grounding of the previously mentioned factors in a fairer and more complete review of the relevant literature, some of which I have linked to in the previous section. 

* Novelty: The novelty is limited; as mentioned, conditioning on intentions, incorporating variational graph structures, and modeling social interactions between agents has been done for trajectory and general cue forecasting. This does not prevent an important contribution in this space by itself, but the paper does not explicitly provide one by diligently defining and modeling the factors they claim to do. 

* Reproducibility: The paper provides implementation details in the Appendix that is useful for reproducibiliy
summary_of_the_review: The paper fails to define and model agent intentions or causality in trajectory forecasting in a principled manner as they claim to do in this work. This, combined with the lack of fair coverage of relevant literature that represents the concepts and modeling approaches dealt with in this paper prevents me from suggesting an accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a framework for multiagent trajectories that augments sequential generative models with latent social structures. The contributions are as follows:

1. The authors modeled dynamic dependencies among heterogeneous agents as a graph comprising categorical agent roles and pairwise interactions
2. The authors proposed the neural network architecture combined with interaction graph learning and adaptive GVRNN (graph variational RNN) based on GVRNN ( Yeh et al. 2019)
3. They validated their model on data from professional soccer and basketball and show not only improving upon existing state-of-the-art approaches in forecasting trajectories, but also inferring semantically meaningful representations.

strength_and_weaknesses: Strength: 
* The first two above contributions may have novelties.
* The experimental results clearly show the superiority of the proposed approach.

Weakness:
* The presentation of the motivation and what the authors did may be clear, but why they did so compared to the existing methods and specific technical contributions may be unclear (e.g., specific formulations and network architectures). Moreover, there may be no ablation study, so the empirical (quantitative) contributions were also unclear. 
* For other unclear points, I described the following specific comments.

Specific comments
* Where is the DIA explanation (may be the proposed method)? 
* What was the soccer dataset the authors used? I did not find the details.

clarity,_quality,_novelty_and_reproducibility: The quality was also good in total because of the above contributions and strengths, but I found the above unclear points. The experiments may not be reproduced because they did not provide the code.
summary_of_the_review: Based on the comment above, I consider that the strength of this paper outperformed the weakness, but I had some concerns at this stage, so I did not give a higher rating.

(After seeing other reviews)
After seeing other reviewers' comments, I consider my rating was too high because I almost agree with these opinions. Therefore, I decreased my rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes to use a specific structure in the latent space of conditional VAEs used for multi-agent time-series prediction. In particular, the paper argues that the existing use of graph neural nets to capture social dependence between agents for predicting future states contains a lot of redundancies due to the fully-connected nature of the assumed underlying graph. To that end, a latent structure is proposed where at each time step first a node/edge labeling of the fully-connected graph of agents is attained, and then the labels are used to select the parameters of the decoder/prior/encoder that are conditioned on the graph, thereby selecting a particular choice among a discrete set of latent graph structures when computing the final state predictions at each time step. The experiments show improvements in time-series prediction of two team sports (soccer and basketball dynamics) over existing methods, as well as some evidence for learning a sensible latent graph structure.
strength_and_weaknesses: **Strengths**

The paper aims to solve an interesting problem with clear applications, and sets up the problem well. The design choices are sufficiently explained (though an ablation study would be nice to verify some of the intuitions).

**Weaknesses**

I have the following concerns regarding the validity of the contributions:

1) The method uses the fully-connected graph structure to infer the discrete latents (agents and type interactions), which are then used to infer the graph structure used for time-series prediction at each step. However, if you are going to use the fully-connected graph as a latent to your latent, why not directly use it as the latent as in GVRNN? To elaborate more, the main contribution of this work in my view is to place a structure on the fully-connected latent graph (since existing works like GVRNN already incorporate the fully-connected latent graph), but if you are going to use the fully-connected graph to learn this additional structure, this seems to essentially defeat the purpose: you are using the fully-connected graph as part of your latent anyways!

2) Since the proposed model is larger (more complex) compared to baselines due to having a set of graphs to choose from instead of a single one (more concretely, the functions involved in encoding/decoding use a discrete set of parameters), how can we be sure that any improvement is not merely due to this increase in complexity? In other words, does simply using “larger” decoders/encoder/priors achieve the same results as your proposed model?

3) I don’t see how the results in Tables 1 and 2 support this statement on page 7: “The empirical results suggest that our proposed modeling strategy effectively both eliminates interfering social signals and discriminates between different agent and interaction types so that future agent movements can be captured more accurately.”

4) None of the reported results show error bars, so it is not clear if the average improvements upon baselines are statistically significant.
clarity,_quality,_novelty_and_reproducibility: The paper does not read well in my opinion, for two reasons in particular: First, several statements are somewhat open-ended and confusing to me such as: “The architecture draws on insights from neural interaction learning to allocate all latent information into the intended variables”, “via attaching latent variables to individual agents, their respective representative capacity is governed by the cardinality of the agent set. Hence, the complexity of information inherent in larger multi-agent systems can be accounted for in a natural way” – making the claims more specific and substantial would greatly help this. Second, important details of the implementation (which clarifies a lot of the actual contribution) is move to the appendix (B1 and B2 should be in the main body in my opinion, which also contain under-explained statements such as “using samples from the encoder and ground-truth” – need to be more specific about “using” – and similarly when explaining the inference time).

The quality is lacking in terms of not providing error bars for experiments.

The work seems somewhat novel to me, as I have not seen this particular structure placed on the latent graph, and setting up this structure and making it train successfully does have merit. But I’m not convinced that the structure is useful, I would like to see some experiments directly comparing the importance of structure vs. larger latent capacity. The setup seems mostly reproducible, but the number of channel dimensions, K, and M seem missing.

**Typos**

Page 2: i index missing in definition of D.

Page 4: it is natural is to

B.1: a vectorial representation

B.2: multiagent sequence
summary_of_the_review: The paper targets an interesting problem: placing a structure on the latent graph for multi-agent time-series prediction; but it lacks clarity in its writing, and its experiments are not convincing as detailed above.

---
### Post-rebuttal update
I thank the authors for addressing my concerns. I've raised my score to 5 since the paper now reads better and includes experiments clarifying the importance of the added latent structure. However, since results still lack error bars (all except the new figure), and also it is not explained how the error bars are obtained, I cannot be sure of the significance of the contributions. Nonetheless, I think the paper's main idea is valuable and interesting, and I encourage the authors to rewrite the paper with more focus on statistically robust and well-explained comparisons with state-of-the-art methods.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers agree that this is an interesting application with a rich area of research. In theory, the proposed method could be applied to a large number of different settings. 

However, there are concerns regarding the novelty of the approach, in the sense that it could be better positioned with regards to existing literature on e.g. pedestrian tracking. Relatedly, the reviewers feel that the additional complexity of the proposed approach is not completely justified by the current experiments.
justification_for_why_not_higher_score: While the proposed approach is interesting, the paper could do a better job at positioning itself with respect to the existing literature, and comparing against these approaches.
Decision: Reject