Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to model interactions together with intention using the conditional generative models. For this, the authors created a model to encode two sets of latent variables and provided how they can be used for trajectory prediction. The evaluation is conducted using three datasets (two in the main manuscript and one in the supplementary).  
main_review: Strengths:
- The work is well-motivated, and introduction clearly describes it.
- The methods sounds technical. 

Major concerns:
- One of contributions seems that it samples two types of latent variables, $z_G$ for interaction modeling and $z_A$ for intention modeling. However, I cannot find the validation of such categorization, except verbal description. I would like to see the influence of $z_G$ and $z_A$. 
- In addition, the encoder of GRIN already extracts certain global features as a result of GCN in Eqn.2, considering the interactions of agents. In this view, it is questionable if it is good to categorize $z_G$ and $z_A$ into inter- and intra-agent latent. I don't see any intention information from the generation of $z_A$, and it seems more like spatial, temporal and inter-agent relations similar to [24,7]. 
- Following the previous concerns, one ablative study is that the demonstration of using two latent variables instead of one. Simply, the authors use $z_G,i$ in Eqn. 7. Since both $z_G$ and $z_A$ are sampled from the same latent space, I believe the result might not be huge different from such a replacement. 
- It will be interesting to see the comparison of [24] in the experiments section in addition to [7,17], as it can be categorized as one of the closest work. 
- The evaluation is made using two datasets. However, the Charged dataset provides simple toy examples, and it seems that the NBA dataset is tailored by authors. I am concerned about the validity of these evaluations. The readers of this paper would want to see more thorough comparisons with more diverse works (not only NRI-families but also recent methods) evaluated on more standard datasets. Although the authors provide the evaluation on one of the frequently used benchmark dataset (ETH/UCY) in the supplementary material, its evaluation criteria is different from the standard evaluation in literature, as they sample 100 trajectories. This also makes me doubt about the validity of this method. 

Minor concerns:
- Figure 1 is missing the condition $X_p$ in the decoder.



-----------------
After read the authors' rebuttal, I raise my score.
limitations_and_societal_impact: The authors provide a discussion on privacy-related societal impact. However, the authors should be more thoroughly discuss about it considering more broader aspects.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes GRIN, an architecture that explicitly models the latent uncertainty of social and individual aspects of future prediction of multi-agent systems. The architecture is trained using a conditional variational inference framework, and uses graph attention networks to allow for social attention during the future generation stage. The architecture is evaluated on a Charged particles dataset, an NBA motion forecasting dataset and pedestrian datasets (appendix). Quantitative and qualitative results show that the proposed approach is better than the baselines, and showcases the utility of disentangling the latent space into social and individual components.
main_review: Strengths:
- The major innovation in this work is the decomposition of the latent space into two parts: the first addresses inter-agent relationships and the second addresses intra-agent intention. The effectiveness of this approach is shown both quantitatively (Tab. 2) and qualitatively (Fig. 4).
- The introduction, related work and method sections are well-written and clearly communicate the message to the reader.

Weaknesses:
- The related work section does not address recent motion forecasting approaches, some of which use a similar message passing mechanism [1,2, 3]. These methods have been explored in the context of autonomous driving.
- Other than disentangling the latent space, I am not entirely sure if the architecture as a whole is novel. The architecture seems quite similar to the one proposed in [1].
- The quantitative improvement in Table 1 over Social-GAN seems marginal. Given that there are more recent approaches that outperform Social-GAN on motion forecasting tasks, it would've been interesting to compare to those methods (e.g., [4]).
- The disentanglement Ablation study performed (line 242 and Table 2) seems somewhat counter-intuitive. It seems to me that the correct ablation would be to 1) remove $z_A$ and double the size of $z_G$; 2) remove $z_G$, double the size of $z_A$ and compute the attention weights using the hidden states $h_i^t$. This would allow one to more accurately assess the effect of disentangling the latent space.

References:
- [1] Casas, Sergio, et al. "Implicit latent variable model for scene-consistent motion forecasting." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII 16. Springer International Publishing, 2020.
- [2] Li, Lingyun Luke, et al. "End-to-end contextual perception and prediction with interaction transformer." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.
- [3] Tang, Charlie, and Russ R. Salakhutdinov. "Multiple futures prediction." Advances in Neural Information Processing Systems 32 (2019): 15424-15434.
- [4] Sadeghian, Amir, et al. "Sophie: An attentive gan for predicting paths compliant to social and physical constraints." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

Clarifications:
- Figure 4: The color of the ball (brown) is very difficult to distinguish given the background of the court. 
- The explanation of Figure 4 at line 266 can be improved by outlining that with a fixed $z_G$ the ball always remains with agent 2. It's not entirely clear what happens when we fix $z_A$.
- For the NBA dataset, is there a special identifier for the ball to differentiate it from players in the encoding? If not, how does the model know to treat it's movement differently?

Minor corrections:
- References 18 and 19 are the same.

limitations_and_societal_impact: Limitations and social impacts are adequately discussed in the conclusion.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work presents a deep VAE model for the multi-agent trajectory prediction problem. The assumption that social relationship between agents and each intention is a bias that should be introduced into the network explicitly.
main_review: I think the paper is moderately clear in terms of the content display and the proposed idea is original. The ablation study between Z_A and Z_G was helpful to support the claim of the inductive bias. 

- This raises a question, what if Z_A and Z_G are treated as one tensor Z_AG? 
- Also, I didn't quite understand what makes Z_A agent only and Z_G graph only? is it by design? More details on this are appreciated. 
- One main concern: In eq(1) X_P and X_F are concatenated in the encoder. Practically, you would only use the X_P in the encoder, isn't this like introducing the target to be predicted into the network? How does this works while testing if this is an only training mechanism? 
- Other works such as Social-STGCNN, EvolveGraph, Trajectron++ and similar lines of works do model each agent as a graph and define the relationship between them through the means of graph edged. They define it either by direct handmade function or by learning such a relation. Also, in Trajectron++ an agent graph is created (similar to Z_G). Where does GRAIN stands between these line of works, especially the concept of Z_A and Z_G seems to exists in previous lines of works in a sense?
- The Qualitative analysis misses failure cases and why it happens.

=+=+=+=+=+=+=+=+=+ POST REBUTTAL =+=+=+=+=+=+=+=+=+

I believe that all the concerns where answered in a proper manner except one point. This point is the ADE/FDE 100 samples on ETH/UCY datasets in which most of the papers use the 20 samples metric and rarely uses the 100 samples metrics. Sharing the same concern as pxYX reviewer. Yet, I will raise my score. I believe posting the 20 samples ADE/FDE will improve the paper. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper develops a multivariable multimodal trajectory prediction model titled GRIN which is based on a CVAE framework. Specifically, future trajectories are conditioned on two latent vectors: one (the “interaction” latent) which is intended to represent the interactions between all pairs of entities, and one (the “intent” latent) which is intended to represent the “intent” of each individual entity. These meanings are enforced by the model architecture, as the interaction latent is used to compute attention values which are used to aggregate hidden representations for all pairs of variables, while the intent latent only impacts the prediction of its corresponding entity. Experiments are run on a few trajectory prediction datasets; GRIN outperforms other approaches which model interactions using latent variables.

main_review: ### Originality: 
The novel component of this paper is the model architecture, as the CVAE formulation for trajectory prediction has been developed/used in prior work. The idea of interpreting a per-entity latent variable as an intent is not novel (e.g. [A]), nor is the idea of modeling relations between agents as latent variables (e.g. [7, 17]). To my knowledge, the combination of the two employed here is novel.

### Quality:
- The data used for comparison and the baseline models chosen are appropriate. 
- The novelty of this approach regards the use of the different types of latent variables; however, the predicted latent variables and what they encode are not explored in much detail. The only visualization provided is in Figure 3, for the synthetic data. It would be additionally interesting to explore whether the latent variables encoded any meaningful information for the other datasets.
- This model uses very small latent vector sizes (2 per latent “type”, for a total of 4 dimensions). No experiments are presented which investigate how the size of these vectors indicate performance.
- The appendix presents experimental results on additional data (ETH/UCY), where an additional baseline (Trajectron) is used for comparison. This model performs comparably on this dataset; it would be useful to understand how Trajectron performs on the other data as well (does it still perform comparably? Better? Worse?)
- The latent variable sampling procedure described in lines 168/169 are somewhat strange to use in this context, since the ELBO contains expectations which are computed over the latent variables ([9] doesn’t run into this issue, since it uses a GAN loss). No experiments are run which investigate this choice - does doing this “min sampling” improve performance?


### Clarity: 
The approach is presented clearly, and builds sensibly on prior work related to multimodal trajectory prediction and interaction modeling. Code is provided, and the paper is detailed enough to ensure reproducibility.

### Significance: 
The model outperforms related approaches which model interactions between entities using latent variables. It is unclear whether this approach could be considered “state of the art” compared to other multimodal trajectory forecasting approaches (as mentioned previously, the experimental results indicate it performs comparably to other approaches). It is unclear whether the predicted latent variables are meaningful beyond leading to improved performance, as only one set of latent variables for one trajectory for one dataset are visualized.


### Final Thoughts:
Though GRIN performs well compared against related approaches, it is not completely clear why a practitioner should use this model over alternatives. There is not enough evidence presented to call this approach "state-of-the-art," and there is not enough evidence presented that the predicted latent vectors are useful for interpretation purposes. If the goal is to present a model which explicitly models interactions between entities (such as NRI or dNRI) and performs better compared to prior approaches which do so, then more comparisons need to be made to better understand which component of the model leads to improved performance and whether the latent variable representation leads to meaningful interpretation.


[A]Salzmann, Tim, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. "Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data." ECCV 2020.



# POST-REBUTTAL UPDATE
Thanks for the comment. It's great to see you were able to fix a few things and get better results. Additionally, the additional ablations regarding the impact of the different latent variables on predictions are a valuable addition to the paper. I'm increasing my score based on these additions.
limitations_and_societal_impact: The limitations described are somewhat general and vague, and would apply to most (if not all) of the baselines as well. Are there any limitations to this approach that are unique to it? Are there observed scenarios where the model tends to fail?

The raised negative societal impacts are valid, and there are no ethical concerns raised by this work in its current form.
review_title: null

Meta-review: This work explores two types of latent variables for multi-agent trajectory forecasting, an inter-agent latent code representing social relations and an intra-agent latent code representing agent intentions.   Scores for this paper were initially split between rejection and accept. Additional clarifications and (ablation) experiments provided during the rebuttal caused multiple reviewers to increase their scores.  While the experiments here are not extensive, they were perceived as being sufficient by the reviewers.  After the rebuttal period all reviewers recommend accepting this paper.

The AC recommends acceptance.
Decision: Accept (Poster)