Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work discuss the problem of posterior collapse in VAE/CVAE problems and extend this when it is used in the concept of multi-agent  interaction modelling and trajectory prediction problem. They argues that the model designed naively based on VAE for this task is prone to ignore agent interaction when predicting the future trajectory of an agent. The framework proposes a novel sparse graph attention message-passing (sparse-GAMP) layer to address this problem. 
main_review: The problem discussed in this paper is very interesting and the authors well justified this critical issue with VAEs theoretically. My main concern is that the experiments and evaluations are not sufficient to validate the problem and efficacy of the solution. I suggest that the paper should include one experiment without any joint (social) interaction (encoding and decoding for each agent independently) and proves that the results using VAE in social setting is the same (or even worse) than this baseline. Also to show the efficacy of the solution, the authors could try their solution in many frameworks using VAE (trajectron, trajectron++ etc) and show that their sparse-GAMP layer can improve the performance of the state-of-the-art methods for this problem. Without these experiments, it is hard to support the acceptance of the paper with the theoretical arguments and a weak experimental validation only.
limitations_and_societal_impact: No, they haven't.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper analyzes the reason behind the social posterior collapse caused in typical VAEs for multi-agent modeling and proposes improvement schemes. The paper test the proposed model on real-world multi-agent trajectory prediction datasets. This paper also proposes a novel sparse graph attention message-passing.
main_review: originality: good
quality: need to improve
clarity: some designs are not easy to understand
significance: just ok

limitations_and_societal_impact: 1 What's the meaning of Position 2 in the sparse graph attention message-passing towards social posterior collapse detection? 

2 The inferred sparse attention is only used in e2v process. The v2e process still causes large storage and complexity.

3 What is the effects of the sparse graph attention message-passing, compared to the ordinary neural-message-passing without sparse attention? From the aspects of error and efficiency?

4 Many excellent previous works are not introduced for performance comparison, some of which are based on CVAE or neural-message-passing, including:
   
[1] Social lstm:  Human trajectory prediction in crowded spaces

[2] Social attention: Modeling attention in human crowds
   
[3] Social gan:  Socially acceptable trajectories with generative adversarial networks.

[4] Stgat: Modeling spatial-temporal interactions for human trajectory prediction
   
[5] Sophie: An attentive gan for predicting paths compliant to social and physical constraints
   
[6] Multi-agent tensor fusion for contextual trajectory prediction
   
[7] Encoding crowd interaction with deep neural network for pedestrian trajectory prediction
   
[8] Collaborative motion prediction via neural motion message passing
   
[9] Trajectron++:  Multi-agent generative trajectory forecasting with heterogeneous data for control
   
[10] Conditional flow variational autoencoders for structured sequence prediction.
   
[11] Neural relational inference for interacting systems.
   
[12] Social-stgcnn: A social414spatio-temporal graph convolutional neural network for human trajectory prediction.

The results in this paper still need to be improved
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The proposed paper introduces the concept of social posterior collapse in variational autoencoder (VAE) architectures trained for trajectory prediction in interactive environments. Social posterior collapse refers to the VAE ignoring interaction information during training, thus making trajectory predictions solely from the predicted agent's history and map information. The authors present a means to quantify the extent of social posterior collapse through a sparse attention mechanism and propose architectural changes to alleviate the issue.
main_review: I really like the new perspective taken by the authors in investigating existing difficulties of training VAEs in the specific context of trajectory prediction in interactive environments. The authors provide valuable insight with respect to the connections between general machine learning findings and trajectory prediction. The authors do an excellent job in contextualizing the work in terms of existing literature. I would recommend two additional papers in the space of sparse distributions and VAEs from NeurIPS last year to include in the literature review.  

[1] Correia, Gonçalo, et al. "Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity." Advances in Neural Information Processing Systems (2020).

[2] Itkina, Masha, et al. "Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoders." Advances in Neural Information Processing Systems (2020).

Employing sparse attention to quantify the extent of the social context used by the VAE to predict future trajectories is a good idea. The claim of quantifying the social context used to make predictions by the CVAE is well supported by the INTERACTION/Argoverse experiments. The authors also provide extensive experimental analysis in the supplementary material.

I appreciated the transparency and the insightful discussion dedicated to the limitations of the proposed method. Although portraying negative results is valuable, and a rare practice in machine learning papers, I am concerned that the pedestrian experiments actually detract from and confuse the message of the paper in the current framing. On page 2 of the paper, the authors frame the task of interest as predicting multiple trajectories simultaneously (lines 40-41). The problem with current trajectory prediction models is defined as their tendency to ignore interactions over the predicted horizon (lines 72-75). However, the successful experiments focus on a single vehicle trajectory prediction task, demonstrating that the interaction history is important for the prediction. The unsuccessful pedestrian dataset experiments, despite providing significant insight into both the dataset and the problem, do not validate the original hypothesis of the paper. I would reframe the beginning of the paper to focus on ensuring that the \textit{historical social context}, rather than future interaction, is accounted for in the models for predicting single trajectories at a time. That way the successful experiments validate this idea. The pedestrian experiments can be kept as a case study extension in the main paper. Alternatively, successful experiments demonstrating the simultaneous prediction of all trajectories in the scene should be included in the paper, and the pedestrian experiments moved to supplementary.

I have some concerns with respect to the clarity of some of the presented material. Could you clarify the connection between the agent nodes and the $T_{i}$ context encodings? Since $T_{i}$ incorporates information from all of the observed agents by agent $i$, while sparse attention determines which agent connections are being used by the model, I would like to better understand the relationship between the two concepts.

Several variables were not defined (e.g., $\mathbf{s}$ on line 158, $I$ on line 110). It is great to see that the authors did their due diligence by running the experiments on multiple random seems. However, I have some questions regarding the standard deviation reported in Tables 1-3, but particularly Table 1. First, are these standard deviations or standard error? The latter is usually reported as a metric of result significance. Second, if these are standard errors, they are rather large, potentially demonstrating the results to not be significant? Third, why are there standard errors reported for only one of the columns in Table 1? Also in Table 1, could you clarify what the difference was between the models 'Ours' and 'Ours*'? I am not sure what the significance of the discussion of the latter model is in lines 261-265?

The presentation of the paper needs improvement in that it is riddled with typos and grammatical errors. Some examples are as follows, but these are far from exhaustive:

- 'the model is prone to ignore social context' (should be 'prone to ignoring').
- 'a model that models each agent's behavior separately without social context. which might suffer from over-estimated variance and large prediction error.' (punctuation/half-sentence typo).
- 'to interfere the decoding procedure' (missing 'with').

The references should also be proofread for consistency in conference naming, capitalization (e.g., 'gan' -> 'GAN', 'bayes' -> 'Bayes'), etc.

Overall, the authors present an interesting approach to evaluating social posterior collapse in VAE models for trajectory prediction. I really liked the idea and the in-depth discussion provided in the paper. With improvements to the presentation of the paper and the framing of the experiments with respect to the problem statement, this work will be a good contribution!
limitations_and_societal_impact: The authors did not indicate any negative societal impact to their work, but did contextualize the limitations of the work in great detail. I would encourage the authors to think about potential inadvertent consequences such as the amplification of implicit bias in the data in the process of sparsification or concerns regarding deployment of neural network architectures in safety critical systems.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper focuses on analyzing VAE-based frameworks for multi-agent trajectory forecasting from a relationship-modeling perspective. Specifically, it argues that standard VAE frameworks can potentially lead to a situation where the interactions between entities are ignored when predicting future trajectories, which they call “social posterior collapse.” To address this, they augment this framework in two ways: first, they convert the model to a conditional VAE (CVAE) where the prior on latent variables is conditioned on input trajectories; second, they add an auxiliary training task wherein samples from the prior are used to train a second trajectory decoder. To analyze the “social posterior collapse” issue, a graph attention layer that uses sparse attention is developed (titled “Sparse-GAMP”), and a metric called Agent Ratio (AR) is defined which consists of the proportion of surrounding agents to which the target agent is attending. Experiments are run on traffic and pedestrian trajectory prediction datasets, where it is shown that, in situations where social posterior collapse is shown to be present, their model is able to alleviate this issue. Furthermore, the auxiliary prediction task can lead to better model performance.
main_review: ### Originality: 
the study undertaken in the paper is the first to my knowledge which attempts to explore the ability of VAE models for multiagent trajectory prediction tasks to properly model interactions between agents. The VAE framework that forms its basis is commonly used in prior work and appropriate for this study (and proper citations are made throughout).


### Quality:
Much care has been taken to develop the social-CVAE framework, from motivation through derivation. Additionally, the development of the AR metric to evaluate the ability of a model to model interactions between agents is useful for understanding the social posterior collapse issue. The experimentation is thorough, with multiple trials for every setting being run and important ablations being included in the appendix. Overall, the presented results are trustworthy, and the presented social-CVAE approach seems to mitigate social posterior collapse in situations where it appears in other approaches.

That being said, there is one set of experimental results presented in the supplementary material that ought to be mentioned in the main paper, as they alter the conclusions one should draw from this work. Specifically, the results presented in Table 6 indicate that the representation of context used by the model significantly impacts whether social posterior collapse occurs or not. When a lane graph-based representation is not included, no models suffer from social posterior collapse; furthermore, in this situation, the AR metric does not seem to correlate with the distance-based error metrics. These conclusions are additionally supported by the pedestrian dataset results presented in Table 3, where no map context is used. This dependence of social posterior collapse on context representation is important, but as far as I could tell was not mentioned in the main paper anywhere.



### Clarity: 
The paper is written clearly, and is organized well. It is clear which components are drawn from prior work and which are novel developments. Sufficient detail is presented for reproducibility. As mentioned in the previous section, though, I think the results from Table 6 should be alluded to in the main paper.

### Significance: 
As this paper cites, there are a number of papers which use VAE frameworks for multi-agent trajectory prediction. Hence, understanding the properties of these models, including how they model interactions between agents, is very important for the development of improved models. This paper takes a first step in this direction; though the results are not state-of-the-art, the analysis of social posterior collapse is interesting and may be useful for designing future models.


### Overall:
This work is a promising first step in understanding how to ensure interactions are encoded properly when using VAE-based multi agent trajectory prediction approaches. I do think that the lack of discussion regarding the impact of the context encoding representation hurts this paper somewhat, as it is possible to draw some incorrect conclusions without it.

#POST-REBUTTAL UPDATE
Thanks for your comments. Based on your responses, I am keeping my recommendation the same - I think this paper should be accepted. Please add the extra clarifications you mentioned in your response, as they will benefit the text.
limitations_and_societal_impact: The authors fully acknowledge that their results are not state-of-the-art, and they acknowledge that this is a limited study whose results might not apply to all models or domains - the results are not over-sold. There is no mention of potential negative societal impacts in this work.
review_title: null

Meta-review: The paper identifies a failure mode of a common approach to modeling interaction using Variational Autoencoders in multi-agent behavior and trajectory forecasting, which the authors term "Social Posterior Collapse". This identified tendency to underfit in settings where relevant social context could improve prediction and generalization performance. The authors propose an approach to solving this problem using a contextual VAE formulation and auxiliary prediction task, and demonstrate that these lead to the expected empirical improvements in multi-agent interaction settings.


Reviewers assessed the focus of the papers as an important and interesting problem that is well diagnosed theoretically. Given that VAEs are commonly used to model multi-agent interactions, the identified failure mode, proposed metric for diagnosing it, and the novel solution to this problem have good potential for impact. 

A number of concerns were raised in the initial reviews. In particular, reviewers suggested additional empirical validation to more clearly pin point the failure case, and to probe its generality as well as the generality of the new proposed solution. In addition, reviewers made valuable suggestions for improving clarity as well as further increasing the contribution of the paper by moving an experiment from the appendix to the main paper.

In the rebuttal and discussion, reviewers indicated that their concerns were largely addressed. The most negative reviewer unfortunately did not follow up on the discussion. The AC reviewed their concerns and the author response, and assessed the concerns as sufficiently addressed. While the paper can be improved further, remaining issues can be addressed in the camera ready version and do not prevent acceptance of the paper. The AC recommends that the paper is accepted, and strongly encourages the authors to take on board all reviewer suggestions to further improve the camera ready version.
Decision: Accept (Poster)