Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: ** Update 8/19 ** 

Thanks to the authors for providing detailed responses to comments. Based on their clarifications and the other reviews, I have updated my score. 

** 

The authors propose a collaborative uncertainty (CU) module to model the uncertainty between agents for multi-agent trajectory prediction.
It is evidenced that when the CU module is incorporated into two recently published models without environment information, there is an improvement in the predictive accuracy.
The authors also provide a visualization for CU, which shows that in situations where there should be no collaborative uncertainty, CU does not produce unintuitive results.

main_review: The references are appropriate, and the exposition has good clarity.
Please also cite these works:
	Ye, Maosheng, Tongyi Cao, and Qifeng Chen. "TPCN: Temporal Point Cloud Networks for Motion Forecasting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
	Gilles, Thomas, et al. "HOME: Heatmap Output for future Motion Estimation." arXiv preprint arXiv:2105.10968 (2021).

Quantitative Experiments:
The quantitative experiments include comparisons to the recent works and leverage well-studied evaluation metrics.
However, I have several concerns.
1.	The authors do not report the number of predicted trajectories used for computing ADE and FDE in any of their results tables. I believe it is K=1.
2.	It is very common for the compared models to rely on the min ADE/FDE across 6 predicted trajectories per test case instead of just 1. This manuscript only reports the results for K=1. As such, it is unclear whether the improvement from the CU module is observed in these critical metrics (min ADE and min FDE for K=6).
3.	Table 3 reports the ADE/FDE for Jean with K=1 from the public Argoverse leaderboard, which conflicts with the Jean results reported in the LaneGCN and VectorNet papers.
4.	The results reported for TNT conflict with the public Argoverse leaderboard.
5.	It is unclear whether the CU module would improve the performance of the current SOTA: HOME, LaneRCNN, and TPCN, which all outperform LaneGCN according to K=1. These should be included in Table 3 and the application of CU should be discussed w.r.t. these works as well.

The toy problem shows simple cases in which the proposed collaborative uncertainty module performs well.
Since the random variable epsilon is effectively noise and independent between synthetic agents, it appears that the synthetic agent trajectories are completely non-interacting (e.g., there are no collisions). Yet, according to Figure 5, there is collaborative uncertainty. Shouldn't this result match Pairs 5-8 in Figure 6?
It is also unclear how this toy problem relates to the trajectories of real interacting agents.

Qualitative results are not shown for scenarios with more than 3 interacting agents in the naive toy problem or more than 2 interacting agents in the Argoverse dataset.
It is therefore difficult to evaluate the estimated CU for important situations with many interacting agents.

limitations_and_societal_impact: 
For improvement suggestions, please refer to the full review.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The work introduces the idea of estimating joint uncertainties between all actors in a motion forecasting setting, instead of the established way of doing so by actor individually. The concept is simple enough to be applied to a variety of different existing motion forecasting systems and the authors demonstrate that the additional loss leads to better prediction accuracy after training.

Overall I liked this paper a lot. It may be a simple idea but it is powerful, as the authors demonstrate in their results. The paper is also very clearly written and the experiments are sufficient.
main_review: **Originality:** 

Estimating single-agent uncertainty is quite common but joint uncertainty or "collaborative uncertainty" is less well-studied. In model-based RL, works like [1] try to capture joint uncertainty as discrete variables but this does not address the uncertainty over the motion of individual actors. The only work that comes to mind, that has attempted something like this in motion forecasting is the recent [2]. In this work, the model captures both a per-actor uncertainty at every step and the collaborative uncertainty, which is expressed as discrete modes that can condition the future rollouts. 
I still think this work is sufficiently novel since it's a more general (theoretical) framework but it may be good to briefly mention this recent work in the related works section.

**Quality:** 

The submission is technically very sound. Well done!

**Clarity:** 

The submission is clearly written and well-organized. I liked the little toy example before the main application to Argoverse and nuScenes. I wasn't able to follow all the math in section 3.3.1 but I hope another reviewer can help point out if this makes sense or not. The rest of the paper is clear.
Together with the supplementary video, Fig.6 is great but on its own, it's harder to parse. Maybe a link in the caption of the figure to the video would be great for the final version of the paper.
I also liked the detailed depictions of the networks and explanation of training details in the appendix, making this easy to reproduce.

**Significance:** 

I think this method is straightforward and easy to adapt into a multitude of motion forecasting systems, leading to better predictive results and more explainability.

**Nitpicks and Questions:** 

1. Starting in the abstract, you write multiple times that something approximates something "nicely" and I think that's a bit too colloquial for a paper
2. line 15: rebuild -> approximate?
3. line 1: no dash between trajectory and forecasting
4. contribution 2 in the abstract mentions improving predictive performance but up until then, only safety+explainability was mentioned
5. Last sentence of abstract sounds like "our predicted output is highly related to the input"
6. Fig. 1 is great but Fig. 2 is kind of redundant wrt Fig. 1. Maybe it'd be possible to redesign Fig. 1 to be a bit bigger and bolder like Fig. 2 and then cut Fig. 2?
7. What's the complexity to compute the loss in eq 4 over the variant with only the diagonal covariance matrix?
8. To read+review, I printed the paper in black/white and Fig. 5a is utterly unreadable that way. But even looking at the color image in the PDF, I can hardly make out what's happening there.
9. Section 4.2.1 is great and more papers should do a quick recap like this.
10. How do you explain the vastly different $\Delta ADE$s across the same model and the same dataset in Tab. 4?
11. Why do you think the Laplacian model leads so consistently to better ADE results in Tab.4?
12. line 285 "less good" -> "worse"
13. line 301 "plug" -> "plugin"? "plug & play"?

**References:**

- [1] Hafner, Danijar, et al. "Mastering atari with discrete world models." arXiv preprint arXiv:2010.02193 (2020).
- [2] Girgis, Roger, et al. "Latent Variable Nested Set Transformers & AutoBots." arXiv preprint arXiv:2104.00563 (2021).
limitations_and_societal_impact: I think you missed the part of the authors' instructions that said to discuss limitations and societal impact of this method. I have to slightly lower my overall score of the paper to be fair to other submissions that discussed these in detail.

For societal impact, I'd mention that this doesn't have any impact beyond other trajectory forecasting models and that the big possible positive consequence could be having more reliable self-driving cars.

For limitations, I'd mention for example that it might not be practical to calculate the predicted CU between all actors in a scene in crowded scenes. Specifically, in a dense urban environment with many pedestrians, this might be tricky and maybe some heuristic might be required here. Also, I'd like to see more of the safety aspect implemented, i.e. the abstract mentioned how this was great for making systems more explainable but then this wasn't really executed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a novel concept — collaborative uncertainty — in the context of multi-agent trajectory forecasting. Unlike many previous works that only model the individual uncertainty of joint future trajectory distribution and use a diagonal covariance matrix, this paper considers both individual and collaborative uncertainty of multi-agent trajectories and uses a full covariance matrix with off-diagonal components representing collaborative uncertainty between agents. The paper argues that such a design could model the GT multi-agent trajectory distribution more accurately. The paper performs synthetic experiments showcasing the method's ability to model joint trajectory distribution better than models that only consider individual uncertainty. Experiments on real-world trajectory forecasting benchmarks also demonstrate that incorporating collaborative uncertainty improves the performance of deterministic baselines.
main_review: **Strength:**

- The proposed concept of collaborative uncertainty is novel in the context of multi-agent trajectory forecasting. Although there are works that model the joint distribution of multi-agent trajectories, little work has addressed the uncertainty (covariance) stemming from the interaction of different agents. Collaborative uncertainty could become an important tool for measuring interactions between agents and improve the interpretability of trajectory forecasting models. I believe this is also a new direction other people could build on.
- The experiments demonstrated the effectiveness of collaborative uncertainty modeling as it improves modeling synthetic data distributions as well as improving the performance of existing deterministic trajectory forecasting methods. It is also nice to see that collaborative uncertainty mainly improves models with interaction modeling, confirming that the performance gain is coming from better uncertainty modeling in agent interaction.
- The paper is generally well-written and well-motivated and super easy to understand.
- The paper also presents detailed derivations of the non-trivial case of Laplace distribution and addresses several practical problems of how to approximate the distribution efficiently and provides proof to justify predicting the auxiliary parameters z.

**Weakness:**

- As admitted by the authors, the biggest limitation of this paper is that collaborative uncertainty has only been applied to deterministic trajectory forecasting models. It is unclear how difficult it is to apply it to generative models such as VAEs and GANs. It is also unclear how effective the method would be for those generative models as these models already have stochasticity and uncertainty built in by latent variables. Given the popularity and effectiveness of generative trajectory forecasting methods, many of them also consider stochastic interaction modeling, the contributions of this paper diminish when not applying to those generative models.
- The method is technically a generative model, and the collaborative uncertainty seems to be able to deal with stochasticity in agent interaction. Thus, it would be good to generate multiple samples and compare with other generative models on both Argoverse and nuScenes datasets for metrics like minADE_k and minFDE_k. The current FDE_1 and ADE_1 results are not very competitive on the nuScenes leaderboard, and many methods with better FDE_1 exist, such as [a]. It would be better to apply collaborative uncertainty to these methods.
- The visualization in Fig 6 only shows simple interaction scenarios when there are only two cars and many scenarios with little or no interaction. It would be more interesting to show the interaction of multiple agents (≥ 3) and how their collaborative uncertainty changes over time. I also don't understand why in Pair 1, the collaborative uncertainty becomes lower as time goes by. One would expect the collaborative uncertainty goes higher as agents are closer and have more chances to interact.

[a] Trajectory Prediction for Autonomous Driving based on Multi-Head Attention with Joint Agent-Map Representation. Messaoud et al. 2020.

**Minor Comments:**

Typos:

- line 105:  change "," to "." or use "and" after ","
- line 257: "shows" → "show"
limitations_and_societal_impact: - The paper discussed its limitations, i.e., currently only applied to deterministic trajectory forecasting models.
- The paper didn't discuss its societal impact, as deemed not applicable by the authors.
review_title: null

Meta-review: All reviewers recommend accepting this paper.

The work addresses the issue of estimating joint uncertainties between actors in a motion forecasting setting.
Experiments are provided on some commonly used benchmarks for motion forecasting.
Reviewers generally felt the paper was clearly written and the experiments were sufficient.

The AC recommends acceptance.
Decision: Accept (Poster)