Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper proposed a new simulated dataset that addressing the lack of agent-to-environment interaction in existing datasets. The paper also proposes a comprehensive set of accuracy, realism, and decidability metrics. Form the datasets and metrics, the author found evidences that the new dataset facilitates better robustness and generalization, that current metrics can be misleading, and that there are still remaining challenges to modeling human trajectories. 
strengths: A new agent-to-environment simulated dataset
Comprehensive metrics including accuracy, realism, and decidability metric
weaknesses: While the results are good, the dataset is simulated and lack real-world evidence
correctness: yes, the metrics are well designed
clarity: yes, but the figures are sometimes hard to understand, e.g. Fig 1 is too complicated
relation to prior work: yes
additional feedback: no
review_title: A new agent-to-environment dataset

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: In this work, the authors mainly focus on human trajectory prediction. They propose a simulated agent-to-environment dataset as well as three groups of metrics. They also report and analyze the benchmark results.
strengths: They build a simulated A2X dataset for human trajectory prediction and evaluate the existing state-of-the-art methods. 
They design three groups of metrics to evaluate the accuracy, realism and decidability of the methods. 
The paper is well written and easy to follow. 

weaknesses: My main concern focus on the understanding of the concept ‘environment’, which is very important for this work. However, the authors might fail to explain what is the environment or what does the environment include? The detailed and clear explanation is helpful to support the motivation and distinguish this work with the previous existing work. The authors always claim the existing dataset is with little to no agent-to-agent environment interaction. There is a question that does the environment include the building and cars in the Fig 1 (a) datasets? I think yes. Besides, there exists some work to use such surrounding semantic information to help the human trajectory prediction, such as [1] obeying physical constraints of the environment. Another important questions: what are the applications of the proposed dataset? How do we know the top-view structure map in real world? In the existing trajectory dataset, the trajectories are captured by the camera on the building or the drone. Besides, such trajectory dataset is more suitable to train RL to find the paths. 

[1] Liang J, Jiang L, Niebles J C, et al. Peeking into the future: Predicting future person activities and locations in videos[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 5725-5734.

correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: See the weakness.
review_title: The authors propose a simulated agent-to-environment dataset as well as three groups of metrics for human trajectory prediction. However, , the authors need to further explain what is the environment and the applications in real world. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes a new A2X dataset to the generalization problem of existing human trajectory prediction datasets. Besides, the authors propose multiverse metrics to better evaluate current multimodal models. The results show the benefits of the dataset and metrics.
strengths: + The paper is well written and is easy to read.
+ Clear motivations and formulations
I think developing multiverse metrics to evaluate the multimodal models is quite challenging and is very important for the community of human trajectory prediction.
weaknesses: - The authors claim the new dataset facilitates better robustness and generalization, but it doesn't seem to be reflected in the experiments. And I think the authors should include ablative experiments that compare A2X with existing datasets.
- The effectiveness of MMC, As reported by the authors, is worse than the mean performance of *k* predictions. I wonder if you choose randomly from the *k* predictions, will the performance is the same as the mean performance, i.e. better than the proposed MMC.
correctness: The authors claim the new dataset enables better generalization. I think detailed experiments should be designed to validate this claim.
clarity:  The paper is well written and is easy to follow.
relation to prior work: This paper is clearly discussed how this work differs from previous contributions.
additional feedback: - The multiverse metrics treat all *k* predictions from the multimodal model with equal possibility. However, there exist many methods that predict results with different probabilities of confidence score. Therefore, I suggest the authors consider the case that *k* predictions with different weights.
review_title: Official Review of Paper

Meta-review: This paper focuses mainly on human trajectory prediction. They propose a simulated agent-to-environment dataset as well as three groups of metrics. They also report and analyze the benchmark results.
Scores assigned by the reviewers to this paper point towards rejection.
It seems like there are some misunderstandings of this paper, the authors tried to clarify them in the rebuttal. However, I would recommend the authors rewrite the paper to submit it to a future venue with a clearer structure and address the reviewers' concerns and misunderstandings.
Decision: Reject