Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents an approach to predict trajectories of multiple agents. The paper attempts to motivate its model as capturing of "Egocentric View" and "Allocentric View", although the technical details on how interactions between such things could be captured is missing. A particular architecture is proposed without comparison to baselines or alternatives. The proposed model is a combination of Transformers and MLPs (on top of 1-D ResNet stems with different inputs).
strengths_and_weaknesses: Strengths

- Unclear

Weaknesses

- The 'model' presented in the paper lacks technical details and it is difficult to understand how the proposed model captures the properties regarding "Egocentric View" and "Allocentric View" mentioned in the introduction. We are not convinced that applying a series of these 'Aggregation Layers' and 'Edge MLP' are sufficient to capture what is claimed in the paper.

- The technical contribution of the paper seems to be limited to a designed model combining existing Transformer layers and MLP. The technical originality is limited and incremental.

- I think the key thing missing is the baseline experiments, ablation, and visualizations to convince that the proposed method captures egocentric and allocentric information well, and that it does it better than naive baselines such as simply concatenating all such egocentric and allocentric features in the beginning. The paper proposes a particular architecture, and we are not sure whether this is the best way to enable what's claimed compared to alternative baselines.

- It is unclear how this paper is related to robot learning. The paper mentions that trajectory prediction in general is needed for autonomous driving, but it does not provide any details how it could realistically be used for the vehicles. INTERACTION Dataset used in the paper is composed of vehicle trajectories from a top-down view and TrajNet++ is based on pedestrian trajectories obtained from a top-down view, which are from very different views compared to the first-person view of an autonomous vehicle.
summary_of_recommendation: The mismatch between what the introduction is arguing and the model details makes it very difficult to understand what the paper is trying to achieve. What the paper needs to show is that the proposed method really captures properties claimed in the introduction experimentally, by comparing against different baselines and visualizing it. Further, we question whether its lacking connection to robot learning data really makes the paper sufficiently suitable for CoRL.
issues: The authors are encouraged to use robot data (or at least robot sim) and better provide technical details and justification behind the proposed approach. As mentioned above, baseline experiments and ablations to justify the claim will help greatly.


**===================Updated review and suggestion after the discussion phase===================**

I really would like to give the accept rating to this paper as it seems to have a certain contribution, but the concerns I have overweigh it. The discussion with the authors partially addressed some of the initial concerns but it simultaneously also added others.

The authors clarified that the paper is introducing the layer normalizing representations based on multiple views. The interaction with the authors also clarified that the rest of the approach is a Transformer application (and MLP) on top of them. This is one design choice adopting layers, and it is OK as it showcases the benefits of the normalization layer. The authors also have done some experimental comparison to the baselines and ablations, although it could have been against stronger approaches.

The concerns I have are as follows, which I believe should be addressed in order for the paper to be ready in a publishable form.

- The paper should be more upfront about the fact that its "aggregation layer" exactly is a Transformer layer over the outputs of their normalization layer. The Transformer paper [26] is being cited only in Section 6, although the paper starts using Transformer much earlier in the paper. [26] is not even being introduced or discussed in the related work section, which will confuse and mislead the readers. The paper should introduce the Transformer paper properly early in the paper so that the readers can understand what it is, and the paper should explicitly point out how it relates to their approach.

- The interaction with the authors revealed the stance that they want to treat any architecture different from theirs as an "extension" of theirs, which we believe is an overclaim and academically misleading. The paper is introducing one particular design (which is perfectly fine as mentioned above), not the best or optimal design. This has to be clarified in the final version.

- Computational complexity advantage claims in the ablations need to be toned down, as the experimental comparison is only done with weaker baselines designed by the authors. There could be better alternatives with other efficient Transformers, which the paper has not explored.

- The new table specifying the # of parameters of the models need to be added in the final version of the paper. This would allow the readers to judge themselves how many more parameters the proposed approach is using compared to the baselines to get a higher accuracy.

Overall, the paper should be more upfront about what exact technical component it is adopting from the previous papers and where their computation advantage is coming from (i.e. Transformers).
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper present a method for multi-agent prediction that is based on GNNs, but performs joint prediction by alternatively taking a local agent view as well as a view akin to "how do the other agents view each other, and me?" The paper is extremely well-written and intuitive, and this is improved by several visual aides, particularly figures 1 and 2. The method achieves good performance, both in terms of prediction and computational complexity. I encourage the authors to provide stronger justification for a few choices, as I try to detail in my comments below. 


strengths_and_weaknesses: This paper was very well written - I sincerely appreciate the effort to bring intuition into the discourse, which unfortunately is not the general case in academic literature. It was actually a delight to read.

I have a few clarifying questions and comments in the "Issues" section below; otherwise, I do not have any substantive suggestions for improvement.
summary_of_recommendation: My main reasons for this recommendation have to do with relevance, readability, and soundness of the approach. The paper is relevant in two ways: multi-agent prediction is important for myriad robotic tasks, and the focus on GNNs is a very active topic. While I am not 100% convinced that this is an extremely original idea, I do think the notion of taking multiple views in the multi-agent setting is interesting.
issues: I appreciate the early discussion about the terms "egocentric" and "allocentric", which will likely be quite helpful for the target audience at such a conference. 

I appreciate the discussion in lines 36-41. However, I am not sure this is a limitation of GNNs; i.e. if the distinction between egocentric vs allocentric views is as critical as the authors suggest. For example, can't an agent identify proper scales between distances (and other spatial relations) for other agents given current sensing modalities? Perhaps the authors could tighten up these claims. In context of vehicles, it could be that they are all sharing state information via V2X technology, in which case obtaining an unbiased graph representation seems feasible, if not straightforward. 

Intuitively, I like the ideas presented in lines 42-50, but (at this point in the reading) wonder about computational complexity and inference time. 

I enjoy the qualitative thinking and "logic" presented in figure 1; this kindof thing can be quite valuable and I think it helps with readability. 

To make the paper more clear and self-contained, the authors should define the important metrics of ADE and FDE. Furthermore, I assume the authors are using the same prediction horizon (12 future steps, given 9 past steps) as the baselines? I might have missed these details and apologize if I have, but this was not obvious from the writing. 

Typo in line 142, "aggregateiton"

This is somehwat of a philophical question, but if some type of oracle has access to all information (i.e. agent 1 has access to n2 and n3 centric data), why not just create a weighted graph in the very beginning that represents a global, properly scaled set of relative spatial information (for example, edges are simply weighted by relatve distance, bearing, heading, etc)? 

INTERACTION and TrajNet++ are perfectly fine datasets. However, there is one issue in the sense that (as the authors of TrajNet++ point out), not all of the relevant methods have even been evaluated on a consistent baseline. Those authors do well to be quite comprehensive, but I am not sure the results in your paper, e.g. in Table 2, represent a comprehensive comparison. There are many other methods besides Social LSTM (there is a typo in Table 2, by the way) and D-LSTM that have achieved good performance on data sets outside of TrajNet++, and furthermore, neither of these methods are explicitly based on GNNs. The suggestion here is to justify the choices of both the data used, as well as the baselines for comparison, so that this is all more rigorous and comprehensive. I recognize that this is nearly impossible (as the sparsity of table 1 indicates), but this could still be improved. 

The authors seek to make a contribution to GNN-based methods of multi-agent prediction. As the authors acknowledge later, in section 6, not all methods are abased on on GNNs, or at least not explicitly. Social LSTM and GAN are such examples, as are several more recent works (I am trying to say that GNNs are not the only currently hot method of doing this). The authors would do well to justify the motivation for focusing on graph formulations, particularly in section 1 or 2, rather than having the reader wait (and try to infer this) based on section 6.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper considers the joint motion prediction of agents in a multi-agent system, motivated mostly by the prediction of pedestrians that could support safer autonomous driving vehicles. In contrast to fully decoupled methods, which propose to predict each moving agent (pedestrian) separately in a system, allowing great scalability at the cost of joint reasoning, this work proposes a novel combined ego- and allo-centric way of jointly predicting the motion of many pedestrians while keeping computation low. This work is demonstrated on a standard prediction benchmarking dataset against other SOTA methods, including other entries of an international competition, where it ranked first in two categories. Finally, many sub-variants of the work are presented and discussed, as a form of ablation study, to highlight the necessary features of the best method proposed.
strengths_and_weaknesses: First of all, this paper reads rather well, with a clear goal, motivation, and premise. The paper is well structured, although I was a little taken aback by the prior work section being second to last: as a reader, I would have preferred to be given the prior work context before reading the rest of the paper so I could better situate this work. This is not a big deal, but I strongly recommend authors consider placing the prior work just after the introduction, as is very common in robotics/engineering papers.
One note that I have to make, is that there is an overwhelming number of small typos in this paper, which are very easy to miss but that I didn't, and which did impact my reading. I really recommend authors do a proper spellcheck of their work, as this is not OK for a published paper. I will provide a few below in my minor comments section, but I really believe this is easy to fix and should have been done before the initial submission.

Now, for more major issues, I do not have any concerns with the actual method described in this work, and I actually loved the idea of mixing egocentric and allocentric views to provide a symmetric, yet nuanced view of the world and improve predictions. The use of a GNN and transformers to allow scalability is smart, and was shown to lead to much better generalization (as evidenced by the ranking to that particular track of the challenge).
My only main issue would be with the presentation of most of the results in the paper: All tables have absolutely NO UNITS, making it impossible for the reader to understand anything about what they report. What are the numbers reported? Is higher/lower better? How are those results obtained, and what do they truly mean?
As a reader, I feel like I was told a beautiful story about a great new idea, but when it came to quantitative results I have no way of telling anybody anything about what performance I would expect from this method, and to why. This is not good, and should be expanded:
1. Authors need to explain how models are tested, and what metrics are reported (and higher/lower is better) before Table 1. This needs only be done once, as I guess all tables report values of the same type (again, this can only be a guess...).
2. There might be a few too many tables, and instead of one or two of them, I would have loved to see the actual results in some test cases. Can you show the actual ground truth of the pedestrians' motion, and then a comparison between your approach's predictions and that of other benchmarks? I loved the story about how decoupled prediction can affect prediction greatly, especially when pedestrians have strong inter-relations, and I would like to see that in some of the (visual) results.
3. I also worry that the second part of the paper is a little too lengthy for what it truly brings to the reader. There is a large discussion of complexity, and of those many variants of the proposed model, and I am left wondering if this really brought me a lot. Can this kind of "ablation study" be summarized in a more concise manner? In particular, do we really care about computational complexity for this work, i.e., is this really the main limitation that should drive the choice of variant of the model? I did see some differences in the tables, when the runtimes are discussed, but again due to the tables being near-impossibleto read I could not really appreciate why runtime/complexity was such a key point that would justify nearly 4 pages being dedicated to variants and complexity discussion.

Minor points:
1. Section 5.1, line 151: What does the "abundance" mean? This does not appear to be a standard term in the field (or any field I am familiar with or could google about), is this a typo/mistranslation or am I missing anything? If this is a standard term, please provide a citation or footnote to explain it, otherwise please correct this term.
2. Short list of some of the typos/grammatical mistakes I caught. Again, please perform a thorough grammatical/spell check, as the state of the text was really poor for a first submission:
	- line 30: "one for each agent" --> once for each agent
	- line 109: textbf1st place --> fix the missing backslash (latex typo)
	- line 119: modelOn --> missing period between end and start of two sentences.
	- Table 3 caption: Comparsion --> Comparison
	- line 142: Aggregateiton  --> Aggregation
	- Figure 4 caption: vairant --> variant
	- line 181-182: impracticable --> impractical?
	- line 200: AA performances the worst --> not an English sentence, this seems to be missing a verb
summary_of_recommendation: In summary, this work proposed a novel and intriguing new method to perform mostly-coupled, yet computationally tractable multi-agent motion prediction, by allowing agents to combine their view with that of other agents in the system. The paper reads well (minus some typos that can be easily fixed), and results seem strong (although presentation needs to be improved there as well). The use of the 8 pages is not optimal yet, with some sections needing to be made more concise so more visual results comparison can be added, but I believe this paper has a lot of potential. I am in favor of accepting this work for presentation at CoRL 2021, should my major and minor points be addressed.
issues: Issues were listed above in my main review (major and minor ones).
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents an efficient multi agent prediction framework that treats multiple agents as a unit, rather than as a composition of multiple single-agent predictions. Within the presented framework, the authors address problems with current methods for normalization and processing by considering all agents' information symmetrically considering both egocentric and allocentric perspectives.
strengths_and_weaknesses: The paper was very clear and easy to understand. The problem was also well motivated in the problem formulation. The entire contribution was the network architecture, and I do believe that there could be more mathematical rigor used to explain the choices of the authors. The results were presented clearly- notably the efficiency analysis.
summary_of_recommendation: My main issue with the paper was that it needed more mathematical rigor in explaining some of the choices made about the specific model architecture, though I did appreciate the rigor in the section about efficiency. I think the organization is good and overall the idea of the paper is good. More explanation on why the model was unique would have assisted in the originality rating as well. The results overall show improvement and good results, so that's why it could be accepted.
issues: The main issue is that there is some mathematical rigor missing in the approach (model) section. This further inform why the model was constructed the way it was, and would improve the overall clarity and contribution of the work.
review_title: null

Meta-review: This paper proposes a novel method for multi-agent prediction. While the reviewers are generally positive about this paper, they also raise a number of important concerns (in particular reviewer ddzJ). The authors should carefully address the reviewers' comments in their rebuttal. Also, I would encourage the authors to compare their work with this paper: https://arxiv.org/pdf/2007.12036.pdf

UPDATE POST DISCUSSION PHASE: I would like to thank the authors for their comments and clarifications during the discussion phase. This paper was extensively discussed by the reviewers and the Area Chair. In general, there is agreement that this paper provides a useful contribution. Yet, concerns persist regarding a number of aspects of this paper, in particular regarding the rigor and accuracy of some of the technical claims. The decision is to accept this paper, but I urge that authors to carefully address all reviewers' comments in the final version of this paper.
Decision: Accept (Poster)