Multi-Agent Trajectory Prediction by Combining Egocentric and Allocentric Views
Authors: Xiaosong Jia
Liting Sun
Hang Zhao
Masayoshi Tomizuka
Wei Zhan
Venue: CoRL2021 Poster (2021)
Keywords: Autonomous Driving
Joint Trajectory Prediction
Multi-Agent Interaction

Abstract: Trajectory prediction of road participants such as vehicles and pedestrians is crucial for autonomous driving. Recently, graph neural network (GNN) is widely adopted to capture the social interactions among the agents. Many GNN-based models formulate the prediction task as a single-agent prediction problem where multiple inference is needed for multi-agent prediction (which is common in practice), which leads to fundamental inconsistency in terms of homotopy as well as inefficiency for the memory and time. Moreover, even for models that do perform joint prediction, typically one centric agent is selected and all other agents’ information is normalized based on that. Such centric-only normalization leads to asymmetric encoding of different agents in GNN, which might harm its performance.  In this work, we propose a efficient multi-agent prediction framework that can predict all agents' trajectories jointly by normalizing and processing all agents' information symmetrically and homogeneously with combined egocentirc and allocentric views. Experiments are conducted on two interaction-rich behavior datasets: INTERACTION (vehicles) and TrajNet++ (pedestrian). The results show that the proposed framework can significantly boost the inference speed of the GNN-based model for multi-agent prediction and achieve better performance. In the INTERACTION dataset's challenge, the proposed model achieved the 1st place in the regular track and generalization track.

Multi-Agent Trajectory Prediction by Combining Egocentric and Allocentric Views
Xiaosong Jia', Liting Sun", Hang Zhao', Masayoshi Tomizuka", and Wei Zhan
Shanghai Jiao Tong University jiaxiaosong 1997 @ (emi@gmall.oooo
University of California, Berkeley bretinsunwwzzann @ berkeley.edu tomizuka @ me. me.berkeley.edu TTsinghua University zhaohang01 24 @4@ggmll..comm gmail.com
Trajectory prediction of road participants such as vehicles and pedes- trians is crucial for autonomous driving. Recently, graph neural network (GNN) is widely adopted to capture the social interactions among the agents. Many GNN- based models formulate the prediction task as a single-agent prediction problem where multiple inference is needed for multi-agent prediction (which common in practice), which leads to fundamental inconsistency in terms of homotopy as well as inefficiency for the memory and time. Moreover, even for models that do perform joint prediction, typically one centric agent is selected and all other agents' information is normalized based on that. Such centric-only normalization leads to asymmetric encoding of different agents in GNN, which might harm its performance. In this work, we propose a efficient multi-agent prediction frame- work that can predict all agents' trajectories jointly by normalizing and processing all agents' information symmetrically and homogeneously with combined egocen- tire and allocentric views. Experiments are conducted on two interaction-rich be- havior datasets: INTERACTION (vehicles) and TrajNet++ (pedestrian). The re- sults show that the proposed framework can significantly boost the inference speed the GNN-based model for multi-agent prediction and achieve better perfor- mance. In the INTERACTION dataset's challenge. the proposed model achieved the 1st place in the regular track and generalization track.
Abstract:
Keywords: Autonomous Driving, Joint Trajectory Prediction, Multi-Agent Inter- action
1 Introduction
Trajectory prediction of road participants like vehicles and pedestrians is of great significance for planning and decision making of autonomous vehicles. Recently, lots of research efforts have been devoted to capture the social interactions among the agents with feature pooling [1, 2] graph neu- ral network [3, 4, 5]. However, most of the state-of-the-an (SOTA) models [5, 6, 7] formulate the problem single-agent prediction, i.c., the models try to learn the marginal distribution of what a target agent may act in the future given the historical observations of all agents' trajectories in the scene. In cases of multiple agent prediction, straightforward way is to run inference multiple times [8], that is, onece for each agent. Under such circumstances, the "joint" prediction results might not be consistent in terms of homotopy, not to mention the low efficiency in terms of memory and time. There are also models which perform joint prediction for multiple agents [9, 10]. How- ever, such work adopts uniform coordinate for all modeled agents, i.e., a centric agent is selected and coordinates are normalized based on the centric agent. Therefore, only the selected agent is
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Observation
Relative Thinking
1st Egocentric View
Allocentric View
Egocentric View
How far are other cars from me?
After thinking what car 2 and may do, what should I do?
How fast are they?
3
should do?
(ego)
If were car 2, what
(ego)
If were car 3, what would do?
(ego)
Figure Illustration of the intuition for each stage of the proposed model the thinking process of interacting agents. Here, we only show the first agent's (blue) thinking process since the other two are symmetrical. First, human driver understands the world by estimating the relative dis- tance and speed of other agents (Relative Thinking), which corresponds to the normalization stage of the proposed model. Then each agent considers how it should react (Egocentric View), which corresponds to the stage of updating the node features by aggregating neighbors' information in the proposed model. After that, an agent put itself in someone else's 'ss shoes for all neighbors (Allocentric View), which corresponds to the update of edge features by accessing source node's 's information in the proposed model. After alternatively repeating the Egocentric View stage and Allocentric View stage for several rounds, all agents make the decision according the interacting situations.
providing egocentric views, while other agents are allocentric views.! The disadvantage of all data in single egocentric view is two-fold. First, not all agents treated in unbiased way due to the lack of egocentric views from other agents, like missing the view of "putting in someone else' shoes". Second, the egocentric view of agent makes the spatial relations between two other agents heterogeneous from the spatial relations between ego agent and other agents. The different distributions of input node feature might harm the GNN-based models' performance since usually shares parameter for the aggregation and fusion function for all nodes.
In this work, we aim to design an efficient GNN- multi-agent joint prediction model. Our key insight that all agents should treated symmetrically in the graph, providing both egocentric and allocentric views. Moreover, we decouple the update of the node and edge feature to improve the efficiency of the network. As a result, we propose to use the "Node update" step to collect and summarize other agents' information in egocentric view, and then the "Edge update" step to collect information in allocentric views from each neighbor. Via such a combined egocentric-and- allocentric views, it is guaranteed that the graph model handles all agents homogeneously which will encourage the shared parameters to learn to extract the underlying interactions among agents. We illustrate the intuition behind the proposed model in Fig. 1
We conducted experiments on two interaction-rich datasets: INTERACTION (vehicles) [12] and TrajNet++ (pedestrian) [13]. The results show that compared to running multiple single prediction models, the efficiency boosted 1.7-4.5 times higher while achieving better accuracy. the IN-
'Egocentric and allocentric are terms usually used in the psychology and cognition area. According to [11], study on human spatial cognition and animal navigation, they define the egacentric spatial coding system the location of objects in space relative to ego body and the allocentric spatial coding system as location of one object is defined with respect to other objects. In this paper, we refer this two concepts. As for the coordinate system transformation, we set the reference's 's coordinate as the origin and yaw angle aligned with the negative x-axis.

---

TERACTION dataset's challenge, the model achieved the 1st place in the regular track and general- ization track We have also conducted experiments on three efficient variants of the multi-inferrnce pngleeprediction model under different view settings and the results show that the proposed model outperforms those variants, which demonstrates the advantage of combining the egocentric and al- locentric views.
2 Related Works
Plenty of works have been done to capture the complex social interactions among multiple agents. There are pooling- posseng-based methods [1, 2] which use the mean/max/sum of all agents' hidden vectors to obtain a single vector to represent the social context. Social LSTM [1] utilized the social con- text of each time step as a part of the input for its next time step to the LSTM. Social GAN [2] obtained the social context by the pooling of output from the encoder RNN the last time-step and fed the social context with each agents' hidden vector to the decoder generate their future trajectories. Though pooling is fast, it loses too much information and lacks of expressive ability to output specific context for each agent. There are also methods which renders the entire scene including the agents and HD Map onto images and captures the interactions implicitly by CNN. Hong et al. [14] adopted pure CNN to process the scene while Multipath [6] further set anchors to output multi-modal prediction and IntentNet [15] set a different motion predictor for different pre- defined intents. PRECOG [9] proposed flow-based generative models conditioned
goals with CNN to process context. However, as pointed [4], those CNN-based methods' performances are significantly influenced by the receptive field, feature cropping strategy, and image resolution and it is very computationally expensive and inefficient compared to directly process the coordinate data. TPCN [7] conducted trajectory prediction from point-cloud perspective which is an interesting ex- ploration across fields. Recently, GNN-based [3, 16, 17, 4, 18, 6, 19, 5] has gained great success and dominated most of the trajectory prediction datasets. Many of them [16. 6, 19, 5] set an ego agent and used it to normalize data. In the concurrently released Waymo Open Motion Dataset's paper [8], they implemented TNT [6] by the forward-N-timet way to do ageet prediction. In ILVM[[0], they proposed an end-to-end joint prediction framework, in which they first used CNN to extract features from LiDAR point clouds and raster map, used GNN capture interactions, and used novel one stage parallel sampling to do joint prediction. As for normalization, they used Rotated Region of Interest Align[21], which is specificlly designed for LiDAR point clouds/images to align. In the concurrent work Scene Transformer [22], they used Transformer for both spatial and temporal dimension, adopted mask strategies and done joint prediction. They normalized data simply by the ego agent.
3 Problem Formulation
Suppose there are N agents in the same scene (they could be either vehicles pedestrians), and our prediction task is to learn the joint distribution of their future trajectories conditioned on their historical states. Let us assume that the current time step is t 0. For each agent i = {1, NN. we have an observation of his/her states (such as coordinate, velocity, yaw angle.... from
Lo
t t= 11 We denote the joint observation as X = [X,, X2, ....XX Xx) where X, = {x 1 x.) Let us also define the future coordinates for all N agents from t = to t = T as Y {Y1 Y2 .... YN where Y; .y..... y,') Then, the joint prediction model is designed to learn the following conditional distribution:
model is approximating a marginal distribution defined as P(Y,, Y2,.,
XX.
(1)
On the other hand, the follows:
X Xg. where agent is the ego agent.
(2)
In the cases of multiple-agent prediction, essentially, multiple-inference is using multiple marginal distributions to approximate the joint distribution by implicitly assuming the multiple marginal dis- tributions are conditionally independent [8]:
N
p(Y: Yz.... YN (1, X2. ..
II ,Y,,|X,, Xy....NN)
(3)
'The generalization track is testing the model's performance new maps/scenarios which have never been seen in the training time.
3

---

Only Forward
Raw Trajectory
PA
Normalization
n1-centric
n2-centric
n3-centric
Normalized Data
Aggregation Layer
Edge MLP
Aggregation Layer
Edge
33
32
Prediction Layer
Trajectory
Trajectory
Trajectory
Figure 2: The structure of the proposed model: we use 3 agents as an example. R22, means raw data. The superscript means the data after normalization. Eij means the normalized data of agent i by the origin and orientation of agent j. First, all agents are encoded in their egocentric views in the normalization step; second, the node features (Aggregation Layer) are updated by the egocentric information separately for different agents; and finally, the edge features (Edge MLP) are updated using allocentric views, summarizing local information from all agents.²
Unfortunately, such assumption is not true in practice, particularly when there are intensive interac- tions among the N agents.
The Model
An overview of the model is shown in Fig. 2. The model has the following three key modules:
View Transformation (Normalization Layer): All agents are encoded symmetrically in the graph. In the normalization layer, we set each agent in his/her own egocentric view instead of selecting one centric agent and normalize all other agents based on Specifically, for the coor- dinate system of each node (agent) and its in-edges' (neighbors), we use the agents" coordinate at to (w.r.o.)) as the origin and let the yaw angle be aligned with the negative x-axis. Since the input node and edge features are temporal sequences, use temporal sequence encoder (ResNet-18 ID version+MLP) to encode the sequence into a feature vector.
Node update (Aggregation Layer): Update each node feature according to their in-edges' fea- tures to incorporate neighbor's information hτ = f(thzi Neighbor(i))) where hÇ is the updated node feature of agent i, hys the feature of in-edge from agent j to i. f the aggregate func- tion and it is multi-head attention in Transformer. The Transformer is proposed in [23] which is permutation- invariant -nntaaiaainnn operator and could capture the relations among a of elements. Since all agents in the same scene are and it is not our main contribution, for more details about Transformer, please refer to [23]. Note that for conciseness we added self-loop for each node which means each nodes' neighbors include itself. Since the input node feature and input in-edge features of each agent are both normalized in each agent' coordinate system, it in the egocentric view.
we directly adopts it. Due to the limited space
'Note that the Edge MLP in the last layer could be removed to reduce computation (in this case Edge
MLP 2)

---

Edge update (Edge MLP): Update each edge by their source node's feature to obtain updated neighbors' information K' g(hje, hi)) where we denote the amnaatenttonnFFeeoorrvaadd in Transformer as g. Since the new information is coming from the source node the neighbor in the neighbor's 's view, is in the allocentric view.
We would like to emphasize the benefit of decoupling the node update step and the edge update step. By decoupling, the node serve as the storage for the interaction information between each agent and its neighbors in the egocentric view. Via such design, the edge feature can be efficiently updated to collect and summarize the allocentric information. Hence, such a combined egocentric- and-allocenn alleentocc encoding and feature updating can help to 1) achieve an unbiased encoding of all agents for joint prediction, and 2) improve the efficiency of the computing process.
5
Experiements
5.1 Metrics
As for the metric, adopt the widely used ADE and FDE. They are defined Average Dis- placement Error (ADE) represents the euclidean distance averaged over time and agents between the ground truth and the prediction. It is then averaged over all cases. The ADE of single case is calculated as:
1 ADE
Σ
(Bnm n.
)n,,)) -xn,)) + (Du.t yn,,)
NT
where N is the number of target agents in the case, T ist the prediction horizon, and á represent the groud truth coordinate, and y represent the predicted coordinate. The unit is meter.
Final Displacement Error (FDE) represents the euclidean distance averaged over agents between the ground truth and the prediction at the last predicted time-step. It is then averaged over all cases. The FDE of single case is calculated as:
FDE
1 N
Σ
V
(En.T I2 + (Jn.T Vn,T)²
(5)
where the number of target agents the case, T ist the prediction horizon, and ý represent the groud truth coordinate, and y represent the predicted coordinate. The unit is meter.
5.2 Results
We tested the performance of the proposed model against several SOTA models on two interaction- rich dataset: the INTERACTION dataset and the TrajNet++. Please refer to for experimental setting details.
In INTERACTION dataset [12], we compared with DESIRE [24], MultiPath [25], TNT [6]. and ReCog [26]. The former three are compared [6]. We also listed the results the top-rank methods on the leader board of the INTERACTION dataset. As the baseline methods, especially on the generalization ability track. We achieved the a place on both the regular test track and generalization ability track of the INTERACTION dataset.
Tab. 1, our method outperforms
In the TrajNet++ [13], existing results in this dataset are all measured only by the performance on the pre-defined 'interesting' primary pedestrian ignoring other agents. Thus, for fair comparisons, we re-implemented the two top-ranked methods under our multi-agent prediction setting: 1. So- SialL.STM [1], typical pooling-based method and the top-ranked method on TrajNet++'s leader board, a good comparison for our GNN-based method. D-L.STM [13], which was proposed in the TrajNet++ dataset paper and achieved SOTA performance in their paper, captures social interac- tions by concatenating top-4 nearest neighbors' hidden states. Note that [13] stated in their paper that predicting pedestrian trajectories other than the primary pedestrian's 's would dampen their results which aligns well with ourfollowing for the single-view model. On the other hand, the GNN-based methods should be further improved to tackle high-density data like CFF dataset, which can be han- dled by pooling- bosengg-based or top-k-neighbon complexity.
ke-eighhor.cooncteeatteie concatenation based method due to the low computational
5

---

Table 1: Comparison with SOTA methods on the INTERACTION Dataset. means the results are not publicly available. means the model is trained by smaller gap between snippets which means more data, which proposed by the challenge winner ReCog [26]... The unit of ADE and FDE are both meters. + means the lower the better.
Val ADF/FDEL
Test ADF/FDEred
Generalization ADE/EDE
DESTRE [24]
0.32/0.88
MultiPath [25]
0.30/0.99
TNT [6]
0.21/0.67
MIFNet
0.1973/0.664
0.5339/1.4248 (2"d))
ReCog* [26]
0.1919/0.6462
0.1878/0.6381 (3rd)
0.5539/1.9187
Mix
0. 0.1826(1** " /0.66423 (2nd)
Ours
0.1723/0.5983
0.1903/0.6563
0.3394/1.1955
Ours
0.1700/0.5927
0.1 1,184022" (1**)
0.3263/1.1422 (1*t)
Table 2: Comparison with top-ranked methods the TrajNet++ dataset under our multi-agent prediction setting. The unit of ADE and FDE are both meters. + means the lower the better.
Val ADE/FDE
Social LSTM
0,2848/0.5604
D-LSTM [13]
0.291570.5738 0.2079/0.4270
Ours
6 Comparison with Variants of the Proposed Model
In this section, we compare the the proposed model against forwarding the Lingle-prediction models N times (FNT) and its three variants in temrs of running speed and performance. By theoretical analysis and experimental results, we will show that the proposed model could achieve the best efficiency and accuracy among several settings of GNN for the multi-agent prediction.
6.1 Baseline- forwarding N times (FNT)
For a fully connected graphs with N agents, there are N(N-1) edges, which means each GNN layer N(N-1)
2
has 2 message passing and N updating. Thus, the computational complexity of L layers of GNN in one forward is L (O( ((N-1) + O(N)) O(N2) since L is hyperparameter and could treat it as a constant. As a result, in Equ. 6, we derive the computational complexity of FNT with GNN-based model on the fully-connected graph.
O(N)
O(N2)
= (
Number of Forwards
Complexity of GNN on Fully Connected Graph
In contrast, as shown Fig. 2, for the proposed model, the computational complexity of the two consecutive steps (Aggregation Layer and Edge MLP) both are O(N2) which makes the overall complexity O(N2).
Represent FNT by a single graph: The incorporation of neighbors' information in GNN essen- tially is: each target node obtains source nodes' information through their in-edges. Thus, FNT is equivalent to: each agent transforms its own states and others' states into its ego-centric coordinate system by normalization. Its own transformed states could be seen as a node while others' trans- formed states could be seen as its in-edges. Then, for each agent, maintains and updates both its node feature and in-edge features in GNN. Finally, it uses its node feature do the trajectory prediction. Fig. 3a gives the illustration of above description.
Here, conduct comparison experiments between FNT and the proposed methods. Experiments results are in Tab. 3. As we can see, compared to FNT, our method could significantly reduce the computational time while achieving better results.
Table 3: Comparison between the proposed method and forward-N-time method. The unit of ADE and FDE are both the meter. The unit of Train/Val time is the second. means the lower the better.
INTERACTION
TrajNet++
Val ADE/FDEL
Train/Val Time +
Val ADE/FDE 1
Train/Val Time
#Parameters 579991
FNT Ours
0.1840/0.6231
1261s/50s
0,2328/0.4659
762s/23s
0.1723/0.5988
713S/27s
0.2079/0.4270
1758/10s
678807

---

Only Forward
Trajectory
Normalization
preset-centr
Row Trijectory
Normalized Data
Mormalization on
nihoertrc
Forward
Forward
Forward
Layer
Normalized Data
Trspeckery
Layer
Nermakiedd Data
Eu
ONN Layer
GNN
Layer
Prodiction Loger
Trajectory
Trajestory 2
Trajectory 3
Happooo
Trajectory
Trajectory
Trajectory
(a) Illustration of computational pro- cess of forward-N-tine (FNT).
(b) Illustration of computational process of one-layer GNN (OL).
(e) Illustration of computational process of average of all agents (AA) and randomly selection (RA).
Figure 3:
6.2 Variant I stop maintaining the edge feature
After finding out the abundance, we explore some other variants which could reduce the computa- tional complexity of multi-agent prediction to (n²²) like the the proposed model and evaluate their precision.
Since the abundance is on the edges, the first intuitive variant is: how about we stopping main- taining the edge feature and only update the node feature? In this variant, there is no multi-hop or multi-order information included. For example, for agent 1, it could not obtain the interaction information between agent and 3. Thus, this variant is equivalent to the one-layer GNN which only updates the node. We denote it OL (one-layer). The illustration is in Fig. 3b. From the figure, we can obtain the computational complexity of OL in Equ. 7.
((N)
((N)
O(N2)
Number of Forwards
Ego Agnet Uses Its Neighbors' Features
However, there might be two problems in one-layer GNN: 1: limited expressive power and hard capture complex interactions. 2: for non-fully connected graph, lack of multi-hop neighbors' information.
Validation Experiments
Here, we compare the results GNN with one layer and two layers to evaluate the OL's influence on the models' performance. We compare the following settings:
FNT: layers GNN + fully-comnected graph
OL: layer GNN fully-connected graph
FNT-Graph: 2 layers GNN connected by distance threshold graph
OL-Graph: layer GNN + connected by distance threshold graph
For more experiment details, please refer Appendix.
The results are in Tab. 4. We can draw the conclusions that 1. using fully-connected graph could obtain better results. Thus, we use fully-connected graph for the rest of experiments in the paper. 2. OL consistently performs worse than FNT.
In conclusion, though not maintaining the edge feature could reduce the computational com- plexity, it loses much expressive power and high-order information which makes it impractical.
6.3 Variant II forwarding only once
Another option to reduce the computation is to only forward once, which means the model uses all nodes in the graph to do prediction instead of just using the ego node. As result, the total computational complexity is reduced to the computational complexity of GNN O(N2).
7

---

Table Validation experiments for the one-layer GNN(OL) refinement. The unit of ADE and FDE are both the meter. The unit of Train/ Val time is the second. J means the lower the better.
ADE/FDE
INTERACTION Val TrajNet++ Val 4
#Parameters
FNT
0. 0.1840/0.6231
0.2328/0,4659
579991
OL FNT-Graph OL-Graph
0. 0.1908/0.6379
0.2337/0
481303
0.1909/0.6704
0.2334/0.4693
579991
0.1991/0.6864
0.2343/0.4805
481303
Table Validation experiments for the Average of Agents(AA) and Random Agent (RA) plan. The unit of ADE and FDE are both the meter. The unit of Train/Val time is the second. means the lower the better.
INTERACTION
Net++
Val ADE/FDE + Train/Val Time +
ADE/DE
Train/Val Time
#Parameters
FNT
0.1840/0.623
1261s/50s
0.2328/0.4653
762s/23s
579991
RA
0. 111938006474
1308s/51s
0.2483/0.4833
776s/24s
579991
AA
0.2187/0.7189
642s/21s
0.2382/0.4762
141s/6s
579991
However, another problem arises: how to normalize data without setting an ego agent? The following two intuitive variants:
Average of Agents (AA): we take the average coordinate and yaw angle of all agents at lo as the origin and the orientation of the positive x-axis respectively.
Random Agent (RA): we randomly select an agent as the ego agent to do the normalization. Note that though it is faster in inference in theory, in training/valldationn, input all possible ego vehicle choices, which means no speed up. In fact, compared to FNT, it is equivalent to adding an auxiliary task of trajectory prediction for non-ego agents.
still need to
The illustration of AA/RA is in Fig, 3c. We compare the performance of RA and AA to the original forward -Ntimmess (FNT) in Tab. We also record the Train /al time which is measured by the time to run epoch the train/val set one RTX 2080 Ti with all GPU memory.
From the results, we can conclude that: Forward only one time could significantly reduce the computational time. 2. However, the two alternative normalization methods dampen the perfor- mance. 3. AA performs the worst. We conjecture that AA too sensitive since it is influenced by all agents in the scene and it cannot effectively reduce the input and output space. 4. TrajNet++ (pedestrian) dataset was less influenced. We conjecture that due to the smaller range of scenes for pedestrians.
In conclusion, forward only once is effeetive for decreasing the computational complexity while a more stable normalization strategy is needed to maintain the performance. Thus, as demon- strated before, in the proposed model, all agents be treated symmetrically and homogeneously is significant for the performance of the model.
Conclusion
In this paper, presented multi-agent prediction model which combines the egocentric and al- locentric views. It achieves SOTA performances on the two interaction-rich behavior datasets: IN- TERACTION (vehicles) and TrajNet++ (pedestrian). In the INTERACTION dataset's challenge, the model ranked the first place both the regular track and generalization track. Besides, com- pared to the common multi-inference iinle-predicttin model with similar architecture, the model could increase the running speed by 1.7-4.5 times.
In this paper, the Transformer and the Edge MLP only are chosen implementation to show the effectiveness of combining the egocentric and allocentric views. For future work, more specific designed structure could be explored. For example, to further reduce the computational complexity, some efficient variants of the Transformer could explored. To better capture the relations of two views, more complicated structure could be tried instead of the Edge MLP (maybe another Transformer). Additionally, we might extend to incorporate the map information and do multi- modal prediction.

---

References
[1] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
[2] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2255-2264, 2018. doi: 109/CV/R.20188.0000 00240.
[3] A. Mohamed, K. Qian, M. Elhoseiny, and C. Claudel. Social-stgenn: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14424-14432, 2020.
[4] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11525-11533, 2020.
[5] W. Zeng, M. Liang, R. Liao, and R. Urtasun. Lanerenn: Distributed representations for graph- centrie motion forecasting, arXiv preprint arXiv:2101.06653, 2021.
[6] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Chai, C. Schmid, C. Li, and D. Anguelov. TNT: target-driven trajectory prediction. CoRR, abs/2008.08 088244 2020. URL https: //arri//xxiii .rrg/ /tw. 08294.
[7] M. Ye, T. Cao, and Q. Chen. Tpen: Temporal point cloud networks for motion forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13318-11321 June 2021.
[8] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp. C. Qi, Y. Zhou, al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. arXiv preprint arXiv:2104. 10133, 2021.
[9] N. Rhinehart, R. McAllister, K. Kitani, and S. Levine. Precog: Prediction conditioned on goals in visual multi-agent settings. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 2821-2830. 2019.
[10] Y. C. Tang and R. Salakhutdinov. Multiple futures prediction. 2019.
[11] R. Klatzky, Allocentric and egocentric spatial representations: Definitions, distinctions, and interconnections. In Spatial cognition, pages 1-17. Springer, 1998.
[12] W. Zhan, L. Sun, D. Wang, H. Shi, A. Clausse, M. Naumann, J. Kümmerle, H. Königshof,
C. Stiller, A. de La Fortelle, and M. Tomizuka. INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moOION Dataset in Interactive Driving Scenarios with Semantic Maps. arXiv:1910.03300 [cs, eess], 2019.
[13] P. Kothari, S. Kreiss, and A. Alahi. Human trajectory forecasting in crowds: A deep learning perspective. IEEE Transactions on Intelligent Transportation Systems, 2021.
[14] Hong, B. Sapp, and J. Philbin. Rules of the road: Predicting driving behavior with a con- volutional model of semantic interactions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8454-8462, 2019.
[15] S. Casas, W. Luo, and R. Urtasun. Intentnet: Learning to predict intention from raw sensor data. In Conference Robot Learning, pages 947-956. PMLR, 2018.
[16] Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Multi-agent genera- tive trajectory forecasting with heterogeneous data for control. ArXiv, abs/2001.03093 2020.
[17] J. Li, H. Ma, Z. Zhang, and M. Tomizuka. Social-wagdat: Interaction-aware trajectory pre- diction via wasserstein graph double-attention network. CORR, abs/2002.06241 2020. URI. https: / rxviv wwwpii....rraaaeeeee 2002..
06241. 26022.0611...........

---

[18] V. Kosaraju, A. Sadeghian, R. Martin,Martin, I. Reid, H. Rezatofighi, and S. Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention net- works. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlche Buc, E. Fox, and R. Gar- netr, editors, Advances Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL wttphttttss//pcceeeteeeeeo 0009 Pappee .ppf
neurierrrpp ps. cccccc///pe.. htc/paper/2219/1111
[19] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forecasting, In European Conference Computer Vision, pages 541-556. Springer, 2020.
[20] S. Casas, C. Gulino, S. Suo, K. Luo, R. Liao, and R. Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pages 624-641. Springer, 2020.
[21] Ma, Shao, H. Ye, L. Wang. Wang, Zheng, and X. Xue. Arbirrary-oriented scene text detection via rotation proposals. IEEE Transactions on Multimedia, 20(1 1:1111-3122, 2018.
[22] Ngiam, B. Caine, V. Vasudevan, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, et al. Scene transformer: A unified multi-task model for behavior prediction and planning. arXiv preprint arXiv:2106.08411 2021.
[23] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In NIPS, 2017.
[24] N. Lee, W. Choi, P. Vernaza, C. B. Choy, H. Torr, and M. Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 336-345,77 2017.
[25] Y. Chai, B. Sapp, M. Bansal, and D. Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 86-99, PMLR, 30 Oct-01 Nov 2020. URL http:// proceedings mlr. wwwpppesssvvee wttpss/vvv0/////// http//aaaa2...... cttess//voo/chaa2aa..f. html.
[26] A. Scibior, V. Lioutas, D. Reda, Bateni, and F. Wood, Imagining the road ahead: Multi-agent trajectory prediction via differentiable simulation. arXiv preprint arXiv:2104. //2/2/ 2021.
10


Reviews:
[{"review_id": "y11Kz-uk3sD", "reviewer": "Reviewer_Wu5R", "summary": "This paper considers the joint motion prediction of agents in a multi-agent system, motivated mostly by the prediction of pedestrians that could support safer autonomous driving vehicles. In contrast to fully decoupled methods, which propose to predict each moving agent (pedestrian) separately in a system, allowing great scalability at the cost of joint reasoning, this work proposes a novel combined ego- and allo-centric way of jointly predicting the motion of many pedestrians while keeping computation low. This work is demonstrated on a standard prediction benchmarking dataset against other SOTA methods, including other entries of an international competition, where it ranked first in two categories. Finally, many sub-variants of the work are presented and discussed, as a form of ablation study, to highlight the necessary features of the best method proposed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First of all, this paper reads rather well, with a clear goal, motivation, and premise. The paper is well structured, although I was a little taken aback by the prior work section being second to last: as a reader, I would have preferred to be given the prior work context before reading the rest of the paper so I could better situate this work. This is not a big deal, but I strongly recommend authors consider placing the prior work just after the introduction, as is very common in robotics/engineering papers.\nOne note that I have to make, is that there is an overwhelming number of small typos in this paper, which are very easy to miss but that I didn't, and which did impact my reading. I really recommend authors do a proper spellcheck of their work, as this is not OK for a published paper. I will provide a few below in my minor comments section, but I really believe this is easy to fix and should have been done before the initial submission.\n\nNow, for more major issues, I do not have any concerns with the actual method described in this work, and I actually loved the idea of mixing egocentric and allocentric views to provide a symmetric, yet nuanced view of the world and improve predictions. The use of a GNN and transformers to allow scalability is smart, and was shown to lead to much better generalization (as evidenced by the ranking to that particular track of the challenge).\nMy only main issue would be with the presentation of most of the results in the paper: All tables have absolutely NO UNITS, making it impossible for the reader to understand anything about what they report. What are the numbers reported? Is higher/lower better? How are those results obtained, and what do they truly mean?\nAs a reader, I feel like I was told a beautiful story about a great new idea, but when it came to quantitative results I have no way of telling anybody anything about what performance I would expect from this method, and to why. This is not good, and should be expanded:\n1. Authors need to explain how models are tested, and what metrics are reported (and higher/lower is better) before Table 1. This needs only be done once, as I guess all tables report values of the same type (again, this can only be a guess...).\n2. There might be a few too many tables, and instead of one or two of them, I would have loved to see the actual results in some test cases. Can you show the actual ground truth of the pedestrians' motion, and then a comparison between your approach's predictions and that of other benchmarks? I loved the story about how decoupled prediction can affect prediction greatly, especially when pedestrians have strong inter-relations, and I would like to see that in some of the (visual) results.\n3. I also worry that the second part of the paper is a little too lengthy for what it truly brings to the reader. There is a large discussion of complexity, and of those many variants of the proposed model, and I am left wondering if this really brought me a lot. Can this kind of \"ablation study\" be summarized in a more concise manner? In particular, do we really care about computational complexity for this work, i.e., is this really the main limitation that should drive the choice of variant of the model? I did see some differences in the tables, when the runtimes are discussed, but again due to the tables being near-impossibleto read I could not really appreciate why runtime/complexity was such a key point that would justify nearly 4 pages being dedicated to variants and complexity discussion.\n\nMinor points:\n1. Section 5.1, line 151: What does the \"abundance\" mean? This does not appear to be a standard term in the field (or any field I am familiar with or could google about), is this a typo/mistranslation or am I missing anything? If this is a standard term, please provide a citation or footnote to explain it, otherwise please correct this term.\n2. Short list of some of the typos/grammatical mistakes I caught. Again, please perform a thorough grammatical/spell check, as the state of the text was really poor for a first submission:\n\t- line 30: \"one for each agent\" --> once for each agent\n\t- line 109: textbf1st place --> fix the missing backslash (latex typo)\n\t- line 119: modelOn --> missing period between end and start of two sentences.\n\t- Table 3 caption: Comparsion --> Comparison\n\t- line 142: Aggregateiton  --> Aggregation\n\t- Figure 4 caption: vairant --> variant\n\t- line 181-182: impracticable --> impractical?\n\t- line 200: AA performances the worst --> not an English sentence, this seems to be missing a verb", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In summary, this work proposed a novel and intriguing new method to perform mostly-coupled, yet computationally tractable multi-agent motion prediction, by allowing agents to combine their view with that of other agents in the system. The paper reads well (minus some typos that can be easily fixed), and results seem strong (although presentation needs to be improved there as well). The use of the 8 pages is not optimal yet, with some sections needing to be made more concise so more visual results comparison can be added, but I believe this paper has a lot of potential. I am in favor of accepting this work for presentation at CoRL 2021, should my major and minor points be addressed.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Issues were listed above in my main review (major and minor ones).", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "TgDk7wwRdM7", "reviewer": "Reviewer_ddzJ", "summary": "The paper presents an approach to predict trajectories of multiple agents. The paper attempts to motivate its model as capturing of \"Egocentric View\" and \"Allocentric View\", although the technical details on how interactions between such things could be captured is missing. A particular architecture is proposed without comparison to baselines or alternatives. The proposed model is a combination of Transformers and MLPs (on top of 1-D ResNet stems with different inputs).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Unclear\n\nWeaknesses\n\n- The 'model' presented in the paper lacks technical details and it is difficult to understand how the proposed model captures the properties regarding \"Egocentric View\" and \"Allocentric View\" mentioned in the introduction. We are not convinced that applying a series of these 'Aggregation Layers' and 'Edge MLP' are sufficient to capture what is claimed in the paper.\n\n- The technical contribution of the paper seems to be limited to a designed model combining existing Transformer layers and MLP. The technical originality is limited and incremental.\n\n- I think the key thing missing is the baseline experiments, ablation, and visualizations to convince that the proposed method captures egocentric and allocentric information well, and that it does it better than naive baselines such as simply concatenating all such egocentric and allocentric features in the beginning. The paper proposes a particular architecture, and we are not sure whether this is the best way to enable what's claimed compared to alternative baselines.\n\n- It is unclear how this paper is related to robot learning. The paper mentions that trajectory prediction in general is needed for autonomous driving, but it does not provide any details how it could realistically be used for the vehicles. INTERACTION Dataset used in the paper is composed of vehicle trajectories from a top-down view and TrajNet++ is based on pedestrian trajectories obtained from a top-down view, which are from very different views compared to the first-person view of an autonomous vehicle.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The mismatch between what the introduction is arguing and the model details makes it very difficult to understand what the paper is trying to achieve. What the paper needs to show is that the proposed method really captures properties claimed in the introduction experimentally, by comparing against different baselines and visualizing it. Further, we question whether its lacking connection to robot learning data really makes the paper sufficiently suitable for CoRL.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "The authors are encouraged to use robot data (or at least robot sim) and better provide technical details and justification behind the proposed approach. As mentioned above, baseline experiments and ablations to justify the claim will help greatly.\n\n\n**===================Updated review and suggestion after the discussion phase===================**\n\nI really would like to give the accept rating to this paper as it seems to have a certain contribution, but the concerns I have overweigh it. The discussion with the authors partially addressed some of the initial concerns but it simultaneously also added others.\n\nThe authors clarified that the paper is introducing the layer normalizing representations based on multiple views. The interaction with the authors also clarified that the rest of the approach is a Transformer application (and MLP) on top of them. This is one design choice adopting layers, and it is OK as it showcases the benefits of the normalization layer. The authors also have done some experimental comparison to the baselines and ablations, although it could have been against stronger approaches.\n\nThe concerns I have are as follows, which I believe should be addressed in order for the paper to be ready in a publishable form.\n\n- The paper should be more upfront about the fact that its \"aggregation layer\" exactly is a Transformer layer over the outputs of their normalization layer. The Transformer paper [26] is being cited only in Section 6, although the paper starts using Transformer much earlier in the paper. [26] is not even being introduced or discussed in the related work section, which will confuse and mislead the readers. The paper should introduce the Transformer paper properly early in the paper so that the readers can understand what it is, and the paper should explicitly point out how it relates to their approach.\n\n- The interaction with the authors revealed the stance that they want to treat any architecture different from theirs as an \"extension\" of theirs, which we believe is an overclaim and academically misleading. The paper is introducing one particular design (which is perfectly fine as mentioned above), not the best or optimal design. This has to be clarified in the final version.\n\n- Computational complexity advantage claims in the ablations need to be toned down, as the experimental comparison is only done with weaker baselines designed by the authors. There could be better alternatives with other efficient Transformers, which the paper has not explored.\n\n- The new table specifying the # of parameters of the models need to be added in the final version of the paper. This would allow the readers to judge themselves how many more parameters the proposed approach is using compared to the baselines to get a higher accuracy.\n\nOverall, the paper should be more upfront about what exact technical component it is adopting from the previous papers and where their computation advantage is coming from (i.e. Transformers)."}}, {"review_id": "BpLcYlEnla6", "reviewer": "Reviewer_zwfy", "summary": "This paper present a method for multi-agent prediction that is based on GNNs, but performs joint prediction by alternatively taking a local agent view as well as a view akin to \"how do the other agents view each other, and me?\" The paper is extremely well-written and intuitive, and this is improved by several visual aides, particularly figures 1 and 2. The method achieves good performance, both in terms of prediction and computational complexity. I encourage the authors to provide stronger justification for a few choices, as I try to detail in my comments below. \n\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper was very well written - I sincerely appreciate the effort to bring intuition into the discourse, which unfortunately is not the general case in academic literature. It was actually a delight to read.\n\nI have a few clarifying questions and comments in the \"Issues\" section below; otherwise, I do not have any substantive suggestions for improvement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My main reasons for this recommendation have to do with relevance, readability, and soundness of the approach. The paper is relevant in two ways: multi-agent prediction is important for myriad robotic tasks, and the focus on GNNs is a very active topic. While I am not 100% convinced that this is an extremely original idea, I do think the notion of taking multiple views in the multi-agent setting is interesting.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "I appreciate the early discussion about the terms \"egocentric\" and \"allocentric\", which will likely be quite helpful for the target audience at such a conference. \n\nI appreciate the discussion in lines 36-41. However, I am not sure this is a limitation of GNNs; i.e. if the distinction between egocentric vs allocentric views is as critical as the authors suggest. For example, can't an agent identify proper scales between distances (and other spatial relations) for other agents given current sensing modalities? Perhaps the authors could tighten up these claims. In context of vehicles, it could be that they are all sharing state information via V2X technology, in which case obtaining an unbiased graph representation seems feasible, if not straightforward. \n\nIntuitively, I like the ideas presented in lines 42-50, but (at this point in the reading) wonder about computational complexity and inference time. \n\nI enjoy the qualitative thinking and \"logic\" presented in figure 1; this kindof thing can be quite valuable and I think it helps with readability. \n\nTo make the paper more clear and self-contained, the authors should define the important metrics of ADE and FDE. Furthermore, I assume the authors are using the same prediction horizon (12 future steps, given 9 past steps) as the baselines? I might have missed these details and apologize if I have, but this was not obvious from the writing. \n\nTypo in line 142, \"aggregateiton\"\n\nThis is somehwat of a philophical question, but if some type of oracle has access to all information (i.e. agent 1 has access to n2 and n3 centric data), why not just create a weighted graph in the very beginning that represents a global, properly scaled set of relative spatial information (for example, edges are simply weighted by relatve distance, bearing, heading, etc)? \n\nINTERACTION and TrajNet++ are perfectly fine datasets. However, there is one issue in the sense that (as the authors of TrajNet++ point out), not all of the relevant methods have even been evaluated on a consistent baseline. Those authors do well to be quite comprehensive, but I am not sure the results in your paper, e.g. in Table 2, represent a comprehensive comparison. There are many other methods besides Social LSTM (there is a typo in Table 2, by the way) and D-LSTM that have achieved good performance on data sets outside of TrajNet++, and furthermore, neither of these methods are explicitly based on GNNs. The suggestion here is to justify the choices of both the data used, as well as the baselines for comparison, so that this is all more rigorous and comprehensive. I recognize that this is nearly impossible (as the sparsity of table 1 indicates), but this could still be improved. \n\nThe authors seek to make a contribution to GNN-based methods of multi-agent prediction. As the authors acknowledge later, in section 6, not all methods are abased on on GNNs, or at least not explicitly. Social LSTM and GAN are such examples, as are several more recent works (I am trying to say that GNNs are not the only currently hot method of doing this). The authors would do well to justify the motivation for focusing on graph formulations, particularly in section 1 or 2, rather than having the reader wait (and try to infer this) based on section 6. \n\n\n\n"}}, {"review_id": "2KHDcHYHBx_", "reviewer": "Reviewer_CH6w", "summary": "The paper presents an efficient multi agent prediction framework that treats multiple agents as a unit, rather than as a composition of multiple single-agent predictions. Within the presented framework, the authors address problems with current methods for normalization and processing by considering all agents' information symmetrically considering both egocentric and allocentric perspectives.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper was very clear and easy to understand. The problem was also well motivated in the problem formulation. The entire contribution was the network architecture, and I do believe that there could be more mathematical rigor used to explain the choices of the authors. The results were presented clearly- notably the efficiency analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My main issue with the paper was that it needed more mathematical rigor in explaining some of the choices made about the specific model architecture, though I did appreciate the rigor in the section about efficiency. I think the organization is good and overall the idea of the paper is good. More explanation on why the model was unique would have assisted in the originality rating as well. The results overall show improvement and good results, so that's why it could be accepted.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "The main issue is that there is some mathematical rigor missing in the approach (model) section. This further inform why the model was constructed the way it was, and would improve the overall clarity and contribution of the work."}}]

Author rebuttal:

Meta-review: This paper proposes a novel method for multi-agent prediction. While the reviewers are generally positive about this paper, they also raise a number of important concerns (in particular reviewer ddzJ). The authors should carefully address the reviewers' comments in their rebuttal. Also, I would encourage the authors to compare their work with this paper: https://arxiv.org/pdf/2007.12036.pdf

UPDATE POST DISCUSSION PHASE: I would like to thank the authors for their comments and clarifications during the discussion phase. This paper was extensively discussed by the reviewers and the Area Chair. In general, there is agreement that this paper provides a useful contribution. Yet, concerns persist regarding a number of aspects of this paper, in particular regarding the rigor and accuracy of some of the technical claims. The decision is to accept this paper, but I urge that authors to carefully address all reviewers' comments in the final version of this paper.
Decision: Accept (Poster)