Agent Prioritization with Interpretable Relation for Trajectory Prediction
Authors: Manh Huynh
Hengbo Ma
Gita Alaghband
Chiho Choi
Venue: Submitted to ICLR 2023 (2023)

Abstract: In this paper, we present a novel multi-agent trajectory prediction model, which discovers interpretable relations among agents and prioritize agent's motion. Different from existing approaches, our interpretable design is inspired by the fundamental navigation and motion functions of agent movements, which represent 'where' and 'how' the agents move in the scenes. Specifically, it generates the relation matrix, where each element indicates the motion impact from one to another. In addition, in highly interactive scenarios, one agent may implicitly gain higher priority to move, while the motion of other agents may be impacted by the prioritized agents with higher priority (e.g., a vehicle stopping or reducing its speed due to crossing pedestrians). Based on this intuition, we design a novel motion prioritization module to learn the agent motion priorities based on the inferred relation matrix. Then, a decoder is proposed to sequentially predict and iteratively update the future trajectories of each agent based on their priority orders and the learned relation structures. We first demonstrate the effectiveness of our prediction model on simulated Charged Particles dataset. Next, extensive evaluations are performed on commonly-used datasets for robot navigation, human-robot interactions, and autonomous agents: real-world NBA basketball and INTERACTION. Finally, we  show that the proposed model outperforms other state-of-the-art relation based methods, and is capable to infer interpretable, meaningful relations among agents.

Under review as a conference paper at ICLR 2023
AGENT PRIORITIZATION WITH INTERPRETABLE RE- LATION FOR TRAJECTORY PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we present a novel multi--gen, trajectory prediction model, which discovers interpretable relations among agents and prioritize agent's motion. Dif- ferent from existing approaches, our interpretable design inspired by the fun- damental navigation and motion functions of agent movements, which represent 'where' and 'how' the agents move in the scenes. Specifically, generates the relation matrix, where each element indicates the motion impact from one other. In addition, in highly interactive scenarios, one agent may implicitly gain higher priority move, while the motion of other agents may be impacted by the prioritized agents with higher priority (c.g., vehicle stopping or reducing its speed due crossing pedestrians). Based on this intuition, design novel motion prioritization module to learn the agent motion priorities based on the in- ferred relation matrix. Then, decoder proposed to sequentially predict and iteratively update the future trajectories of each agent based on their priority or- ders and the learned relation structures. We first demonstrate the effectiveness prediction model on simulated Charged Particles (Kipf al. 2018) dataset. Next, extensive evaluations are performed on commonly-us
datasets for robot navigation, human-roke robot interactions, and autonomous agents: real-world NBA basketball (Yue et al., 2014) and INTERACTION (Zhan et al. 2019). Finally, we show that the proposed model outperforms other state-of-the-ane art relation based methods, and is capable to infer interpretable, meaningful relations among agents.
INTRODUCTION
Impact
Past
Interaction Future Trajectory
Past Trajectory
Interpretable Relation
1 2 Prioritization
In-order
Trajectory
Encoding
Prediction
(a) existing paradigm of relation-based trajectory prediction methods
(b) Our proposed approach
Figure 1: Different from the common paradigm on inferring relation for trajectory prediction, our approach aims to learn interpretable relations, prioritize agent motions, and make in-order prediction based on their priorities.
Multi-agent trajectory prediction is an essential component in a wide range of applications from robot navigation to autonomous intelligent systems. While navigating in crowded scenes, au- tonomous agents (i.c., robots and vehicles) not only themselves interact, but also should have ability to observe others' interactions and anticipate where other agents will move in near future. This ability is crucial for autonomous agents to avoid collisions and plan meaningful machine- human/machine- machhie interactions.
Designing robust and accurate trajectory prediction model has attracted much of recent research efforts. In fact, meaningful reasoning about interactions among agents provides valuable cues to im- prove the trajectory prediction accuracy, especially in highly interactive scenarios. However, how

---

Under review as a conference paper at ICLR 2023
leam/discover meaningful relations among agents from historical motion data to improve prediction accuracy remains a challenging task in recent research. There is a large body of recent research fo- cused on modeling interaction among agents for future trajectory prediction. Some notable research in this field learn interaction features using advanced deep learning techniques such as graph neural networks (Scarselli al. 2008), social pooling mechanism (Alahi et al. 2016, [uppta et al. 2018), or attention networks (Kamra et al. 2020). These works follow a common paradigm, as shown in Figure D. to infer the relations among agents using their historical motions. The main limitation of these approaches is that they lack a mechanism to learn the motion importance of each agent in the scene. realistically interactive scenarios, it is often that the agent movements implicitly imply that an agent gains higher priority to decide where and when to move, their movements will impact the others in the scene. For example, in driving scenarios, vehicles yield to crossing pedestrians (higher prioritized agents). In the basket ball game, the other players' movements are likely to be conditioned (i.e., impacted) ) by the ball-congrolling offensive player.
To handle the aforementioned limitations, we propose a new approach, shown in Figure Ip. to prior- itize each agent motion based on their interpretable relations. Our model first learns the interaction among agents at each time-step. Inspired by the relation learning method (Fujii et al. 2021 for imal movements, we design a inter-agent encoder that consists of two sub-encoders, each of which represents innate movement and navigation capacities of agents. While the navigation encoder cap- tures the agent relations based on the moving directions (i.e. where to move), the motion encoder infers the relation based on the motion capacity (i.e., how move). Next, the prioritization module quantifies the importance score (i.c., priority) of each agent by measuring their motion impacts on other agents. Based on the orders of priorities, sequential predictions are made to allow the predicted future trajectories of agents with higher priorities to impact on the lower prioritized ones within their relation structures. In summary, our contributions are:
We propose a novel prediction pipeline with motion prioritization module to prioritize the impor- tance of each agent based on its motion impacts on other agents within their interpretable relation structures.
We design an interpretable interaction encoder to capture the agent relations from both navigational and motion perspectives. The relationships among agents are learned interpretably at each observed time step to produce meaningful relation structures for prioritization task.
We evaluate our prediction model on several highly interactive datasets: Charged Particles, NBA, and INTERACTION. We show that the proposed model is able to learn meaningful interaction features and outperforms of models on these datasets.
RELATED WORKS
Multi-Agent Trajectory Prediction Multi-agent trajectory prediction is an actively researched problem due to its broad applications in robot planning (Schmerling et al. 2018), traffic predic- tion (Liao al. 2018), sport video analysis (Felsen et al. 2017). Recent research have focused on modeling the relations among agents seeking to improve trajectory prediction. In general, existing approaches employ common structures such as graph neural network, social-GAN (Mohamed al., 2020), attention network, transformer, etc. to learn agent interactions from their motions. Notably, Kosaraju et al. (Kosaraju et al., 2019) on graph attention network (Veličković et al. 2017) to decide how much information to share between agents. Kamra et al. (Kamra al.) 2020) developed ded- icated attention mechanism for trajectory prediction from the inductive bias of motion and intents. Kipf et al. (Kipf al. 2018) proposed neural relational inference (NRI) proposed by taking the form of a variational auto-encoder. Jiachen et al. (Li et al. 2020) improved NRI by using graphs that evolve over time. Although these approaches can capture some interactions among agent, they lack mechanism to reason the agent priorities, important feature cues to improve trajectory results.
Relation Discovery Another closely related research theme focuses on discovering the Granger- causal (GC) relationship among agents. SENN (Alvarez Melis & Jaakkola, 2018) is the first self- explanatory network, class of intrinsically interpretable models, which explains the contributions concepts (i.e,, raw inputs) to predictions. SENN was applied to infer GC relationship via gener- alized vector autoregression model (GVAR) (Marcinkevids & Vogil 202T). which captures the GC relationship via coefficient matrices. These models have shown promising performances learn
2

---

Under review as a conference paper at ICLR 2023
Motion Decoder
Motion Encoder
Zyni
Za.i
Inter-agent
Motion Prioritization
Encoder
Past trajectory
Future trajectory
Relation Inference
Figure 2:
(The overview of our prediction model. Our model consists of three main modules: Motion
Encoder (Section B.T. Inter-agent Encoder (Section 5.3). and Decoder (Section 3.2) with Motion
Prioritization and Relation Inference (Section 3.4
agent interaction in simulated environments, where the GC relationships are known and static. Re- cently, ABM model (Fujii al. 2021) was proposed to further extend GVAR model to capture animal interactions such as approaching or repulsing. ABM relies on scientific knowledge and as- sumes that if an agent goes straight then there is interactions. For relation discovery in trajectory prediction, (Makansi et al. 2021) analyzed feature attributes to provide insights on the impacts of relation feature and discussed its links causality inference. Recently, M21 (Sun al. (022) learn to predict, but limits to pairs of relations, and only applicable for traffic scenes. Different from these works, build tions in real-world interacting scenarios. Based on the learned relation structures, agent motions are prioritized for the purpose of future trajectory prediction.
pipeline and extend relation modeling schemes to discover agent rela-
METHODOLOGY
In this work, tackle the problem of multi-agent trajectory prediction. Given the past trajectories X2... XN [Txxd] E R NT.N of N agents with d-dimensional spatial locations the past To time steps, aim to predict their trajectories Y [Y Y2...., YN RT, TTxx R RT, xd in the next T₂ future time steps. The overview of our prediction model is shown in Figure 2 Our model consists of three main modules: (1) Motion Encoder: encodes historical motion of each agents. (2) Inter-agent Encoder: encodes interactions among agents; (3) Decoder: infer relations.
X [[,,,
prioritize motion, sequentially predict future trajectories of agents. Next, we present the details of each module.
3.1 MOTION ENCODER
The motion encoder X) captures the individual movement of agents by learning the hidden motion feature h E R.N xdds conditioned on X. We encode the hidden state of each agent as:
e, ReLU((onvID(XX)) RT.xC.
[o h1] = GRU(e,), A/ Rdh,
(2)
where f is the hidden motion state of target agent at current time step t, of is the output feature of GRU, C. is the size of embedded feature e;.
3.2 MOTION DECODER
The motion decoder ((()h) decodes the future trajectories of each agent i from the learned hidden motion feature h.. We formulate this decoding process as:
1o |, GRU(N)).
(3)
Apt oo PP: fc(o,*1 )x ++1 1 App y
(4)
N(4 ++1 ,0
(+1 pr. - Rd
(5)
where fe is fully connected layer. To cope with multi-modal nature of future trajectories, we predict bivariate Gaussian distribution N(1,, +1 pt) in each future time step i (lo + 1,...,a lo T,).

---

Under review as a conference paper at ICLR 2023
where 14 HyzxHy); of*1 (azz 0y), ps the mean, standard deviation, and corre- lation coefficient. We then randomly select samples from the distribution for the final multiple trajectories prediction. Due complex interactions among agents, relying historical motions is not adequate for accurate predictions. Thus, in the next stages, we extend the decoder to incorporate relation features from inter-agent encoder and from the motion prioritization module. The decoder formulated q{Y hh, Ras Zm). where 2n and Zm are relation features from inter-agent encoder and motion prioritization, which will be presented next.
My);
3.3 INTER-AGENT ENCODER
The encoder Pa(za: x depicted in Figure t learns interpretable relation features 2a given the past locations of all agents. It also produces relation matrices (i.e., coefficient matrices)
④, consisting of relation matrix & E RNXN at each observed time step / E (10 - To + fo). " is an asymmetric square matrix, which elements indicate the motion impact of an agent another. Thus, the value of $ is high if there is strong motion impact from agent to j. In other words, agent i contains rich information that improves future predic- tion of agent j. Conversely, of 0 indicates no relation i + j. The relation matri- ces are learned during the training process via inducing sparsity loss, introduced in Section 3.4
To learn interpretable and meaningful relation matrix, we design an disentangled inter-agent encoder, which consists of an interaction- based navigation encoder Fx (2,(() and an interaction-based movement encoder Fm
h
Zu.i
T1
to
Our design inspired by that autonomous agents innately possess common navigation and movement functions, which enable them ability plan the future movements. While the navigation encoder captures the relation based on the future directions of agents, the movement encoder infers the relation based on the strength of their movements. Specifically, we imple- ment our inter-agent encoder following the concepts of self-explanatory neural networks SENN(Alvarez Melis Jaakkola 2018p, and its extensions to to discover causal relationships (GVAR (Marcinkevičs & Vogt 2021). ABM (Fujüi et al., 202I)). ). These networks consists of link function g(-). basis concepts (z()). and explainable function ((x) to each concept to the predictions. The general form is:
F
Fm
uM
F
Figure 3: The architecture of Inter-agent encoder.
fx) = (((())))()))....aaa.
In our work, we consider the concepts (xx) being motion feature hú, the relation matrix 4 is the result of explainable function (zx), and g(.) is the Equation C can be written
ta
2
4'ht + 24 E Rd.
t=
g' can be decomposed to & and $ that is 0' = Φ O TC where C is element-wise mul- tiplication, é represent independent noise. Each matrix represents the agent relationships from motion and navigational perspectives. Formally, $L and Φt are learned from motion and naviga- tion functions as: ge = Ft (X')= E Nd, x and के. Ff E R Nd. XNN-)x1 where w/, r"] E I2d= is the input feature, concatenating the observed location X all agents and its relative locations to others o ( F' and F,, are neural networks. Equation 7 makes our relation modeling to capture agent relations in each observed time step, which is different from NRI (Kipf al. 2018) and dNRI (Graber & Schwing 2020). We note that ABM (Fujii et al.) 2021) also shares a similar structure, but with limitations that hinder the ability to extend SENN to work in human interactive scenarios. The key differences highlighted below:
4

---

Under review as a conference paper at ICLR 2023
(1) ABM model designs specific navigation function Fn capture the relationship of few animal species, where there relationship occurs within a close distance (i.e., local scope). Thus, this model does not generalize well to other weal-world applications, where the relation could be occurred in further distance. In this work, we extend the navigation functions to captures relations in both local and global scopes F2 (u") Sa F2 (ut, ) + (1 Sa F F((x)) where F2 and Fg are local and global navigation functions, which capture the relations within local areas and entire scene, respectively. The contribution of each function weighted by Sa. sigmoid function with learnable parameter a; [0, 1]. We present the implementation and impacts of these functions in the supplementary material.
(2) While previous models (Fujii et al. 2021 Marcinkeviës & Vogi 2021) infer the causal rela- tionships without focusing on long-term trajectory prediction task. In contrast, our work focus on learning interpretable relation, which is meaningful for motion prioritization in the later stage and the entire pipeline trained improve the multi-step future prediction.
3.4 MOTION PRIORITIZATION
The motion prioritization module Pmu (Zm, r/ Y) learns to prioritize the impact of the agents based on the learned relation matrix from Inter-agent Encoder. It also produces the anture-trajectories conditioned relational features 2m for each agent. We present the details of each module below:
Contur-Condddioned Relation Inference Given the learned relational matrix D at each observed time step E {to - To to), we quantify the relation matrix among agents S E N as:
Sig = To To-
Σ
(8)
Sūj
N Σ
W
[0, 1]: was =
(9)
Σ P 2j-0.jt 0.j41 Si.
where 4 = [ ..... fo), consisting the relation matrix at each observed time step, and o' de Φt Due to the dynamic movements of agents, the recent relation matrix is more relevant to predict future time steps compared to those in further past. To achieve this, we design a temporal kernel k(t, to) Sa, (1/lfu
der)) with Sa, is a sigmoid function with gain ay- The kernel emphasizes the relation feature as it closer to the current time step, and weaken those that is away. Next, we construct the iuture-conditioned relation feature Zm,i of target agent as:
h, = GRU(CCnnIID(() E R. Zm,i = w h2 E Rd.
(10)
(11)
Motion Prioritization The intuition is that an agent gains higher priority when its motion is less affected by other's motion and more impacts on others. In fact, the impact score can be measured from the relation matrix. We associate each agent with priority score m; = Eド Si.. Based on the priority score, we propose to predict future trajectories in order of their priorities. This is to allow the predicted trajectories of higher prioritized agents have impacts on the ones with lower pri- orities. Considering motion prioritization, we can extend the motion decoder as qY(h,, žas 2..). In fact, the predicted trajectories of each agent can be updated (i.e., refined) multiple times to encourage the self-corrected among agent's 's future trajectories and to fully utilize the future-conditioned rela- tion features. To implement this idea, we iteratively decodes the future trajectories multiple times defined by N, E IR. Intuitively, if N, = O, then there no updates on predictions. In other words, the motion prioritization does not have effects. Otherwise, there will be N, loop over the prioritized list {i.j..., N), where m. 2 my. The details of decoding procedure and the impacts of N, on prediction accuracy presented in our supplementary material.
5

---

Under review as a conference paper at ICLR 2023
Loss Function
We jointly train the proposed models by using the following losses:
Tp
To
((0) = Σ predd (fread(y, Yi) + Σ
Euparsity(
(12)
Lered = min (11 Y2
(13)
C Esparsity
[a| + 11
(14)
To
where Lpred is the min-over-k mean-square- (MSE) prediction loss, which encourages diver- sities among K predictions sampled from the predicted Gaussian distribution. Csparesty is sparsity-inducing penalty term used to learn the relation matrix. In our implementation, elastic-net-style penalty term (Nicholson et al. 2017) (Equation 14 with a 0.5, and || || the Frobenius norm.
the
EXPERIMENTAL RESULTS
We evaluate our model on simulated Charged Particles (Kipi et al. 2018) dataset and two complex interaction datasets, which commonly favors the applications of robot navigation and autonomous system: NBA (Yue et al. 2014) and INTERACTION (Zhan et al. 2019).
Charged Particles Charged particles (Kipf et al. 2018 simulated deterministic system, which is controlled by simple physics rules. In each scene, there are 5 charged particles. Each particle has either positive or negative charge with equal probability. Particles with the same charge repel each other, and vise versa. We set T = 100 and To = 80 for each scene. We generate 50K scenes for training, and 10K each for validation and test respectively.
NBA Dataset This dataset contains tracking data from the 2012-2013 NBA season, and is pro- vided in eL al. (021). The dataset consist of trajectories of a ball and 10 players from both teams (i.e., 5 players each team). We preprocess the data such that each scene has 50 frames and spans approximately 8 seconds of play, and the first 40 frames are historic, i.c. T 50 and To 40.
INTERACTION consists of different realistic and interactive driving scenarios in roundabout, un-signalized intersection, signalized intersection, merging and lane changing. In total, the dataset is collected from 11 locations using drones or fixed cameras. We follow the same train/validationntest splits proposed by (Zhan al. 2019). We set To 30 frames (3 seconds), Tμ 10 frames (1 second).
The evaluations made in comparison with state-of-the-ar models: Social-GAN Guptta al. 2018), Fuzzy Query Attention (FQA) (Kamra et al., (020), Dynamic Neural Relational Inference (dNRI) (Grberr & Schwing 2020,, and GRIN (Li all 202Tp. We evaluate our model using the commonly-used metrics: Average Displacement Error (ADE) and Final Displacement Error (FDE), which are the L2 distance between the ground truth and predicted trajectories, and the L2 distance between the ground truth final destination and predicted final destination. Following GRIN (Li et al. 2021, 1, we report the best-of-100 displacement error of each trajectory/destination We present the details of comparison models and implementations in the supplementary material.
4.1 QUANTITATIVE RESULTS
We provide quantitative comparisons with related methods on Charged Particles and NBA datasets
in Table D We follow the same experiment setups used on GRIN (L al. 202T). Our approach outperforms relation modeling methods (NRI (Kipf et al. 2018). dNRI (Graber & Schwing 2020 in ADE/FDE, which demonstrates the efficacy discovering agent relations in each observed time step. Additionally, our model also outperforms state-of-the-arr model (Kamra et al. 2020) and interpretable model (Li al. 202T) on both datasets by large margins. concludes that the proposed prioritization module plays pivot role in improving prediction accuracy. We perform the ablation study (see Table ( to investigate the impact each model component. The brief description of each module is follows: E using only motion encoder/decoder, Fm, is the motion encoder; F,

---

Under review as a conference paper at ICLR 2023
Model
Charged Particles ADE/FDE
NBA
Components
Charged Particles
NBA
ADE/FDE
E
Fm FR Re Pri
ADE/FDE
ADE/FDE
NRL Kipl all 2018
0.63/1.30
2.10/4.56
0.51/0.96
18333.46
dNRI
Kariner N Schwing 2020
0.94/1.93
2,02/4.52
0.50/0.94
_8/3.52
FQA
0.82/1.76
S-GAN (upa C 2018
0.66/1.25
1.88/3.64
0.45/0.87
1 1,87/3 50
GRIN Ours
0.52/1
1,72/3.59
0,48/0,77
1.442.90 99
0.420.72
1.66/889
0.42/0.72
1.66/2.89
Table I: Comparisons with Other Relation based Trajectory Prediction Models.
Table Ablation Study on Charged Particles and NBA Dataset.
(a) Ground truth
(b) Prediction
(c) Relation matrix
(d) Ground truth
Figure 4:
Qualitative results on Charged Particles dataset. Darker colours corresponds to weak
relation between agents in relation matrix.
the navigation encoder, Re is future-conditioned relation inference, Pri motion prioritization module. We can see that the relation-based navigation encoder(i.e., E Fn ) and relation-based motion encoder (i.e., + Fm show their positive impacts when reducing the prediction errors on Charge Particles compared to the variant only individual motion encoder E. However, it is interesting to observe that these variants worsen the results on NBA sport dataset. because the NBA consist of mostly realistic and complex interactions among players. The players make strategy movements based on the future movements of other higher prioritized players. This explains why adding relational inference (i.e., module Re) condition one player's movements on other's future trajectories helps reduce the prediction errors. Interestingly, our full model with Pri (last row) shows best results overall. This indicates that each player has different motion impact that affects other players' movements in NBA sport games.
reasonable
4.2 QUALITATIVE RESULTS
We show the qualitative results consisting of the predicted trajectory and learned relation matrix. In Figure 3 the charged particles preserve very interesting swirling interactions, such as among two particles and 4 in the first row, and two groups of swirling particles (1,4), and (0,3) in second row. The learned relation matrices show that these interactions can be captured in the corresponding squares (1,3), (3.1), (4, 2), (2,4) for the first scenario, and (3,0), (0,3) (1,4), (4,1) for the second scenario. We note that the stronger relation among agents, the more yellowish (i.e., Sty + 1) the corresponding square in the relation matrix, while blue (S., + 0) indicates there is relation.
In NBA dataset, we observe several interesting interaction scenarios, shown in Figure row, there are several notable interaction pairs (2,7), (0, 5), (4,9) captured in our relation matrix. It can observed that in this scenario player is following along with player 2, moving from one side to another side of the court. At the same time, players 5, 9 play defense against players 0 and 4, respectively. second row, our model learns there are group of motion among players (0, 6. 8,
the first
1, 7, 3), where player 0 has dominant motion impact other players. In the last row, there two

---

Under review as a conference paper at ICLR 2023
ADE 1/20/100
FDE FDE 1/20/100
Models
GRIN
Li al. 2021
1.80/1.27/0.66 1.46/1.16/0.61
3.31/2.33/1.26 2.69/2.11/1.22
Ours
Table 3: Quantitative results INTERACTION datasets with different number of prediction sam- ples 1/20/100.
main groups (1,7) and (2,9, 4 ,5 6.), especially player 9 plays large attention to player 2 as player defending against player 2, who trying to score.
D
Ground truth
(b) Prediction
(c) Priority map
(d) Relation matrix
Figure 5: Qualitative results on NBA dataset. Offensive team (i.c., controlling the ball) colored in red, defensive team is colored in blue. The ball is colored in black. Solid line represents observed trajectory. Dotted line represents predicted trajectory. In priority map, the larger circles represent higher priority scores of agents.
4.3 RESULTS INTERACTION DATASET
In this section, we present evaluation on INTERACTION dataset comparisons with GRIN Li et al. [02T] on different number of prediction samples: 1, 20, 100, as reported in Table Both mod- els are trained with 100 epochs. Overall, our model outperforms GRIN model in both ADE/FDE in different number of predicted trajectories. Finally, we provide visualization on different driving sce- narios such as intersection, roundabout and merging INTERACTION dataset, shown in Figure K Generally, interaction types such as following, or yielding are very common in driving scenarios. The results show that our model is able to capture these types of interactions several scenarios. As we can see that in first row (intersection scenarios), agent 2 is waiting for agent 0 and to pass before can turn left. Meanwhile, agent makes slight interaction (i.e., yield) to agent 3. In the roundabout scenarios (second row), the most notable interaction is between 2 and 4. In this scenario, agent 2 is waiting for agent to cross, before it can go into the roundabout section. the last row, there is interesting merging scenarios from 3. and agent following when there merge from
8

---

Under review as a conference paper at ICLR 2023
pesirtis
ground-soe truth prediction
gound4uuth
Trajectory prediction
(a) Priority map
(a) Relation matrix
Figure Visualizations from scenarios: intersection (first row), roundabout (second row), and merging (third row) in INTERACTION dataset.
3 to 2 lanes. In addition, the priority map for each scenario indicates the agent's importance, which provides meaning cues for improving trajectory prediction.
LIMITATIONS
Although we have demonstrated that the proposed model has successfully captured several inter- action behavior in the given datasets, discovering causalities in realistic scenarios such as in au- tonomous driving still challenging. As possible solution, exploring scientific knowledge about traffics and the priorities of different agent types could be incorporated to improve the model accu- racy. Secondly, our model discovers relations and prioritization from bird-eye views, which provides global-view relation agents. However, robots ego-vehicles also observe the relations with sur- rounding agent from the front camera views. Thus, the future work of considering fusing relations from different views be explored to strengthen the relation features. Lastly, discovering relation and trajectory prediction (if the identity of subject is known, which is not the case in our work) can raise issues regarding privacy sensitivity.
CONCLUSION
In this paper, we introduce a novel multi-agent trajectory prediction. Our model discovers inter- pretable interaction feature from historical trajectories of all agents. We also developed motion prioritization module, which prioritize those agents which have higher impacts on other future mo- tions. Based on the prioritized scores, the decoder makes sequential prediction of each agents with iterative predictions. broader impacts, our prediction could used to infer relations other scenarios such as human motions, or inferring robot-human interactions and predictions.
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 961-971, 2016.

---

Under review as a conference paper at ICLR 2023
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances in neural information processing systems, 31, 2018.
Panna Felsen, Pulkit Agrawal, and Jitendra Malik. What will happen next? forecasting player moves in sports videos. In Proceedings of the IEEE international conference on computer vision, pp. 31.3333433511 2017.
Keisuke Fujii, Naoya Takeishi, Kazushi Tsutsui, Emyo Fujioka, Nozomi Nishiumi, Ryoya Tanaka, Mika Fukushiro, Kaoru Ide, Hiroyoshi Kohno, Ken Yoda, et al. Learning interaction rules from multi-animal trajectories via augmented behavioral models. Advances in Neural Information Processing Systems, 34, 2021.
Colin Graber and Alexander Schwing. Dynamic neural relational inference for forecasting trajecto- ries. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 1018-1019, 2020.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: So- cially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018
Nitin Kamra, Hao Zhu, Dweep Kumarbhai Trivedi, Ming Zhang, and Yan Liu. Multi-agent trajectory prediction with fuzzy query attention. Advances in Neural Information Processing Systems, 33: 22540, 2020.
Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pp. 2688- 2697. PMLR, 2018.
Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martin, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. Advances in Neural Information Processing Systems, 32, 2019.
Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Heterogeneous multi- agent multi-modal trajectory prediction with evolving interaction graphs. 2020.
Longyuan Li, Jian Yao, Li Wenliang, Tong He, Tianjun Xiao, Junchi Yan, David Wipf, and Zheng Zhang. Grin: Generative relation and intention network for multi-agent trajectory prediction.
Advances in Neural Information Processing Systems, 34, 2021.
Binbing Liao, Jingqing Zhang, Chao Wu, Douglas Mcllwraith, Tong Chen, Shengwen Yang. Yike Guo, and Fei Wu. Deep sequence learning with auxiliary information for traffic prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 537-546, 2018.
Osama Makansi, Julius Von Kügelgen, Francesco Locatello, Peter Gehler, Dominik Janzing, Thomas Brox, and Bernhard Schölkopf. You mostly walk alone: Analyzing feature attribution in trajectory prediction. arXiv preprint 10.55221 2021.
Ričards Mareinkevičs and Julia E Vogt. Interpretable models for granger causality using self- explaining neural networks. arXiv preprint arXiv:2101.07600 2021.
Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgenn: A cial spatio-temporal graph convolutional neural network for human trajectory prediction. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14424 14432, 2020.
William B Nicholson, David S Matteson, and Jacob Bien. Varx-l: Structured regularization for large vector autoregressions with exogenous variables. International Journal of Forecasting, 33(3): 627-651, 2017.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEFE transactions neural networks, 20(1) 20(1):61-80 2008
10

---

Under review as a conference paper at ICLR 2023
Edward Schmerling, Karen Leung, Wolf Vollprecht, and Marco Pavone. Multimodal probabilistic model-based planning for human-robot interaction. 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3399-3406. IEEE, 2018.
Qiao Sun, Xin Huang, Junru Gu, Brian C Williams, and Hang Zhao. M2i: From factored marginal trajectory prediction to interactive prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6543-6552 2022.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks, arXiv preprint 17xiv::710..1000 2017.
Yisong Yue, Patrick Lucey, Peter Carr, Alina Bialkowski, and lain Matthews. Learning fine-grained spatial models for dynamic sports play prediction. In 2014 IEEE international conference on data mining, pp. 670-679 IEEE, 2014.
Wei Zhan, Liting Sun, Di Wang. Haojic Shi, Aubrey Clausse, Maximilian Naumann, Julius Kum- merle, Hendrik Konigshof, Christoph Stiller, Arnaud de La Fortelle, et al. Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps. arXiv preprint arXiv:/910.0000 1910.0308,, 2019.

---

Under review as a conference paper at ICLR 2023
SUPPLEMENTARY MATERIALS: AGENT PRIORITIZATION WITH INTERPRETABLE RE- LATION FOR TRAJECTORY PREDICTION
Anonymous authors Paper under double-bling brind review
In this supplementary material, we first provide the details Inter-Agent encoder (Section We then discuss algorithm of future trajectory decoding process with motion prioritization (Sec-
tion 2 We further provide the implementation and training details our model(Section 3) and
additional qualitative results (Section 4 Charged Particles Graber & Schwing (2020). NBA Yue
ct al. (2014). and INTERACTION Zhan et al. (2019) datasets.
DETAILS OF INTER-AGENT ENCODER
We introduced in our main paper that the Inter-Agent encoder consists of movement function Fm and navigation function Fz, which capture agent relations from motion and navigation perspectives. inspired by animal movements Nathan et al. (2008); Fujui et al. (202T). Specifically, each function produces a relation matrix (4m m and 2x)) where each element of 2m (produced by Fn) represents the amplitude of interaction, and each element of 4,, (produced by 2,)) represents the sign of in- teractions (e.g., positive two agents approach, negative: two agents repulse). To further extend the model to generalize well to real-world robot applications, where the relation could be occurred in further distance. We implement our navigation function to capture interaction from local and global scopes, which capture the relations within local areas and entire scene, respective. Thus, Fr can be decomposed into local function Fü and global function FR. The local function F2 capture interaction with emphasies given pre- pre-nefined thershold dignore follows:
(aannor)(aa,, (v*d 1 2
( = Soal ||22.111
x 2,
(1)
while the global function capture the interaction in entire scene (without threshold).
1 2 2,
(u,) = Saa( || # (Sa., (v+
(2)
where Sa,, and Sa,, are sigmoid functions with gain ag and ag, respectively; dignor is threshold for defining the local area. Thus, if the distance of two agents interaction. phj مd - is the relative distance of agent j to agent with r² v:). eii is the velocity of agent (i.e., v²) in the direction of pt thus Sa, (vf 1) x sign represents of effects of on i. Then, the navigation function is formulated
greater than dignores then there is
(h') &. (1)) + F
(3)
where S. is sigmoid function with learnable parameter C. [0,
In addition to our ablation studies presented in (Section 4, Table in main paper), we validate the impact of the proposed navigation functions trajectory prediction. In this experiment, train/test the different variants of Inter-Agent encoder on Charged Particles dataset. As shown in observe that local navigation function F`, and Fş both improves the prediction accuracy. This indicates the important roles of these functions in our model.
Table
2 DETAILS OF DECODING ALGORITHM
We present the details of trajectory decoding process with motion prioritization Algorithm | We note that the predicted trajectories of each agent be updated (i.e., refined) multiple times to

---

Under review as a conference paper at ICLR 2023
Table 1: Comparisons with other relation based trajectory prediction models.
Model
Charged Particles
ADE
FDE
GRIN
Li 2021
0.52
1.09
E
0.54
1.01
Fm
0.52
0.95
E + Fm + F
0.50
0.94
E F., + Fl + F
0.48
0.93
2.5
ADE
3.5
ACE
3.00
2.55
1.75
Figure 1: Analysis of number of iterations (N, on Charge Particles (left) and NBA (right).
encourage the correction among agent's future trajectories and fully utilize the future-conditioned relation features. To implement this idea, we iteratively decodes the future trajectories multiple with is
times defined N,. For example, if the prioritized list of agent is O the agent with highest priority, 5 is the lowest one. One iteration (i.e., N₂ 1) corresponds to sequential prediction in order of 3 + + agent with higher priority influences on trajectories of agent with lower priority. We study the impact of different number of iterations to update the future trajectories, shown Figure It is interesting to observe that only one step of refinement is enough to achieve the best prediction on Charged Particles. On NBA dataset, although we achieve best results at 5 iterations, the results also seem to be saturated in the first iteration. This concludes that relational features has been fully shared from the pribriiiri agent to the lower one in list prioritized agent in the first iteration.
4 + 5. This allows the predicted trajectories of
Algorithm 1: Algorithm for trajectory prediction with motion prioritization
Input: List relational matrices = ( To+l
600] in To historical time-steps
Output: Predicted trajectories Y [Y,, Y2, YN e RT.xd of all N agents in the next Tр future time steps.
Calculate relational weights using Equation
Estimate the priority score (i.e., impactness) for each agent. m4 = > Of Sig
V=N
Rank agents with priority scores in descending order: O = (i,j.... N): ordered list of agents,
where m; 2 My-- for step in N₂ do for in do Caleulate relational feature fr for target agent using Equation 10 and 11. Update predicted trajectory
Y! = (c(GRU(( 2m,
IMPLEMENTATION DETAILS
We implement our model using Py Torch Pytorch| (0118) which has BSD-style license. We train the model with the Adam Zhang 2018) optimizer with the learning rate of 0.0001 and 100 epochs. The
2

---

Under review as a conference paper at ICLR 2023
t=10
t=60
t=120
t=240
(a) Ground truth
(b) Prediction
(c) Relation Matrix
(d) Ground-truth
Figure 2: Additional qualitative results on Charge Particles dataset
batch size is 128. For all datasets, implement the movement encoders with 2 MLP layers, each has hidden size of 50. The hidden size of GRU in both encoders and decoders is 48. In the training loss, we set a 0.5, and 1.
ADDITIONAL QUALITATIVE RESULTS
In this section, we present additional qualitative results on three datasets: Charged Particles, NBA, INTERACTION.
Results on Charged Particles Figure shows the trajectory prediction and the learned relation matrix at different time steps. We observe that at different time steps, particles involves in different interactions. For example, at t=10 and t=60, there are mainly swirling interactions between particles (2, 4) and 3). At t= 120, particle reaches toward particle 0, and then goes further away at t=240. The learned relation matrix show that these subtle movements and interactions can be captured.
Results NBA Next, show predictions and priority scores of players in basketball games (Figure 3) NBA dataset consists of highly complex interactions among players. the first scenario, player 0 and 4 in offensive team (red) have higher priority passing the ball to player 4; thus, their movements highly influence others' motions. In the second scenario, player 0 holding the ball but does not have large impact; this is because the motions of the ball and player 0 is not changing much. Other players may suspect that the player 0 could pass the ball other offensive players. Thus, the movements of other players the red team are more important to observe. Interestingly, player 2 has the most critical move and thus it generates the largest impact on other players.
(i.e., higher impact) as the player
Results on INTERACTION We further visualize the trajectory prediction, priority map, and re- lation matrix for two interactive scenarios: merging and roundabout in Figure + We can see that our model can capture intriguing relation matrix and leams the motion importance of agent effectively. In the merging scenarios (top row), agent gains highest importance as it going to merge and have high relations with others agents in front (agent 1), and behind (agent 3). In the roundabout scenario (bottom row), we observe two groups: (1,3) and (0,2,4) on different sides of the roundabout. In- terestingly, see that the relation between agent 0 (pedestrian) and agent 2 (vehicle) can be

---

Under review as a conference paper at ICLR 2023
Ground truth
Prediction
Priority map
Ground truth
Prediction
Priority Map
Figure 3: Qualitative results on NBA dataset. Offensive team (i.e., controlling the ball) colored in red, defensive team is colored in blue. The ball is colored in black. Solid line represents observed trajectory. Dotted line represents predicted trajectory. In priority map. the larger circles represent higher priority scores of agents.
(6) Trajectory prediction
Priontty map
matrix
Figure 4: Additional qualitative results on INTERACTION dataset.
captured even though they still far-way. The agent 2 gains most motion priority within the group (0,2,4) because has relationship with both agent 4 and
REFERENCES
Keisuke Fujii, Naoya Takeishi, Kazushi Tsutsui, Emyo Fujioka, Nozomi Nishiumi, Ryoya Tanaka, Mika Fukushiro, Kaoru Ide, Hiroyoshi Kohno, Ken Yoda, et al. Learming interaction rules from multi-animal traeectories via augmented behavioral models. Advances in Neural Information Processing Systems, 34, 2021.
Colin Graber and Alexander Schwing. Dynamic neural relational inference for forecasting trajecto- ries. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 1019, 2020.
Longyuan Li, Jian Yao, Li Wenliang, Tong He, Tianjun Xiao, Junchi Yan, David Wipf, and Zheng Zhang. Grin: Generative relation and intention network for multi-agent trajectory prediction.
Advances in Neural Information Processing Systems, 34, 2021.
Ran Nathan, Wayne M Getz, Eloy Revilla, Marcel Holyoak, Ronen Kadmon, David Saltz, and Peter E Smouse. A movement ecology paradigm for unifying organismal movement research. Proceedings of the National Academy of Sciences, 105(49) 19059, 2008.
Automatic Differentiation In Pytorch. Pytorch, 2018
Yisong Yue, Patrick Lucey, Peter Carr, Alina Bialkowski, and Iain Matthews. Learning fine-grained spatial models for dynamic sports play prediction. In 2014 IEEE international conference on data mining, pp. 670-679 IEEE, 2014.
Wei Zhan, Liting Sun, Di Wang. Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Kum- merle, Hendrik Konigshof, Christoph Stiller, Arnaud de La Fortelle, et al. Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps. arXiv preprint S011:19900000833 2019.
Zijun Zhang. Improved adam optimizer for deep neural networks. In 2018 IEEE/ACM 26th Inter- national Symposium Quality of Service (IWQoS), pp. 1-2. leee, 2018.
4


Reviews:
[{"review_id": "hMiyOX7Q58X", "reviewer": "Reviewer_8XRE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new method for multi-agent trajectory forecasting. Unlike existing methods that just encode inter-agent interactions, the proposed method additionally learns to predict the priority of agents that explains the dependency between agents regarding their future trajectories (i.e., future trajectories are generated in order based on this priority.) The proposed method is evaluated on multiple datasets (Charged Particles, NBA Dataset, and INTERACTION) and confirmed to outperform multiple state-of-the-art methods in terms of ADE and FDE.", "summary_of_the_review": "While I acknowledge the novelty of the proposed method, its effectiveness remains questionable to me. Specifically, I would like to see the following points clarified.\n\n1) How is the proposed method scalable for a larger number of agents, in terms of forecasting performance (ADE/FDE) and required computational cost (runtime)?\n2) How is the proposed method compared to existing methods in terms of interpretability? I believe that it is another important aspect as \"interpretable\" is emphasized in the paper. What can be interpreted only by the proposed method and not by any other method?", "strength_and_weaknesses": "\n## Strong points\n- Overall the paper is clear and nicely written.\n- The proposed work is well motivated. I understood the importance of taking into account the priority in multi-agent trajectory forecasting. \n- Overall, the proposed method is technically sound and designed reasonably. Although the proposed method is built on top of some known techniques (standard encoder-decoder-style forecasting + an extension of causal relationship modeling such as SENN, GVAR, and ABM), introducing the concept of priority for trajectory forecasting seems to be a novel idea as far as I checked.\n- Extensive experimental evaluation is presented. The proposed method is evaluated on multiple datasets. Its effectiveness is demonstrated compared to multiple baseline methods, and with multiple metrics. The ablation study in Table 2 shows the contribution of each technical component of the proposed method. Qualitative results nicely explain how the proposed method works.\n\n## Weak points\n- The paper could have been better if it were clear how large a number of agents the proposed method can deal with. As presented in Section 3.3, the proposed method involves a relation matrix with the size of NxN that becomes inevitably huge as the number of agents increases. Moreover, due to the nature of priority-driven forecasting, the decoding part of the proposed method cannot be parallelized. It remains unclear to me how increasing the number of agents affects the forecasting performance and computational cost (or runtime) of the proposed method. As far as I checked, the numbers of agents considered in the experiments were just 5 (Charged Particle and INTERACTION) and 10 (NBA), although there are other datasets with a lot more agents (e.g., ETH/UCY, Stanford Drone Dataset). \n- As a relatively minor point, it's not perfectly clear from the current manuscript how is the proposed method better compared to prior work in terms of \"interpretability\".\n- Notations could be improved. It was difficult to understand how \\Phi, \\Phi_t, \\Phi_{i, j}, and \\Phi^t_m are related in Section 3.3 and 3.4.\n\n\n## Additional feedback\n- A few typos were found:\n  - a inter-agent -> an inter-agent (p.2)\n  - we now can -> we can now (p.5)\n- It would be better if qualitative results of baseline methods are also provided.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XVK7afbmR23", "reviewer": "Reviewer_fWdF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for multi-object trajectory prediction. Specifically, the proposed method consists of two different encoders and one motion decoder. The conventional motion encoder  inputs past trajectories and outputs the features and hidden motion state of each agent. And the motion decoder receives hidden states and decodes the future trajectories of each agent. Compared to existing methods, the authors introduce a relation matrix that explicitly measures the impact between each agent, as additional inter-agent encoder. To be specific, the inter-agent encoder consists of a navigation and a movement encoder and both of them are implemented in self-explanatory manner, as appeared in literature.\n\nThe authors validate their ideas on three dataset, one simulated dataset and two real ones. Comparing to existing work, such as Social-GAN and GRIN, they demonstrate better performance in terms of ADE and FDE.", "summary_of_the_review": "Overall, I appreciated the writing, clearness and quantitative results of this paper. However, I have concerns in terms of the novelty and the qualitative results. I am willing to raise my rating if my concerns can be addressed. Thanks.", "strength_and_weaknesses": " Strength:\n1. This paper is well-written and easy to follow\n2. Novelties are well-explained and motivations are clear\n3. The proposed method achieves SOTA performance\n\nWeakness:\n1. In terms of novelty, the relationship modeling is minor from my perspective. \n2. The visualization, or qualitative results, seems to be problematic.\n3. Architecture details are missing, which makes it hard to re-produce the reported results.\n4. Extensive results on larger datasets, e.g. NuScenes or Waymo, are expected.\n(See below for more detailed explanations for weakness)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D0RjQlngP8I", "reviewer": "Reviewer_H2is", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work presents a model for motion prediction. The core component of the method is the prioritization of the agents. Agent with higher priority will have better prediction. The work comes supported with experiments and good analysis. ", "summary_of_the_review": "The paper is interesting, well-motivated. \nExtra results and deeper analysis is needed to justify some components and extra literature review is needed. ", "strength_and_weaknesses": "Introduction: \n- Introduction is clear and work is well motivated based on the example of yielding to pedestrian. \n\nLiterature review: \n- Literature review is limited in terms of motion prediction methods. The review jumps directly to GNNs methods with focus on attention ignoring other related works that uses GANs, non-attention GNNs.. etc. \n- For relation discovery the work of “You mostly walk alone: Analyzing feature attribution in trajectory prediction” is highly related and a proper contribution comparison is necessary. \n\nMethodology: \n- The problem definition needs a re-visit, for the example the dimensions of Y is missing. \n\n- 3.1: Reason to use GRU? \n- 3.3 The $\\Phi$ is an NxN matrix the defines the relationship between agents. Aka it’s an adjacency matrix that defines the importance of agents where agents are graph nodes and $\\Phi$ is what defines the edges. Then, what’s the difference between this and ordinary graph attention or a a hand crafter graph edges? \n- Page 5: Did you notice that $F_n(u^n)= $ acts as a complimentary filter? \n- Motion prioritization: It is still not clear to me how it works? Is there a feedback from higher priority predict agents into the model to predict the lower priority ones? What happens if if there is an out of priority order prediction, reverse priority? \n\nResults: \n- Proper introduction of best-of-N ADE/FDE is not present. \n- Expected to see results on classic motion prediction datasets such as ETH/UCY, SDD .. \n- What about memory footprint and inference speed of the method \n- I expected an analysis that shows how higher priority agents results are better than lower priority ones, proving the intent of the prioritization component. Something like this agent(s) ADE/FDE without prioritization is X/Y and this agent(s) is A/B, when the prioritization component is used we see that the first agent(s) ADE/FDE increases while the second agent(s) ADE/FDE decreased. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject