Interaction Modeling with Multiplex Attention
Authors: Fan-Yun Sun
Isaac Kauvar
Ruohan Zhang
Jiachen Li
Mykel Kochenderfer
Jiajun Wu
Nick Haber
Venue: NeurIPS 2022 Accept (2022)
Keywords: interaction modeling
social
relational graph
multi-agent
graph neural network
attention

Abstract: Modeling multi-agent systems requires understanding how agents interact. Such systems are often difficult to model because they can involve a variety of types of interactions that layer together to drive rich social behavioral dynamics. Here we introduce a method for accurately modeling multi-agent systems. We present Interaction Modeling with Multiplex Attention (IMMA), a forward prediction model that uses a multiplex latent graph to represent multiple independent types of interactions and attention to account for relations of different strengths. We also introduce Progressive Layer Training, a training strategy for this architecture. We show that our approach outperforms state-of-the-art models in trajectory forecasting and relation inference, spanning three multi-agent scenarios: social navigation, cooperative task achievement, and team sports. We further demonstrate that our approach can improve zero-shot generalization and allows us to probe how different interactions impact agent behavior.

systems can Interaction Modeling with Multiplex Attention
Fan-Yun Sun Stanford University
Isaac Kauvar Stanford University
Ruohan Zhang Stanford University
Jiachen Li Stanford University
Mykel Kochenderfer
Jiajun Wu Stanford University
Nick Haber Stanford University
Stanford University
Abstract
Modeling multi-agent systems requires understanding how agents interact. Such systems are often difficult model because they can involve variety of types of interactions that layer together to drive rich social behavioral dynamics. Here we introduce method for accurately modeling multi-agent systems. We present Interaction Modeling with Multiplex Attention (IMMA)) forward prediction model that uses a multiplex latent graph to represent multiple independent types interactions and attention to account for relations of different strengths. We also introduce Progressive Layer Training, a training strategy for this architecture. We show that approach outperforms mate-of-the- of-the-ar arr-art models in trajectory forecasting and relation inference, spanning three multi-agent scenarios: social navigation, cooperative task achievement, and team sports. We further demonstrate that approach can improve zero-shot generalization and allows us to probe how different interactions impact agent behavior.
Introduction
Modeling multi-agent systems is important for wide variety of applications, including self-driving cars, crowd navigation, and human human-maching machine collaboration. The dynamics of be challenging model as they are usually governed by various independent types of interactions. Consider modeling crowd navigation at a social gathering. where agents' behaviors might be governed by least two types interactions: target destinations (c.g. a person trying meet up with friend) and collision avoidance (e.g. navigating busy sidewalk). In social scenarios like this, humans often appear to navigate and make predictions based on estimated high-level intentions and relations among the other people [16 45 46). Motivated by this observation, we aim to build forecasting model for multi-agent systems that infers high-level abstractions the form of graphs, entirely through the task of predicting agent trajectories.
Leading approaches in modeling multi-agent systems, such Neural Relational Inference (NRI) 22 Relational Forward Model (RFM) (8), and their extensions [12] (7). use graph neural networks (GNNs) infer edge types for every pair of entities in the interacting systems. However, this inductive bias does not explicitly handle the multiple layers of interactions present in social multi- agent systems and, as shown empirically in Section has led least two shortcomings: reduced performance on long-term predictions and decreased interpretability. Our approach, Interaction Modeling with Multiplex Attention (IMMA), overcomes these issues by using multiplex graph latent structure to model multiple interaction types, with attention graph layers that can capture the strength of relations.
Project website: htttps://cs.stannn stanford, cdu/ sunfanyun/immal A multiplex graph is graph with multiple layers. Nodes are replicated over layers, but each layer have different counectivity [14].
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Multi-agent system
Multiplex Attentional Graph
Infer
Predict
Layer (e.g. target friend)
Layer (e collision avoidance)
Figure 1: Multi-agent systems are often guided by a variety of interactions, such as target destinations (e.g. person trying to meet up with friend) and collision avoidance (e.g. navigating busy sidewalk). Our model uses multiplex attentional graph to infer multiple, independent types of relations among agents and yield improved performance in predicting the dynamics of the system. In this figure, strength of interaction is indicated by graph line thickness.
Furthermore, propose training approach for IMMA that we call Progressive Layer Training (PLT), Learning in high-dimensional space often leads to highly-entangled representations that are uninterpretable [44), but adding explicit disentanglement objectives often leads to decreased model performance [10 18 Inspired by the concept "starting easy" and curriculum learning [5], we train model to learn progressively, one latent layer time. In addition to improving performance, training IMMA with PLT also improves interpretability -ofitaiiiittooffriingggg the potential to answer questions such as: "Who causes agent to behave this way?"
Using our method, we conduct experiments real-world scenarios: social navigation, cooperative task achievement, and team sports. We evaluate models on trajectory prediction, we analyze the relations inferred by our model, and we explore the important role these inferred relations play in making accurate predictions. Additionally, we show that IMMA is effective at zero-shot generalization and conditional generation. summarize,
range of social multi-agent environments inspired by
We propose Interaction Modeling with Multiplex Attention (IMMA), model that uses new latent space structure which we call Multiplex Attentional Graph. It models relations and utilizes these relations to make forecasting predictions.
We propose training technique that is enabled by the multiplexed latent structure: Progres- sive Layer Training (PLT). PLT enables our model to learn types of interactions one at time and furthers the performance and interpretability of IMMA.
Empirically, we show that our method (IMMA w/ PLT) outperforms the models three different social multi-agent enviromments. To understand the interpretability our approach, we evaluate our method on relational inference and conduct conditional generation experiment. We also compare various ablated and modified versions of method, including incorporation of a well-established approach for learning disentangled representations, understand the contribution of each components. Furthermore, showcase the superior sample efficiency and generalization ability of our model.
art
2
Related Work
Many approaches have been proposed model multi-agent interactions, especially as a crucial step toward achieving better performance in tasks such as model-based based reinforcement learning [37] (8].
trajectory prediction [34], and motion planning 40. particular, social dynamics modeling has been
studied in domains such as human crowds [34]. team sports [22 22 27,, and traffic participants [35]
where the interactions between entities lead highly complicated motion behavior patterns. Many earlier works, such as Social Forces [17]. are deterministic regression models that model humans as objects affected by forces. The rise of deep learning ushered in various neural network based models such as Social-LSSTM M- LSTM model with social pooling layer, social-GAN [13 GAN combined with sequence-to-sequene model, Social-Attention
42 42paa recurrent neural network 25], Researchers also proposed to model
that makes use of attention mechanism, and others

---

interactions with networks that were inspired by Theory of Mind framework [32 33 different as we explicitly infer an interpretable discrete structure and perform inference over it.
Our work is
In recent years, graph representation learning has been investigated for relational reasoning in multi- agent systems, where nodes represent interacting entities and edges represent their relations [4] Most existing graph-based methods infer interactions implicitly [ 19 (26, 29 35 36), meaning that little no interpretable abstractions can be extracted from the models. Other works take pre-defined graph topology based on heuristics as input M 23 NRI (22]. DNRI 12), RFM (39], and EvolveGraph [7 takes a step forward to reason about relations by inferring the topology of the underlying interaction graphs. NRI [22] takes the form of a variational auto-encoder while RFM 39), having near-identical architecture as NRI, simply uses a forward prediction framework. All these methods model interactions as latent interaction graphs and leverage them to make future predictions. Our work falls into this line of research with significant improvements to performance in terms of both relational inference and trajectory prediction because of (1) the novel multiplexed latent structure, (2) the incorporation of an attention mechanism, and (3) a progressive training strategy.
3 Methods
In this section, we define our problem and introduce our proposed model (IMMA) and training strategy (PLTT). Our input consists of trajectories of N agents. We denote by X the feature vector of agent& i at time f, c.g. location. We denote by x² = {xx], x'y] J the set of features of all N agents at time and by X = (xmT..7, I,.,, N} the set of features covering time steps from Ti to Th. The task is to take trajectory observations as input and learn to predict successive trajectories of all agents. We aim to estimate p( p(X +T, |X L. where Th is the historical horizon and T is the forecasting horizon. Note that even though agents can have relations and goals that vary in different instances, these must be inferred entirely through the supervision signals of the trajectory prediction task (with no supervision of the relations themselves).
3.1 Interaction Modeling with Multiplex Attention
We propose Interaction Modeling with Multiplex Attention (IMMA)- a model that consists of an encoder and decoder trained jointly. The encoder takes sequence of observations (i.e. trajectories) as input and outputs a latent interaction graph. Subsequently, the decoder predicts future trajectories by performing message passing among agents (nodes) with the inferred latent interaction graph. IMMA assumes that the dynamics of multi-agent systems can be abstracted to a latent graph structure (i.e. a vector of categorical latent variables) z = {zg}, where 2ij represents relations between agents i and j. We optimize the variational lower bound, as in the conditional variational autoencoder (CVAE) 38 21:
C(O, Φ) = E
(x (T)) /////llogoo llog (X (X +1+1:T +T, X 1.Th z)] DKL (qo(z (q6(t// p(z)). (I)
The encoder (approximated posterior) (2 X ITT is neural network with parameters ø, the decoder (likelihood) pe (TT++1:TTTT X I:TT, 2) denotes the f
(2)
is a neural network with parameters 0, and p(z)
the latent graph z is inferred by performing edge-type classification. That is. z represents the likelihoods of edges in the latent graph belonging to set of classes. However, in social dynamical systems multiple types of independent interactions can coexist, with different strengths of relations within an interaction type. In this case, representing the interactions with a single graph with many edge types requires many more edge types to accommodate all possible combinations of interactions. More concretely, consider a navigational environment consisting of two independent types of relations: friendship (i.e. target-seeking interaction) and proximity (i.e. callision-avoidacce interaction). To model this environment with a single graph with many edge types, intuitively we might expect to need at least four edge classes represent the interactions (none, friendsipp, only proximity, and friendship + proximity) and many more if we want to model the interactions with higher fidelity (e.g. treating strong friendship differently from weak friendship). Furthermore, since z is learned to encode likelihood of an edge belonging to a particular class, it may struggle to capture relative strengths of interactions between agents.
In previous works 22 27
39
12
Whhe the latent space is sufficiently regularized by construction, we could impose no prior on the latent space, omit the KL divergence term in Eq. (I). and simply train forward prediction model as in 39
3

---

Encoder
Multiplex attentional graph
Decoder
GNN
K
h++
/++2
Graph layer
Graph GRU
Graph
GNN
GRU
Graph layer
Readout
Reedout
Observation
Fumb
5/+1
Figure 2: Architecture of IMMA. Each agent's 's trajectory (e.g. past position across time) is indepen- dently embedded into latent node state. A multiplex graph latent representation is inferred from all agents' trajectories, IMMA is trained using Progressive Layer Training, where the GNN to compute layer one is first learned and then a second GNN is introduced compute layer two (while the GNN weights for layer one is frozen). The decoder is GraphGRU that uses the same multiplex graph at each prediction step, while updating the latent node state. Note that the input node representations to the decoder are agent-centric. A readout network predicts & from h at each time step.
In the following, we describe how IMMA (depicted in Fig. tackles these limitations, as we introduce the encoder and decoder in more detail.
Encoder with Multiplex Attentional Latent Graphs The goal of the encoder to infer latent interaction graph. To achieve this, we use GNN on fully-connected directed graph without self-loops, where each vertex represents an agent. Let x, denote the feature vector of agent j across all input frames (x),...,xx] x The GNN embeds agent trajectories (e.g. Xj) into edge embeddings, which are then passed through MLP to derive unnormalized edge weights (e.g. @(r,j)) ( is vector with dimensionality K, hyperparameter that determines the number of layers in the latent graph. Details of the GNN can be found in the Appendix.
Previous approaches attempt to classify edges into one out of the K possible edge types. They derive Zyy by applying softmax on (a.)) such that Σ z = 1 with z the k-th layer of the latent graph. We propose a novel way to construct the latent space using multiplex attentional graph. Instead of performing edge-type classification, we perform multi-class prediction on edge embeddings, yielding multiplexed layers, and introduce an attention mechanism between agents within each of these layers. That is,
exp(e( (a.1)
(0 X1:7,) softmax, (e EJEN, exp(e
vk= K,
(2)
where Nj denotes the neighbors of agent j (we assume a fully connected graph; thus, N, includes all other agents except itself). Note that Zij is different than Zyz. With Eq. layers of the latent graph are separable by construction. By separable, we mean that individual layers the latent graph (e.g. z! and =²) can be inferred by separate encoders that does not share any parameters. Intuitively, a multiplex latent space allows us to model independent interactions naturally, and the attention mechanism captures strengths of relations.
Decoder The task of the decoder is to predict the dynamics of the system using the latent interaction graph and the past dynamics. That is, we want to model (T IxI:T. z). One common issue when training such a decoder that it might learn to ignore the latent variables while achicving only marginally worse prediction loss. To avoid this problem, our decoder first learns agent-centric representations h femb((x (x,) (i.e. representations are embedded individually for each agent), and then uses these as node embeddings to perform message passing. The decoder thus has to rely on the inferred latent graph exchange information between agents. After learning agent-centric representations, we use GraphGRU [9] and a readout function to decode future trajectories:
h +1 GraphGRU(ht K' .2), x x + four (oou(hhtt (h2+1)
(3)
h h+ GraphGRU((,, h z), x x fout foue(h (h),²)
(4)
The details of GraphGRU block are elaborated in the Appendix. Unlike a typical Graph Attention Network [1] where the graph at every message passing layer is inferred, the same (multiplex) graph

---

trajectories
predicted
truth
Ground Truth
MLP
RFM
Ours
Figure Visualization the latent graph and agent trajectories of the Social Navigation Environment. (top) Predicted trajectories. The smaller the circle, the further it is into the future. The leftmost column shows ground truth trajectories and the ground truth graph used to simulate those trajectories. (bottom) Inferred latent graphs. Edge strength between agent and is represented by darkness of the cell at row and column j. The red agent's relational prediction inaccurate with RFM-in the row highlighted by the arrow, the green agent is incorrectly given higher weight than the blue agent-aa and thus the predicted trajectories deviate from the ground truth, especially on long-horizon predictions.
used across all decoding steps. We can re-evaluate the encoder for every time step to obtain dynamically changing latent graph, however we leave this to future work.
3.2 Progressive Layer Training
The role of the encoder is to infer set of graph bases (z), 2?...) that reflect the underlying dynamics of the environment. Since the messages are first passed within layers before summing at the end, would be useful for the latent graph layers to represent different types of interactions between agents.
However, it is difficult for the model to learn all the interactions, let alone have them be disentangled. Disentanglement is typically achieved by imposing additional loss, yet these approaches tend to come with the trade-off decreased reconstruction quality [1O] 18 [24]. l. Motivated by curriculum learning [5] and progressive learning [28] (or more broadly the concept of "starting easy"),
propose to overcome this trade-off with Progressive Layer Training (PITT) a strategy of learning set of good graph bases by learning the high-level and most consequential interactions first and then progressively growing the network to model lower-level and more intricate interactions. We first train IMMA with a single-layered latent graph zt, then add second layer z² by training a second encoder while freezing the weights of the first. This process can be repeated until no further improvement observed. The fact that we can choose have a separable latent space and do not have to share any parameters between these encoders is result of our latent space being multiplexed. Directly adding new components to trained network can introduce sudden shock to the gradient. To stabilize training, we adopt "fade-in" (20) to smoothly blend the new and existing network components. Refer the Appendix for more details.
Experiments
We design experiments to answer the following questions:
Q1: Does our approach (IMMA PLT) consistently outperform prior methods across diverse set of social multi-agent environments and benchmarks?
Q2: Does the use of multiplex attentional latent graph give more interpretable abstractions?
Q3: How much does each component contribute the final performance?
Q4: Is approach sample efficient? Can it generalize well to novel environments?
5

---

To answer QI, we test our approach in three multi-agent environments that each exhibited multiple types of social interaction: our simulated Social Navigation Enviroument, PHASE [00, and the NBA dataset (used in
We develop the first benchmark ourselves, while the latter two are established benchmarks. For each environment, the model receives multi-agent trajectories of length 24 as observation and predicts 10 future time steps. For Social Navigation we use 100k multi-agent trajectories (in total for training. validation, and testing), for NBA dataset we use 300k multi-agent trajectories, and for PHASE we use 836 multi-agent trajectories. We also perform data augmentation, as described the Appendix. To answer Q2, we evaluate the inferred interaction graphs for the Social Navigation Environment and PHASE, since we have the ground truth interaction graph. Note that we are not supervising on any explicit relations of agents. We further conduct qualitative experiments to understand the abstractions provided by the latent graph. To answer Q3, we conduct an ablation study. For Q4, we perform a sample efficiency experiment and an experiment of zero-shot generalization. Experiments for Q3 and Q4 are conducted the Social Navigation Environment as we have full control over it.
[22 27 47 15]
4.1 Experiment Setup
Here we describe the environments and baselines. More details are in the Appendix.
Social Navigation Environment The Social Navigation Environment simulates a socially inter- active crowd where agents seek to meet up with other agents while simultaneously navigating to avoid collisions within the crowd (as schematized in Fig. D.. This environment allows fine-grained control by simulating lower-level interaction dynamics using Optimal Reciprocal Collision Avoidance (ORCA, (6) like many other crowd navigation works 88 19. With this environment, we can easily vary environmental parameters to generate diverse trajectories for generalization experiment. We experiment primarily with an environment consisting of five agents. For each simulated episode, social interaction graph of the agents is randomly generated: each agent is assigned to be the follower of another agent. Agents are initialized to positions along the edge of a circle. Dynamics is simulated using ORCA by assigning the position of each agent's leader (at each time step) as the target destination of that agent. These environment settings yield reasonably complex trajectories that were clearly influenced by the social interaction graph and collision avoidance, as seen in example trajectories of Fig.
PHASE dataset PHASE 30 is simulator for physically-grounded abstract social events, in which social recognition and social prediction algorithms can be henchmarked. In PHASE, two animated agents move in a continuous 2D space and interact with each other and other objects (2 balls). Their actions are generated using a physics engine and a hierarchical planner [30). We choose the "collaboration" task for our experiment, in which two agents collaborate to move one of the balls to pre-specified landmark location. Diverse trials are generated by randomizing the agents' and balls' starting locations as well as orientations, target ball, and target landmark. Note that this dataset is substantially smaller than the two other datasets.
NBA dataset The National Basketball Association (NBA) dataset consists of position trajectories for ten interacting basketball players and the ball and uses real-world player tracking data. The task
of
observed trajectories
observed trajectories predicted trajectories
predicted trajectories
ground
ground truth
Figure Qualitative analysis of our approach on PHASE data (left) and the NBA dataset (right). For
the NBA visualization, the orange agent is the basketball, and colors represent teams.

---

Table 1: Trajectory prediction results (mean + std, over runs) on all three datasets.
Mama
Methods
IMMA
MLP
GST_LSTM H1
NRI
EvelreStogii
RFM (kkpp 11
RFM 99]
2NRI
5G
385
MG y PLT
Nasiation
AI36
0.34 0.000
0.306 0.08
8217 1 0.005
6.100 8.00
0.100
0.02
10155
0.03
10.151
0.001
0.166 1 0.002
0.18
00004
0.139
0.81
FDE
0.13 à 0002
0.527 0:000
0.360 + 0.009
0,321 1L00S
0.325 U1044
0.317 0.00
00300
0.008
0.327 + 0.00
0.2M
(1005
1.279 0
ADE
1.004
0996
0.815
0.000
FDE
1,003 0.024
2.527 0.063
1,772 à 0.024
1,522 0.019
1.581
0.000
1880 0.042
1607 0.044
1.385 à 0.044
1,503 0.048
1.484 0.014
NBA daaset
1
0.015
0,019
Table 2: Relational inference accuracy on the Social Navigation environment and PHASE
Matric
Boodite Martood
IMMA
(1]
ExcheGrngh
Oseead Nangation Gragh Acc (921 (PEAASI Ezoph (S1
21.3 D47
57.8 0.10
78.33 D096
Tods 1:045
71.51
0.9
15:77
1:80
148
77466
0.16
81.38
8:29
5241
$5.90
98.96
5121 1.85
to predict 10 future time steps (4s) based on a history of 24 time steps (10s). Team information is an additional input feature. The unit reported is meters.
Baselines
We consider the following basclines and ablations.
MLP: large MLP network that takes the concatenated features of all agents across all timesteps
as input and output trajectory predictions of all agents at the same time.
GAT I.TM: network with Graph Attention layers 41 followed by LSTMs.
NRI [22]:
VAE model with recurrent GNN modules.
EvolveGraph [19 model that expands upon the framework of NRI and introduces double-
stage training pipeline to account for an evolving interaction graph.
RFM [39] recurrent GNN model of similar architecture NRI, but with supervised loss instead of a variational lower bound loss.
RFM (skip 1): RFM model with the first edge type is "hard-coded" "non-edge" (i.e. messages are passed along edges of the first layer).
INRI 4 NRI model that uses a multiplex latent graph structure (i.e., sigmoid activation).
IMMA (SG): IMMA with a single layer of attentional latent graph.
IMMA (MG): IMMA with multiple layers of attentional latent graphs (a multiplex graph) trained simultaneously. without PLT.
We are aware that there are other relevant baselines such as Social-LSTM D. Social-GAN 13 and
Trajectron++ (35] etc. We choose to compare with EvolveGraph, because it consistently outperforms
these models across datasets
4.2 Trajectory Prediction
To answer Q1, we evaluate all models on their prediction performance of agent position on future trajectories with two metries: average displacement error (ADE) and final displacement error (FDE). Both of these are metrics widely used in trajectory prediction literature: ADE is defined as the average deviated distance all entities within the prediction horizon, and FDE is defined as the deviated distance of all entities at the last predicted time step.
The results on the Social Navigation Environment, PHASE, and the NBA dataset are shown in table I We found that using multiplex attentional latent graph with progressive layer training (MG w/ PLT) yields the best performance on all three datasets, outperforming previous tfte-of-the-art models. On the Social Navigation Environment and the NBA dataset, IMMA without progressive layer training (MG) still performs better than all baselines, showing the effectiveness using multiplex attentional graph as latent structure. We visualize the prediction results of our method and the best performing baseline (RFM) in Fig. on the Social Navigation Environment. Visualizations of PHASE and the NBA dataset are shown Fig. (more can be found in the Appendix).
7

---

4.3 Relational Analysis and Interpretability
To answer Q2, we conduct experiments to understand what abstractions can be extracted from the inferred multiplex attentional graph. We first compare the performance of our approach to the same set of baselines used in the previous experiment on relational inference. Intuitively, this is to evaluate how well models can answer the question "What causes an agent behave this way?" Refer to the Appendix for details on how we calculate the graph accuracy. We conduct this experiment in the Social Navigation Environment and PHASE, since we have access to the ground-truth social interaction graph for these two environments.
IMMA more accurately estimates the underlying social interaction graph (Table The results can also explain the benefits of training IMMA with PLT: by leveraging the graph accuracy prediction that arises from using single graph layer, but with the expressive power of a full multiplex graph.
To further understand how IMMA uses the inferred social graph, we assess how the trajectory predictions are altered if we manually manipulate the latent graph. More formally, we ask our decoder to generate new trajectories conditioned on the new latent graph pe px +1:4. +T' [X 1:Th, 2==w). ). In Fig. []] we see that with IMMA, changing the leader of an agent clearly alters the predicted trajectory to target that new leader while keeping the predictions for other agents intact, whereas the generated trajectories of the baseline consist of unrealistic turns (red agent 0) and the predicted trajectories of other agents deteriorates at the same time. Using the trajectories shown in Fig. [ we ran a human study and our model's prediction is judged more reasonable than the RFM model's in 76.67% of the trials. Find more details of the human study in Appendix section C.
4.4 Ablation Study
To answer Q3, we experiment with three ablated versions of our model. We ensure that all ablated versions have around the same amount of total parameters and have two layers of latent graphs. Additionally, we measure the information dependency between the two layers of latent graph with Normalized Mutual Information (NMI) score (details in the Appendix). Results are shown in Table 3 The first column in the table shows the performance of model where the latent graph structure are edge types between all pairs of agents. It has an NMI score of 1.0 (complete dependency) because the second layer of latent graph is simply one minus the first layer of latent graph. We observe that the biggest gain arose from using a multiplex attention graph and Progressive Layer Training furthers the performance. We also see that both components contribute to decreased information dependency
Original Prediction
Conditional Generation
Leader of agent @
change leader of ! to
change leader of @ to x
change leader of O to ④
RFM
Ours
observed trajectories
predicted trajectories
ground truth
Figure 5: Conditional generation results of the RFM model and our method (IMMA w/ PLT).
8

---

Table 3: Ablation study results.
Ablated versions of IMMA
Ours (IMMA w/ PLT)
Multiplex Attention Graph
X
V
Disentanglement Loss [T0]
Progressive Layered Training
X
X
X
ADE FDE 1
0.317
0.005
0.294 0.006
0.386
0.006
0.279 0.002
Graph Acc. (9))
71.53 = 0.56
77.98 = 0.16
69.47 = 0.72
81.38 = 0.29
NMI score
0.042
0.13
Table 4: Zero-shot generalization results on the Social Navigation Environment.
Hactre Kathedk
DMMA (Uan)
Maxus
ML.
GAL [1]
NR 22
IUM
RFM 24
PNRI [1
MG
MtC PLT
quoe
ADE RDE
0.300 = 0.006
0.36 + 0.009
0.300 0.015
0.29==000 0:03
0.309 = 0.002
6,005 = 0:001
0.20 = 0.003
0.313 + 0.00
0.197 0/033
0.092 0.001
B.632 = 0.01
0635 + 0:01
0.465 = 0.00
0.421 0.00
0.18 = 0.005
0.11
0.005
0.410 0.000
0419 à 0.00
0.389 9/06
0.363 0.002
Gruph
22
76:00
ADD
11360 & 0180
0:918 1 0010
11317 - 0181
0.155 01N1
11S 0100
0.130 1 8000
1.151
1 0.003
0.E58
1100
1.1.9 0
8.139
0.30 0.000
0.36
0002
0.166
0.00
0.314 0.00
11312
0.001
0.003
8.00
0.308
0.003
01513
11.00
0,00
6:279 IL002
Coop
21.0 0.67
57.18 0.14
12,08
59.99
71
13.97
80.0
1.44
78.35
81.38 0.29
ADD
LA4e 0.305
[1327 - (113)
0355 0112
1141
1 0116
0:311
R015
0:300 1 0.013
00211 11001
11:10 0.0000
8.195 ILOI1
0.315
0.033
0.968
0.702
0.762
0.009
0.002
0,486
Copb.
19:80
34.57 0:00
10.56 0:16
0:00
51.57
19:11
5.00
07.05 1.30
60:45 +0.55
64.25 + 6,31
between two latent graph layers without explicitly encouraging disentanglement. Although imposing an additional loss to disentangle between two latent graph layers does yield the smallest NMI score, it comes with the trade-off of decreased forecasting accuracy as observed in other works [0 18
4.5 Sample Efficiency and Generalization
To answer Q4, investigate sample efficiency and observe that our method consistently outperforms the strongest baseline (RFM) on smaller sample complexity regimes (Fig. (6). We test zero-shot generalization performance in response to three environment modifications: agents moving twice as fast (2x speed), crowding the agents into half as much space (2x smaller environment), and doubling the number of agents (2x more agents). As shown Table MG and MG+PLT versions of IMMA outperform the baselines in all metrics. Additionally, as shown in Appendix Table the change in performance on the generalization scenarios relative to the original environment was similar or better than the baselines in all metrics
0.35
RFM
RFM
80
Ours
Ours
0.30
Graph Acc. Acc.
60
ADF
0.25
50
0.20
0.4
RFM
0.15
0.3
J0-
Ours
dataset size
dataset size
dataset size
Figure 6: Sample efficiency of RFM and Ours. Lower ADEs and FDEs and higher graph accuracies
are better.

---

Conclusion
Multi-agent dynamics can often be complex as a result of many layers of underlying social interactions. Agents can be influenced by multiple independent types of relationships with each agent, leading to properties (such as intentionality or cooperation) that are often absent in simpler physical systems. We developed a method that uses multiplex attentional graphs as latent representations model the dynamics that can arise from such multi-layered multi-agent interactions. Addressing key aspects of social interaction, the dynamics of social multi- systems. We foresee minimal direct negative societal impact please see Appendix for further discussion. Future work will extend this approach active learning settings and investigate using our method to guide model-based reinforcement learning agents.
method consistently outperforms other oofte--of-he-attt methods in modeling
6
Acknowledgements
I.K. is a Merck Awardee of the Life Sciences Research Foundation, and a Wu Tsai Stanford Neuro- sciences Institute Interdisciplinary Scholar. This work is also in part funded by the Toyota Research Institute (TRI), the Stanford Institute for Hluman-Centered AI (HAD) ONR MURI N00014-22-1- 2740, Meta, Salesforce, and Samsung. N.H. is further funded by the GSE Transforming Learning Accelerator.
References
[1] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robiequet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pages 961-971, 2016.
[2] Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprini arXiv: /252, 2017.
[3] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in Neural Information Processing Systems (NeurIPS), volume 29, 2016.
[4] Peter W Battaglia, Jessica Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1rXiv.1806.01261, 2018.
[5] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning (ICML), pages 41-48, 2009.
[6] Jur van den Berg, Stephen J Guy, Ming Lin, and Dinesh Manocha. Reciprocal n-body collision avoidance. In Robotics Research, pages 3-19. Springer, 2011.
[7] Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang. Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time-series forecasting. Advances in Neural Information Processing Systems, 33:17766-17778. 2020.
[8] Changan Chen, Sha Hu, Payam Nikdel, Greg Mori, and Manolis Savva. Relational graph learning for crowd navigation. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10007-10013, 2020.
[9] Changan Chen, Yuejiang Liu, Sven Kreiss, and Alexandre Alahi. Crowd-robot interaction: Crowd-aware robot navigation with iitention-base deep reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 6015-6022. IEEE, 2019.
[10] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances Neural Information Processing Systems (NeurIPS), volume 31, 2018.
10

---

[11] Nachiket Deo and Mohan M Trivedi. Convolutional social pooling for vehicle trajectory prediction. IEEE Conference on Computer Vision and Patter Recognition Workshops, pages 1468-1476, 2018.
[12] Colin Graber and Alexander Schwing. Dynamic neural relational inference for forecasting trajectories. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages
[13] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pages 2255-2264 2018.
[14] Zaynab Hammoud and Frank Kramer. Multilayer networks: aspects, implementations, and application in biomedicine. Big Data Analytics, 5(1):1-18, 2020.
[15] Sandro Hauri, Nemanja Djurie, Vladan Radosavljevic, and Slobodan Vucetic. Multi-modal trajectory prediction of nba players. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1640-1649, 2021.
[16] Fritz Heider and Mariann Simmel. An experimental study of apparent behavior. The American Journal of Psychology, 57(2):243 259,, 1944.
[17] Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics. Physical review E. 51(5):4282, 1995.
[18] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations (ICLR), 2017.
[19] Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. Stgat: Modeling spatial- temporal interactions for human trajectory prediction. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 6272-6281, 2019.
[20] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv: 2017.
[21] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013.
[22] Thomas N. Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard S. Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning (ICML), pages 2693-2702, 2018.
[23] Hoang M Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In International Conference on Machine Learning, pages 1995-2003, PMLR, 2017.
[24] José Lezama. Overcoming the disentanglement vs reconstruction trade-off via jacobian supervi- sion. In International Conference on Learning Representations, 2018.
[25] Jiachen Li, Hengbo Ma, and Masayoshi Tomizuka. Conditional generative neural system for probabilistic trajectory prediction. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6150-6156. IEEE, 2019.
[26] Jiachen Li, Hengbo Ma, Zhihao Zhang, Jinning Li, and Masayoshi Tomizuka. Spatio-temporal graph dual-attention network for multi-agent prediction and tracking. IEEE Transactions on Intelligent Transportation Systems, 2021.
[27] Jiachen Li, Fan Yang. Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. Advances in Neural Information Processing Systems (NeurIPS), 13:19783-19794, 2020.
[28] Zhiyuan Li, Jaideep Vitthll Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progressive learning and disentanglement of hierarchical representations. arXiv preprint arXiv:2002. 10549, 2020.
11

---

[29] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgen:: social spatio-temporal graph convolutional neural network for human trajectory prediction. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pages 11424-14432, 2020.
[30] Aviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu, and Joshua B Tenenbaum. Phase: Physically-groundee abstract social events for machine social perception. In AAAI Conference on Artificial Intelligence (AAAI), pages 845-853, 2021.
[31] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems (Neur/PS)Worksinnn 2017.
[32] David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1(4):515-526, 1978.
[33] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International Conference on Machine Learning (ICML), pages 4218-4227. PMLR, 2018.
[34] Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. International Journal of Robotics Research, 39(8):995-933 2020.
[35] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In European Conference on Computer Vision, pages 683-700, 2020.
[36] Alvaro Sanchez-Gonzalee Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning simulate complex physics with graph networks. In International Conference on Machine Learning (ICML), pages 4459-8468. PMLR, 2020.
[37] Wenling Shang, Alex Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Learning world graphs accelerate hierarchical reinforcement learning, arXiv preprint 2019.
[38] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
[39] Andrea Tacchetti, H. Francis Song, Pedro A. M. Mediano, Vinicius Zambaldi, János Kramár, Neil C. Rabinowitz, Thore Graepel, Matthew Botvinick, and Peter W. Battaglia. Relational for- ward models for multi-agent learning. In International Conference on Learning Representations (ICLR), 2019.
[40] Annemarie Turnwald and Dirk Wollherr. Human-like motion planning based on game theoretic decision making. International Journal of Social Robotics, 11(1):111-170 2019.
[41] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint
2017.
[42] Anirudh Vemula, Katharina Muelling, and Jean Oh. Social attention: Modeling attention in human crowds. In IEEE International Conference on Robotics and Automation (ICRA), pages 4601-4607. IEEE, 2018
[43] Ezra Webb, Ben Day, Helena Andres-Terre, and Pietro Lió. Factorised neural relational inference for multi-interaction systems. arXiv preprint A9i::19055.8721, 2019.
[44] William Whitney. Disentangled representations in neural models. arXiv preprint
arXiv: 16022.238333 2016.
[45] Amanda L Woodward, Jessica A Sommerville, Sarah Gerson, Annette ME Henderson, and Jennifer Buresh, The emergence of intention attribution in infancy. Psychology of Learning and Motivation, 51:187-222, 2009.
12

---

[46] Eiling Yee. Abstraction and concepts: when, how, where, what and why? Language, Cognition and Neuroscience. 34(10):1577-1255 2019.
[47] Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, and Patrick Lucey, Generating multi-agent trajectories using programmatic weak supervision. arXiv preprint arXiv: (a0n:080337711 2018.
[48] Lunjun Zhang, Ge Yang, and Bradly C Stadie. World model as graph: Learning latent landmarks for planning. International Conference on Machine Learning (ICML), pages 12611-12620, 2021.
[49] Stephan Zheng, Yisong Yue, and Jennifer Hobbs. Generating long-term trajectories using deep hierarchical networks, In Advances in Neural Information Processing Systems (NeurIPS), volume 29, 2016.
13


Reviews:
[{"review_id": "rRBXK81luln", "reviewer": "Reviewer_Q2Zh", "summary": "This work tackles the problem of modeling multi-agent systems that contain a high degree of interaction between agents. To model such systems, this work proposes Interaction Modeling with Multiplex Attention (IMMA), which uses a multiplex graph latent structure to model different types of interactions in different layers of the multiplex. In addition, attention graph layers are used to capture the strength of the relations. Furthermore, thanks to the structure of the multiplex latent layers, this work proposes progressive layer training (PLT) where one can train different interaction layers separately, and iteratively adding different latent layers. The proposed approach is evaluated on three different datasets showcasing strong results on all. Furthermore, ablation studies show how having multiple latent layers and/or having PLT help the model’s prediction performance. Furthermore, the Social Navigation Environment provides the ground-truth interaction structure of each scene. This work shows that the predicted latent structure predicts ground truth more accurately than prior approaches.", "questions": "- I’m wondering where the results of EvolveGraph and NRI are obtained. EvolveGraph reports minADE_20/minFDE_20, and use 5 historical timesteps (2.0s) to make the prediction over the next 4 seconds. Could the authors clarify if this was their own implementation?\n- Furthermore, there are multiple other works that report results on the NBA dataset, following the same configuration mentioned in my previous questions. These include STAR[1], PECNet[2], NMMP[3], GroupNet[4]. How come comparisons were not performed against these ones?\n- Related to my previous question, can IMMA produce multiple modes of predictions or is it always unimodal?\n- I’m confused about the footnote (2) on page 3 which states that one can ignore the KL divergence term. I have looked through the provided reference and could not find why it is possible to ignore this term. Can the authors clarify this?\n- Figure 5: it is not clear to me why the predictions produced by IMMA are better than the baseline when the latent graph is manipulated. - What makes the new generated turns unrealistic in the baseline when compared to IMMA’s predictions? I will say that it is an interesting feature that the predictions of the other agents do not change.\n\nReferences:\n\n[1] Yu, C., Ma, X., Ren, J., Zhao, H. and Yi, S., 2020, August. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In European Conference on Computer Vision (pp. 507-523). Springer, Cham.\n\n[2] Mangalam, K., Girase, H., Agarwal, S., Lee, K.H., Adeli, E., Malik, J. and Gaidon, A., 2020, August. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In European conference on computer vision (pp. 759-776). Springer, Cham.\n\n[3] Hu, Y., Chen, S., Zhang, Y. and Gu, X., 2020. Collaborative motion prediction via neural motion message passing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 6319-6328).\n\n[4] Xu, C., Li, M., Ni, Z., Zhang, Y. and Chen, S., 2022. GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6498-6507).\n", "limitations": "These are adequately addressed in the conclusion and the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well organized and well-written.\n- I believe the application of multiplex attention in the “latent space” is novel to the problem of multi-agent motion forecasting with interaction modeling.\n- The results are strong and convincing on multiple datasets. The ablation studies provide further evidence of the utility of having multiple levels of attention. \n\nWeaknesses:\n- In the appendix, it is mentioned that training is performed with $q_\\phi(z|X^{1:T_h})$. This, combined with the footnote on page 3 (on which I have a question below), it seems like the proposed approach should not be formulated/classified as a CVAE, and is more of a model which has a specific hidden structure on the bottleneck that is not regularized by any external forces (e.g., KL divergence).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pURzF73UVDk", "reviewer": "Reviewer_PvS1", "summary": "The authors proposed a prediction model that uses a multiplex latent graph to represent multiple types of interactions and attention to account for different relations called Interaction Modeling with Multiplex Attention (IMMA). They also introduce a training strategy called Progressive Layer Training (PLT). The experimental results showed that their approach outperformed baseline models in trajectory forecasting and relation inference, spanning three multi-agent scenarios: social navigation, cooperative task achievement, and team sports. They also demonstrated better zero-shot generalization results in the social navigation environment.\n\n", "questions": "1. I don’t know all of the trajectory prediction studies, but multilayer graph neural networks in trajectory prediction may not be a new idea and the following studies were quickly found [1] [2]. \n2. The idea of PLT may seem to be a special case of the original paper (Li et al. 2020 ICLR, not arxiv). The authors can clarify the difference from the original idea.\n3. The strength of the method may be the decomposition of multilayer relations like Fig.1. However, the results including Fig 3 may show only one layer. How did the authors merge it? And I want to know whether each layer can extract multiple relations like Fig. 1 or not. \n\n[1] Inhwan Bae and Hae-Gon Jeon, Disentangled Multi-Relational Graph Convolutional Network for Pedestrian Trajectory Prediction, AAAI, 2021\n[2] Rui Zhou et al. Grouptron: Dynamic Multi-Scale Graph Convolutional Networks for Group-Aware Dense Crowd Trajectory Forecasting, ICRA 2022\n", "limitations": "The authors addressed the limitations and potential negative societal impact of their work. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Stremgth:\n* They proposed a prediction model that combines multilayer graph neural networks and PLT for interpreting the multiplexed latent structure. \n* The experiments showed that their approach outperformed baseline models in trajectory forecasting and relation inference, spanning three multi-agent scenarios. They also demonstrated superior sample efficiency and better zero-shot generalization results.\n\nWeakness:\n* The novelty of this method seems to be combining multilayer graph neural networks and PLT, but the authors mentioned that each component has novelty, which may be exaggerated. I asked this question in the following “Questions”. \n* The result was clear but there was no code available (I saw “Checklist”) and there were some questions in the experiment description. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SxicR6ApZzo", "reviewer": "Reviewer_FJKV", "summary": "This paper presents a new method for modeling and predicting multi-agent interaction. Observing that such interactions could be affected by multiple factors such as target destinations and collision avoidance during crowd navigation tasks, the key idea was to model them using multiplex graphs. Specifically, rather than classifying types of edges of graphs as done in existing work, the proposed approach instead perform multi-class predictions for multiple edge types and construct multiplex attention matrices. To make latent graph layers diverse, the proposed work introduces a progressive learning scheme (progressive layer training) that gradually increases the number of graph layers to learn. Experimental results on multiple datasets show the effectiveness of the proposed approach over existing methods.\n", "questions": "Some suggestions:\n\n- Figure 1 could be improved to precisely describe what the proposed method actually learns. Does it really learn to model collision avoidance, or does it just learn who follows whom?\n- Honestly, I can see few differences between RFM and Ours in Figure 5 or cannot tell which of results from RFM and those from Ours are more natural.\n- What are the limitations and failure cases for the proposed method?", "limitations": "As also described above, the paper currently has few discussions on the limitations and typical failure cases for the proposed approach.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- **Originality**: modeling multi-agent interaction using multiplex graphs seems novel, at least for trajectory forecasting tasks. It is well motivated by the observations that human interactions can be affected by multiple factors.\n- **Quality**: I agree that the proposed approach is well designed and technically sound. The questions that the paper addresses are clearly presented (Q1-Q4 in Section 4) and answered in the experimental results. Experiment details as well as potential societal impacts are clearly described in the appendix.\n- **Clarity**: Overall the paper is clearly written and easy to follow.\n- **Significance**: I believe that the proposed work is relatively significant mainly because it outperformed existing methods on multiple datasets. The zero-shot generalization experiment was interesting and showing the usefulness of the proposed approach.\n\n## Weaknesses\n- The motivating example for multiple factors affecting interactions, i.e., target friend and collision avoidance in Figure 1, may be a bit confusing. If I understand it correctly, multiplex graphs are built rather in a bottom-up fashion in the proposed method, where we cannot ensure that each graph layer has such semantic meaning. In fact, I'm still a bit confused what the multiplex graph actually learned. Is this just a leading agent as shown in Figure 5 or something beyond?\n- While it was great that the proposed method performs very well, I would also like to see typical failure cases. Are there some cases where interaction modeling or prediction are likely to fail?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1B0rPRpbjHp", "reviewer": "Reviewer_diNz", "summary": "The paper proposes a multiplex attention method for multi-agent interaction modeling. The main motivation is that there can be more than one interaction type between agents. To address this, the paper proposes to use multiple latent graphs to encode the different interactions between agents and employs the VAE framework to infer the latent codes. To ease learning, the paper also proposes progressive layer training, which is a curriculum learning method that gradually adds latent graphs to the model. Evaluations are performed on multi-agent trajectory forecasting using one in-house simulated dataset and two public datasets, PHASE and NBA. The method outperforms prior methods in forecasting accuracy, relational inference accuracy, and sample efficiency.", "questions": "- In Fig. 3, how are the latent codes identified for changing the leader? If I understand correctly, there should be multiple latent codes. A more compelling experiment for interpretability is probably to show different latent codes can lead to different types of interactions (besides leader-follower). This also validates the key motivation of this paper better.\n- Are latent edges directional? i.e., is $z_{ij}$ different than $z_{ji}$? Eq (2) only makes sense if edges are directional, otherwise $i$ and $j$ are not symmetric when computing $z_{ij}$.\n- Zero-shot generalization experiments are not convincing in my opinion. The performance of all methods has dropped but in some cases the performance gap between the current method and the baselines becomes smaller. Normally, the method should outperform the baselines by a similar percentage in this new setting. Doesn’t this indicate the method actually generalizes worse than some of the baselines?\n- How are column agents selected for the relational inference in Table 2? Since the method has multiple latent graphs, do you choose the largest latent code as the latent code for each column agent?\n\n", "limitations": "- The paper didn’t really talk about its limitation despite indicating in the form it has done so.\n- One clear limitation is that it cannot handle changing interaction types. As the multi-agent interaction evolves over time, their interaction could also evolve. For example, a follower passes a leader to become the new leader. Recent work on dynamic NRI [5, 6] has attempted to address this limitation.\n\n[5] Graber, Colin, and Alexander Schwing. \"Dynamic neural relational inference for forecasting trajectories.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.\n\n[6] Xiao, Ruichao, Manish Kumar Singh, and Rose Yu. \"Dynamic Relational Inference in Multi-Agent Trajectories.\" ICLR 2021.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n\n- The paper is generally well-written and ways to understand. The motivation is clear for encoding multiple types of interaction between agents and their relative strength.\n- The paper performed extensive experiments and analyses on three datasets. Both trajectory forecasting and relation inference are evaluated. The latent graph manipulation experiments are interesting which show the latent codes encode leader-follower behaviors.\n- The paper outperforms the baselines significantly.\n\n**Weakness:**\n\n- The main idea, multiplex attention, of this paper is quite **similar to factorized NRI [1]**, where multiple latent graphs are used in a variational inference framework. The technical novelty of this paper is limited. In some sense, the multiplex attention is not a big change from NRI since the latent code itself is multi-dimensional (therefore each dimension can be treated as a separate latent code) and the main difference of the proposed approach is to not share encoder weights.\n- The paper didn’t compare with strong multi-agent trajectory forecasting baselines on classic trajectory forecasting benchmarks such as ETH/UCY where a large number of agents are interacting with each other. The paper could compare with SOTA methods such as [2, 3, 4].\n- Some design choices are not well motivated. What is the motivation for soft-max across agents in Eq (2)? This constrains the interaction strength from $i$ to other agents to sum to 1 for interaction $k$. However, agent $i$ can have strong interaction strength with multiple different agents for interaction $k$. Therefore, I wonder why the method doesn’t just formulate inference of $z_{ij}^k$ as a binary classification (this interaction exists or not) instead of multi-class classification across other agents.\n\n[1] Webb, Ezra, et al. \"Factorised neural relational inference for multi-interaction systems.\" ICML 2019 Workshops.\n\n[2] Mangalam, Karttikeya, et al. \"From goals, waypoints & paths to long term human trajectory forecasting.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[3] Yuan, Ye, et al. \"Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[4] Wang, Chuhua, et al. \"Stepwise goal-driven networks for trajectory prediction.\" IEEE Robotics and Automation Letters 7.2 (2022): 2716-2723.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept