Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work tackles the problem of modeling multi-agent systems that contain a high degree of interaction between agents. To model such systems, this work proposes Interaction Modeling with Multiplex Attention (IMMA), which uses a multiplex graph latent structure to model different types of interactions in different layers of the multiplex. In addition, attention graph layers are used to capture the strength of the relations. Furthermore, thanks to the structure of the multiplex latent layers, this work proposes progressive layer training (PLT) where one can train different interaction layers separately, and iteratively adding different latent layers. The proposed approach is evaluated on three different datasets showcasing strong results on all. Furthermore, ablation studies show how having multiple latent layers and/or having PLT help the model’s prediction performance. Furthermore, the Social Navigation Environment provides the ground-truth interaction structure of each scene. This work shows that the predicted latent structure predicts ground truth more accurately than prior approaches.
strengths_and_weaknesses: Strengths:
- The paper is well organized and well-written.
- I believe the application of multiplex attention in the “latent space” is novel to the problem of multi-agent motion forecasting with interaction modeling.
- The results are strong and convincing on multiple datasets. The ablation studies provide further evidence of the utility of having multiple levels of attention. 

Weaknesses:
- In the appendix, it is mentioned that training is performed with $q_\phi(z|X^{1:T_h})$. This, combined with the footnote on page 3 (on which I have a question below), it seems like the proposed approach should not be formulated/classified as a CVAE, and is more of a model which has a specific hidden structure on the bottleneck that is not regularized by any external forces (e.g., KL divergence).
questions: - I’m wondering where the results of EvolveGraph and NRI are obtained. EvolveGraph reports minADE_20/minFDE_20, and use 5 historical timesteps (2.0s) to make the prediction over the next 4 seconds. Could the authors clarify if this was their own implementation?
- Furthermore, there are multiple other works that report results on the NBA dataset, following the same configuration mentioned in my previous questions. These include STAR[1], PECNet[2], NMMP[3], GroupNet[4]. How come comparisons were not performed against these ones?
- Related to my previous question, can IMMA produce multiple modes of predictions or is it always unimodal?
- I’m confused about the footnote (2) on page 3 which states that one can ignore the KL divergence term. I have looked through the provided reference and could not find why it is possible to ignore this term. Can the authors clarify this?
- Figure 5: it is not clear to me why the predictions produced by IMMA are better than the baseline when the latent graph is manipulated. - What makes the new generated turns unrealistic in the baseline when compared to IMMA’s predictions? I will say that it is an interesting feature that the predictions of the other agents do not change.

References:

[1] Yu, C., Ma, X., Ren, J., Zhao, H. and Yi, S., 2020, August. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In European Conference on Computer Vision (pp. 507-523). Springer, Cham.

[2] Mangalam, K., Girase, H., Agarwal, S., Lee, K.H., Adeli, E., Malik, J. and Gaidon, A., 2020, August. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In European conference on computer vision (pp. 759-776). Springer, Cham.

[3] Hu, Y., Chen, S., Zhang, Y. and Gu, X., 2020. Collaborative motion prediction via neural motion message passing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 6319-6328).

[4] Xu, C., Li, M., Ni, Z., Zhang, Y. and Chen, S., 2022. GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6498-6507).

limitations: These are adequately addressed in the conclusion and the appendix.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a multiplex attention method for multi-agent interaction modeling. The main motivation is that there can be more than one interaction type between agents. To address this, the paper proposes to use multiple latent graphs to encode the different interactions between agents and employs the VAE framework to infer the latent codes. To ease learning, the paper also proposes progressive layer training, which is a curriculum learning method that gradually adds latent graphs to the model. Evaluations are performed on multi-agent trajectory forecasting using one in-house simulated dataset and two public datasets, PHASE and NBA. The method outperforms prior methods in forecasting accuracy, relational inference accuracy, and sample efficiency.
strengths_and_weaknesses: **Strength:**

- The paper is generally well-written and ways to understand. The motivation is clear for encoding multiple types of interaction between agents and their relative strength.
- The paper performed extensive experiments and analyses on three datasets. Both trajectory forecasting and relation inference are evaluated. The latent graph manipulation experiments are interesting which show the latent codes encode leader-follower behaviors.
- The paper outperforms the baselines significantly.

**Weakness:**

- The main idea, multiplex attention, of this paper is quite **similar to factorized NRI [1]**, where multiple latent graphs are used in a variational inference framework. The technical novelty of this paper is limited. In some sense, the multiplex attention is not a big change from NRI since the latent code itself is multi-dimensional (therefore each dimension can be treated as a separate latent code) and the main difference of the proposed approach is to not share encoder weights.
- The paper didn’t compare with strong multi-agent trajectory forecasting baselines on classic trajectory forecasting benchmarks such as ETH/UCY where a large number of agents are interacting with each other. The paper could compare with SOTA methods such as [2, 3, 4].
- Some design choices are not well motivated. What is the motivation for soft-max across agents in Eq (2)? This constrains the interaction strength from $i$ to other agents to sum to 1 for interaction $k$. However, agent $i$ can have strong interaction strength with multiple different agents for interaction $k$. Therefore, I wonder why the method doesn’t just formulate inference of $z_{ij}^k$ as a binary classification (this interaction exists or not) instead of multi-class classification across other agents.

[1] Webb, Ezra, et al. "Factorised neural relational inference for multi-interaction systems." ICML 2019 Workshops.

[2] Mangalam, Karttikeya, et al. "From goals, waypoints & paths to long term human trajectory forecasting." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[3] Yuan, Ye, et al. "Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[4] Wang, Chuhua, et al. "Stepwise goal-driven networks for trajectory prediction." IEEE Robotics and Automation Letters 7.2 (2022): 2716-2723.
questions: - In Fig. 3, how are the latent codes identified for changing the leader? If I understand correctly, there should be multiple latent codes. A more compelling experiment for interpretability is probably to show different latent codes can lead to different types of interactions (besides leader-follower). This also validates the key motivation of this paper better.
- Are latent edges directional? i.e., is $z_{ij}$ different than $z_{ji}$? Eq (2) only makes sense if edges are directional, otherwise $i$ and $j$ are not symmetric when computing $z_{ij}$.
- Zero-shot generalization experiments are not convincing in my opinion. The performance of all methods has dropped but in some cases the performance gap between the current method and the baselines becomes smaller. Normally, the method should outperform the baselines by a similar percentage in this new setting. Doesn’t this indicate the method actually generalizes worse than some of the baselines?
- How are column agents selected for the relational inference in Table 2? Since the method has multiple latent graphs, do you choose the largest latent code as the latent code for each column agent?


limitations: - The paper didn’t really talk about its limitation despite indicating in the form it has done so.
- One clear limitation is that it cannot handle changing interaction types. As the multi-agent interaction evolves over time, their interaction could also evolve. For example, a follower passes a leader to become the new leader. Recent work on dynamic NRI [5, 6] has attempted to address this limitation.

[5] Graber, Colin, and Alexander Schwing. "Dynamic neural relational inference for forecasting trajectories." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.

[6] Xiao, Ruichao, Manish Kumar Singh, and Rose Yu. "Dynamic Relational Inference in Multi-Agent Trajectories." ICLR 2021.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed a prediction model that uses a multiplex latent graph to represent multiple types of interactions and attention to account for different relations called Interaction Modeling with Multiplex Attention (IMMA). They also introduce a training strategy called Progressive Layer Training (PLT). The experimental results showed that their approach outperformed baseline models in trajectory forecasting and relation inference, spanning three multi-agent scenarios: social navigation, cooperative task achievement, and team sports. They also demonstrated better zero-shot generalization results in the social navigation environment.


strengths_and_weaknesses: Stremgth:
* They proposed a prediction model that combines multilayer graph neural networks and PLT for interpreting the multiplexed latent structure. 
* The experiments showed that their approach outperformed baseline models in trajectory forecasting and relation inference, spanning three multi-agent scenarios. They also demonstrated superior sample efficiency and better zero-shot generalization results.

Weakness:
* The novelty of this method seems to be combining multilayer graph neural networks and PLT, but the authors mentioned that each component has novelty, which may be exaggerated. I asked this question in the following “Questions”. 
* The result was clear but there was no code available (I saw “Checklist”) and there were some questions in the experiment description. 


questions: 1. I don’t know all of the trajectory prediction studies, but multilayer graph neural networks in trajectory prediction may not be a new idea and the following studies were quickly found [1] [2]. 
2. The idea of PLT may seem to be a special case of the original paper (Li et al. 2020 ICLR, not arxiv). The authors can clarify the difference from the original idea.
3. The strength of the method may be the decomposition of multilayer relations like Fig.1. However, the results including Fig 3 may show only one layer. How did the authors merge it? And I want to know whether each layer can extract multiple relations like Fig. 1 or not. 

[1] Inhwan Bae and Hae-Gon Jeon, Disentangled Multi-Relational Graph Convolutional Network for Pedestrian Trajectory Prediction, AAAI, 2021
[2] Rui Zhou et al. Grouptron: Dynamic Multi-Scale Graph Convolutional Networks for Group-Aware Dense Crowd Trajectory Forecasting, ICRA 2022

limitations: The authors addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new method for modeling and predicting multi-agent interaction. Observing that such interactions could be affected by multiple factors such as target destinations and collision avoidance during crowd navigation tasks, the key idea was to model them using multiplex graphs. Specifically, rather than classifying types of edges of graphs as done in existing work, the proposed approach instead perform multi-class predictions for multiple edge types and construct multiplex attention matrices. To make latent graph layers diverse, the proposed work introduces a progressive learning scheme (progressive layer training) that gradually increases the number of graph layers to learn. Experimental results on multiple datasets show the effectiveness of the proposed approach over existing methods.

strengths_and_weaknesses: ## Strengths
- **Originality**: modeling multi-agent interaction using multiplex graphs seems novel, at least for trajectory forecasting tasks. It is well motivated by the observations that human interactions can be affected by multiple factors.
- **Quality**: I agree that the proposed approach is well designed and technically sound. The questions that the paper addresses are clearly presented (Q1-Q4 in Section 4) and answered in the experimental results. Experiment details as well as potential societal impacts are clearly described in the appendix.
- **Clarity**: Overall the paper is clearly written and easy to follow.
- **Significance**: I believe that the proposed work is relatively significant mainly because it outperformed existing methods on multiple datasets. The zero-shot generalization experiment was interesting and showing the usefulness of the proposed approach.

## Weaknesses
- The motivating example for multiple factors affecting interactions, i.e., target friend and collision avoidance in Figure 1, may be a bit confusing. If I understand it correctly, multiplex graphs are built rather in a bottom-up fashion in the proposed method, where we cannot ensure that each graph layer has such semantic meaning. In fact, I'm still a bit confused what the multiplex graph actually learned. Is this just a leading agent as shown in Figure 5 or something beyond?
- While it was great that the proposed method performs very well, I would also like to see typical failure cases. Are there some cases where interaction modeling or prediction are likely to fail?
questions: Some suggestions:

- Figure 1 could be improved to precisely describe what the proposed method actually learns. Does it really learn to model collision avoidance, or does it just learn who follows whom?
- Honestly, I can see few differences between RFM and Ours in Figure 5 or cannot tell which of results from RFM and those from Ours are more natural.
- What are the limitations and failure cases for the proposed method?
limitations: As also described above, the paper currently has few discussions on the limitations and typical failure cases for the proposed approach.
review_title: null

Meta-review: metareview: The reviewers agreed this paper was presented well and a valuable contribution. We urge the authors to take the reviewers' comments into account in the final version. 

Also, please increase the size of the tables -- the font size is quite small (maybe too small). 
confidence: Certain
Decision: Accept