Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces RAIN, a method to learn relational graphs between different nodes ("agents") of a system from time-series data describing the trajectories followed by said nodes. 

RAIN is composed of a) an encoder computing an embedding of the trajectories seen so far, b) a graph extractor that infers the interaction strength between nodes. The authors also combine RAIN with a "pairwise attention" (PA) mechanism, which greatly improves the reported performance of RAIN.

The authors describe their method and then evaluate it on spring-ball models, Kuramoto oscillators and motion capture data of a person walking.
strength_and_weaknesses: **Strengths**

The main strength of the paper appears to be its empirical evaluation. Evaluating on three different very different kind of systems (plus a multi-level variant of the ball-spring system) provides a significant insight into the advantages of RAIN over NRI, which can only assign discrete interaction strengths to graph edges. 

The results section is further strengthened by a varied analysis combining trajectory predictions, connectivity matrices and correlation coefficients. I found that this multi-modal approach helped my appreciation of the results despite being unfamiliar with previous works.

Finally, the problem of learning continuous interaction graphs from data seems challenging and highly relevant to both future research and applications.

**Weaknesses**

The first issue with the paper is the lack of a Background section clearly describing the problem and introducing the key variables and general approaches. Without a background section, the paper is hard to follow since it is unclear what assumptions can be made about the system or what constitutes success. 

Secondly, I found the Section 2 lacked clarity, partially due to the issue above but also due to writing and inconsistent notation. 
For instance, it is not clearly stated before the method whether one trajectory describes the evolution of the entire system, or whether it is one trajectory per node. Based on the section 3, I believe it to be the latter, but sentences like the following at first let me believe it was the former, when trying to understand the method:
"The encoder receives $T_{enc}$ steps of trajectories $x_1, x_2, ... , x_{T_{enc}}$ , each consisting of $R$ state variables of $N$ agents, and produces corresponding $T_{enc}$ hidden states, $h_1, h_2, ... , h_{T_{enc}}$."

Relatedly, the authors state "we calculate the attention between _same-time hidden states_ to assign weights to their contribution". The term "same-time hidden states" is never clearly defined, despite being key to PA. Additionally, Figure 2 clearly features a sum over different time-steps to compute the Pairwise Attention, which seems contradictory.

Another point of confusion is that variables used in figures 1 and 2 ($h'_{ij}, A_\theta, w_t$) are absent from the main body of the paper, including equations. Given the two previous points, I found Figures 1 and 2 to be of little help to my understanding, overall.

I also found it confusing that nodes (e.g. balls in the ball-spring system) be called "agents", given that they take no action. That said, feel free to ignore this comment if this is standard terminology in the field.

Finally, on the experimental side, results reported lacked confidence intervals or other measures of uncertainty. I also found it slightly odd that the only baseline reported dates from 2018. 


clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper generally lacks clarity, as mentioned in the section above. In addition to the aforementioned points, authors could consider including an algorithm block to summarize their method.

**Quality**

The results appear to be strong when compared to NRI, but it is unclear to me how strong and relevant of a comparison that is given NRI dates from 2018 and was designed for graphs with discrete interactions. I am also curious for the authors to describe how NRI would perform given access to a higher number (N > 4) of edge types. Is it possible, for instance, that NRI(16) outperforms RAIN+PA, but at a huge computational cost?

**Novelty**

Being unfamiliar with previous works in this field, it is difficult for me to assess novelty.

**Reproducibility**

If authors follow through with their claim of releasing data and models on Github, reproducibility should not be an issue.



summary_of_the_review: I believe the paper addresses a challenging and relevant problem, and proposes a novel architecture to do so. It also presents a range of strong empirical results, albeit one that raises a few questions about the reported baseline. Also, the paper suffers from a global lack of clarity and a missing background section, which is its main weakness in my opinion.

Given that, I do not believe the paper is ready for publication, but I am willing to reassess if the issues mentioned are resolved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a RAIN to infer interactions among agents. It learns the attentive weight and unknown system dynamics, which is also helpful to predict future trajectories. The author(s) verified that RAIN can infer the system dynamic parameters and interaction graphs, and can outperform baseline discrete models.  

strength_and_weaknesses: Strength:

The triplet of (trajectory encoder + interaction graph + trajectory decoder) seem to work well, with the important added tricks such as modified RNN's pairwise attention (Eq 4-5), prior-less graph with transformer-like attention (Eq 7-8). Those methodologies seems sufficiently different from previous works and are novel.

Table 2 indicate a significant performance gain by using RAIN. 

Weakness:

The paper draws similarity between the neural network and the physical dynamic system. However, the reviewer can't find any physical constraints are directly applied in the neural networks, nor at the neural network losses. The reviewer would like to see the actual dynamic system encoded in the NN architectures to generate more guaranteed realistic trajectories.

The experiment results include a simulated data (Table 2) and motion capture data of human walking (Table 3) seems toy-ish (although some previous works also uses those data). The reviewer would like to see how this method perform in a more complex scenario such as in https://waymo.com/open/
clarity,_quality,_novelty_and_reproducibility: The writing is clear, novel, and seems to offer enough implementation detail to reproduce. But the sharing of the code would be even better.
summary_of_the_review: The main contribution of the paper is the proposal of the RAIN + PA which achieves a surprisingly good result in Table 2. Probably the contribution of adding PA to the RAIN is more important (empirically) than the main model (RAIN).

The introduction of PA is not unheard of, but is new in the interaction/prediction setting. This is the main reason it is marginally above the acceptance threshold.

The reason it is not getting higher ratings is because the performance of the main model RAIN without PA is somewhat disappointing. Also, the reviewer would like to see the method tested on some more non-trivial datasets to verify its validity.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors propose a relational attentive inference network (RAIN), in which the interactions between agents are continuous. A pairwise attention (PA) is adopted to refine the representations and a transform structure is also employed to extract the interaction weights. The proposed method achieves good performance on interaction strength inference and promising results on motion prediction.
strength_and_weaknesses: **Strengths**:  
1. Well-organized and clearly written.
2. Strong motivation with promising results.  


**Weaknesses**:  
1. In Section 3.1, when evaluating on the simulated physical systems, the procedure of obtaining the ground-truth of connectivity/interaction matrix is not clear, since the method in the unsupervised manner. Why use a single inter layer to get the "ground truth" matrix? More details and explanations should be included.


2. In Section 3.2, when it comes to the motion prediction problem, especifically using CMU motion dataset, existing methods usually predict the difference from the previous time step, but authors predict the raw value here. What's the reasons/insights here?  Also, in Figure 6(c), are there any ground-truth interaction weights? or how to tell which one is better when comparing the interaction weight matrix? 


3. In Section 2.1, for the model complexity, for the pair-wise attention weights calculation, at each time step, it would be $O(N^{2})$, assuming there are $N$ objects. Also, when employing the transformer structure, the calculation cost would be very large. More explanations about reducing  time complexity should be included, some experiments would be better to illustrate.


4. Experiments are not sufficient, only NRI is compared (2018), more existing SOTA baselines should be included.

5. Missing related works, it is not a mandatory task, but I strong encourage authors to include a related work section to introduce some recent works on, i.e., interaction inference/learning[1,2], trajectory/motion prediction [3, 4], etc, and to highlight the differences between the proposed method and existing works.  


6. Minor mistakes:  
(1) In Figure 2, is it "+" or "$\bigoplus$"?  
(2) In Figure 5 (D), x/y axis label is missing.  
(3) In References, the names of conferences or journals, i.e., format and first letter, are not consistent.
I highly encourage authors to revise the submission to avoid such small mistakes, as well as some typos.

[1] GRIN: Generative Relation and Intention Network for Multi-agent Trajectory Prediction. NeurIPS2021  
[2] Evolvegraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning. NeurIPS2020  
[3] Adaptive Trajectory Prediction via Transferable GNN. CVPR2022  
[4] GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction With Relational Reasoning. CVPR2022  
clarity,_quality,_novelty_and_reproducibility: Details are in [Weaknesses].

**Clarity**:  
The clarity is good but with some minor mistakes.

**Quality**:  
Clear-written and well-organized.

**Novelty**:  
Motivation is good and novelty are good as well.

**Reproducibility**:  
The training procedure is clear,  and authors also provide the code, even though I searched to have. 
summary_of_the_review: I list my concerns in [Weaknesses], I am happy to discuss and increase the rating if my concerns are addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a model called RAIN for interaction and trajectory prediction. Given the history trajectories of multiple moving agents, the RAIN model predicts their interaction weights and future trajectories.

The RAIN model works as follows. 1) It first encodes the history of each agent with an LSTM encoder. 2) It then uses pairwise attention to enrich the per-agent feature vectors. 3) It then uses a graph attention network to further infer the interaction weights. 4) Finally, it uses an LSTM decoder to predict their future trajectories.

The authors evaluated the RAIN model on two physical systems (a spring-ball system and a phase-coupled Kuramoto oscillator system) and a human walking dataset. They used the NRI method as their baseline. The evaluation result shows that the RAIN model significantly increases the accuracy of correlation prediction and reduces the trajectory MSE.
strength_and_weaknesses: --- Strengths

- This paper presents a sound method for predicting agent interaction and future trajectory. The proposed method achieves better performance than the baseline method on public benchmarks.

- The result shows that adding pair-wise attention helps.


--- Weaknesses

- There is not sufficient comparison against the state-of-the-art baselines to demonstrate the advantage of the proposed method. In the evaluation section, NRI is the only public baseline that the authors compared their method against, and it's quite an old baseline. Also, there is not much discussion about the existing works in the paper.


--- Other comments

- I think there are some typos in Eq (5). It should be V_j^{t,m} instead of V_i^{t,m}. Also, I think there should be a sum over j. Otherwise, why does the subscript j disappear in the left-hand side of the equation?

- I suggest the authors adjust the positions of the result tables and figures. For example, Table 1 and Table 2 are referenced in the same paragraph in the text, but one is on Page 6 and one is on Page 9.
clarity,_quality,_novelty_and_reproducibility: - The presentation is OK, but I think there are typos in their equations.

- There are not enough existing works discussed in this paper.
summary_of_the_review: This paper presents a sound method for predicting agent interaction and future trajectory. The proposed method achieves better performance than the baseline method on public benchmarks. So I am giving it a weak accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
The main contribution of this paper is the relational attentive inference network (RAIN) for inferring continuously weighted interaction graphs without any ground-truth interaction strengths. This is achieved using a novel pairwise attention mechanism and a graph transformer to extract heterogeneous interaction weights for each pair of agents. The RAIN model is shown to accurately infer continuous interaction strengths in simulated physical systems in an unsupervised manner. Additionally, the model is able to predict trajectories from motion capture data with an interpretable interaction graph.

Strength: 
Intuitive method, strong empirical results. New baselines that got added during the rebuttal period and other improvements of the paper. 

Weakness:
No theory or guarantees, some of the results miss confidence intervals (e.g. Table 2). This should be addressed for the CRC.

justification_for_why_not_higher_score: No theory or guarantees, some of the results miss confidence intervals. 

justification_for_why_not_lower_score: Intuitive method, strong empirical results. New baselines that got added during the rebuttal period and other improvements of the paper.
Decision: Accept: poster