Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a variational model to infer the underlying directed graph of a dynamical system from observed features of the agents over a time period. Regularization terms were also introduced to encourage learing of directed interactions. Evaluations on a benchmark of datasets show good results.
strengths_and_weaknesses: Strengths:

1. The idea of iterative learning of the adjacency matrix seems new.


Weaknesses:

1. There are many regularization terms in this approach, and all of them seems to be crucial for the overall performance of direct interation discovery, which makes the information bottleneck formulation a little bit vague.

2. It is not clear how the inner loop is conducted in section 4.4.


questions: 1. How sensitive is this approach to the initialization of the adjacency matrix A? Why the fully connected one is a good choice?
2. What kind of GNN was used in your experiments? Does the choice of GNN matter?
3. Is your hybrid loss strategy applicable to other baseline approach?
limitations: Yes they do.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose the Iterative Structural Inference of Directed Graphs (iSIDG), which is able to infer the directed interactions between agents in dynamic systems with information bottleneck theory and extra regularization. The proposed framework is examined on several simulated and synthetic networks to identify the interactions. 
strengths_and_weaknesses: Strengths: 
- This paper focuses on inferring the underlying structure of agent interactions in time-varying systems, which is a complex problem due to the dynamic feature.
- Apart from using information bottleneck of VAE to infer the structure, the proposed method also consider the sparsity and connectiveness of the asymmetric adjacency matrix property and design extra regularization term to address the related features. Ablation study also shows that the sparsity regularization term can especially help to infer direct connections.

Weakness:
* Size of the dataset (number of nodes) is very small here. Is it because of the limitation of the datasets or the proposed framework is very computationally expensive? 
* It would be beneficial if the authors add a ground truth subfigure in Fig. 2.
* The proposed method is only applied on simulated or synthetic datasets. I strongly recommend examining the framework on at least two real-world datasets to show the priority of the proposed method to reconstruct the graph structure. Simulated or synthetic data are usually more "predictable" since the data generation process is known and with less uncertainty, while high complexity and unknowns is more likely to be involved in real-world datasets.
* I suggest the authors to look into the following related paper on reconstructing network
* * https://openreview.net/forum?id=rJgTciR9tm
* * https://www.nature.com/articles/s41467-019-09774-x


questions: - In the problem formulation, the network structure (adjacency matrix) is considered as consistent across time, while only the node feature is changing. In many real-world datasets, the pairwise connection can appear and disappear over the time and new nodes can also join at some time point. Can the author further explain how their framework can be applied to such scenarios?
- What is the model complexity here? I would like to see the duration of model training compared with baselines. It also helps to see if the proposed method can be scaled up to large complex systems.

limitations: See Weakness
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies the structural inference problem of reconstructing the asymmetric adjacency matrix of the graph in an unsupervised fashion. Specifically, the paper introduces an iterative process based on Variational Information Bottleneck that uses GNN-based encoder to infer directed interaction represented in the latent space, which is then used to update the asymmetric adjacency iteratively starting from a fully connected graph. The updated adjacency is decoded to predict the future states of the nodes. The paper also introduces additional regularization terms in the objective function for realistic structure and eliminate the influence of indirect interactions.
strengths_and_weaknesses: **Strength**
- The paper is well-written and the presentation is clear. 
The preliminaries section provides a detailed explanation of the problem which the paper is trying to solve, and Figure 1 gives a clear overview of the model architecture.

- The proposed iterative scheme for structural inference seems to be novel and simple, yet effective. Although I have some concerns about the efficiency (described in the Weakness part), the proposed method shows clear performance improvement.

- The paper clearly addresses the problem of "the influence of the indirect connections" of the structural inference methods, and provides the solutions for eliminating such challenges.

- Extensive analyses are provided in the Appendix which gives further insights. Especially, the analysis of the scalability experiments shows meaningful results, even with the limitations. 

**Weakness**

- The proposed method seems to be quite slow due to the iterative scheme, thus comparing time/memory efficiency may address my concern. Especially, as iSIDG shows marginal improvements on some datasets compared with fNRI (e.g. TF, Springs, Kuramoto), I believe the efficiency comparison is critical. Furthermore, the efficiency problem may be increased for larger datasets. I would like to see the efficiency comparison on Springs 100 and ESC datasets (as used in section E of Appendix).

- The influence of the hyperparameters of Equation 21 is not sufficiently explained. Although the ablation study shows the results of each term in the loss, without the analysis, the significance of the proposed hybrid loss is not fully clarified.
questions: - In Table 3, we can observe that the hyperparameters vary with different scales according to the dataset. Is the performance sensitive with respect to the hyperparameter choice? Why is the scale of the hyperparameter different? 

- I would like to see statistical tests on the results, as some datasets (e.g. TF, Springs, Netsim 1 and 2) show large variance.
limitations: - The authors present scalability limitations in section E of the Appendix, which is a common limitation among the baselines.
review_title: null

Meta-review: metareview: This is a bordeline paper.

All reviewers liked the paper but were a bit concerned with the number of regularization terms (the loss combines VAE and information bottleneck approaches) and thus the number of hyperparameters. Secondly, the reviewers were also initially concerned with the size of the benchmarks but the discussion phase convinced them that these are standard in the field. Thirdly, some reviewers pointed to more related work. This, however, was not a crucial point.  

Since all reviewers finds the paper interesting and well-written, acceptance is recommended. 
confidence: Less certain
Decision: Accept