Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a directed graphical model called deep latent space model for modeling the graph adjacency matrix generation process, where the latent variables are organized in a hierarchical way with three meaningful components: “latent position”, “community membership” and “node random factors” for each node. The learning leverages the variational framework originated from VAE, with GCN as the core component for posterior parameterization. Experiments on link prediction and community detection show that the proposed method is able to achieve better empirical results. 

main_review: 
The paper proposed an interesting graphical model for modeling the latent factors in directed graph adjacency matrix generation. The proposed solution that leverages VAE for optimizing the ELBO is also valid. Empirical results on the selected benchmarks seem to be significant as well. 

However I have several concerns regarding the current draft.

1. Regarding the correctness of the paper, I’m a bit worried about the claims in Sec 5, saying that “we theoretically prove that the latent variables learned by our proposed method, …, are interpretable, …”. Given the fact that the paper uses a variational method to learn a variational posterior, where the posterior takes a mean-field approximation, there’s no guarantee that the model will learn the true posterior. So no matter how good the ideal posterior is, the framework proposed by the paper is not guaranteed to learn that.

2. I’m not sure if the paper is well motivated. The model design seems to be arbitrary, with three components (latent position, community membership, node random factors) combined in an arbitrary way without justification. For example, why one has to adapt the IBP as a prior for Bayesian nonparametric purposes?  Given that you are doing truncation anyway,  I’m wondering whether you would gain anything using the Bayesian nonparametric for community assignment? 

3. The benefit of the deep hierarchical graphical model is not properly justified either. From the appendix it seems that the authors are using a 3-layer decoder by default. To justify the benefit of hierarchical graphical models, one should run the experiment with a 1-layer graphical model, parameterized with a 3-layer neural network. 

4. The experiments are conducted on small scaled graphs. How practical is it for the proposed method to be applied on realistic social networks? Having results on the latest benchmarks like OGB would be more convincing. 

summary_of_the_review: The paper lacks a motivation and justification for the proposed model design. The theoretical claim might not be correct. The experimental results are not convincing enough and larger scaled experiments on OGB are needed.

---
# After rebuttal: 
I would thank the authors' effort in addressing my concerns. However the concerns like the technical contribution and experimental improvements are not fully resolved, but I do appreciate the authors' effort in making the code accessible which improves the reproducibility in the community. I suggest the authors double check the code provided and make sure it is consistent as well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies a deep latent space model for directed graph representation. The model is basically a hierarchical variational auto-encoder architecture and some latent variables like social activity and popularity factors of node account for the edge directions. Some latent variables are interpreted as positions in the latent space or community membership. The model, therefore, is claimed to be interpretable and superior in experiments.

----Update after rebuttal

The authors basically answered my questions and I'd like to increase the recommendation score. I still think this submission has incremental technical contributions so my new recommendation score is 6 (marginally above the acceptance threshold).
main_review: Overall this paper is well-written and easy to follow, but the technical contribution is incremental, given that VAEs have been used to represent undirected graph-structured data [1,2,3]. It seems a natural extension of hierarchical VAEs to directed graphs, with some sort of variables for edge direction. The strong points include (1) a VAE with some well-tailed designs to directed graphs, (2) efforts to interpret latent variables, and (3) seemingly promising experimental performance.
 
The technical weakness mainly comes from the design of the model architecture and experiments. (1) I’m not fully convinced by the model design that variables in the decoder (e.g., z^(i) ) are lay-by-layer influenced by those in the encoder (e.g., H_z^(i)). I’d like to see an intuitive explanation and experiments that show z^(i) conditioned on H_z^(i) performs better than alternative model architectures. (2) As there are so many latent variables to be inferred (n*L*4), seen in Equation 11, I doubt the effectiveness of VI. Is there any model collapse when you optimize Equation 11? You can refer to [4,5,6] for mode collapse in VI. (3) In terms of experiments, I would ask the authors to try some directed graph networks as baselines, not limited to variational or Bayesian. As the authors said, most baselines are “designed for undirected graphs and are not appropriate for learning on asymmetric adjacency matrices”, and the authors modified these baselines for directed graphs. More commonly-used directed graph models as baselines will be convincing, such as [7,8,9]. (4) It’s good to present a model sensitivity analysis of hyper-parameters.
 
Minor weakness: (1) the \theta in the second last line on page 3 is claimed to denote the collection of model parameters, but it has been used to denote the collection of latent representations in the first paragraph in section 3. Fix this misuse. (2)In the first sentence of the second paragraph on page 4, the authors said four types of latent variables, but in the first sentence of the second paragraph on page 2, they said three types. (3) In the first sentence on page 7, “A extreme” should be “An extreme”. (4) Misplaced subgraph titles in Figure 3 in Appendix.

[1] Thomas N Kipf and Max Welling. Variational graph auto-encoders. In NIPS Workshop on Bayesian Deep Learning, 2016.

[2] Aditya Grover, Aaron Zweig, and Stefano Ermon. Graphite: Iterative generative modeling of graphs. In International Conference on Machine Learning, pp. 2434–2444, 2019.

[3] Guillaume Salha, Stratis Limnios, Romain Hennequin, Viet-Anh Tran, and Michalis Vazirgiannis. Gravity-inspired graph autoencoders for directed link prediction. In ACM International Conference on Information and Knowledge Management, pp. 589–598, 2019.

[4] Lucas, James, et al. "Understanding posterior collapse in generative latent variable models." (2019).

[5] Deasy, Jacob, Nikola Simidjievski, and Pietro Liò. "Constraining variational inference with geometric jensen-shannon divergence." arXiv preprint arXiv:2006.10599 (2020).

[6] He, Junxian, et al. "Lagging Inference Networks and Posterior Collapse in Variational Autoencoders." International Conference on Learning Representations. 2018.

[7] ​​Zhang, Xitong, et al. "MagNet: A Neural Network for Directed Graphs." arXiv preprint arXiv:2102.11391 (2021).

[8] Ou, Mingdong, et al. "Asymmetric transitivity preserving graph embedding." Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining. 2016.

[9] Funke, Thorben, et al. "Low-dimensional statistical manifold embedding of directed graphs." arXiv preprint arXiv:1905.10227 (2019).

summary_of_the_review: Given the above content, I think this paper is an okay submission, and I will give boardline rejection. I’m willing to increase the score if the authors (1) intuitively and empirically motivate the layer-by-layer variable dependency in the decoder, (2) analyze the effectiveness of VI and how your method prevents mode collapse, (3) conduct comparative experiments with suggested baselines, and (4) analyze how your model is sensitive to the community number.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a VAE-based graph representation learning model for directed graphs. The model is composed of a GCN encoder and a hierarchical latent space model decoder. The decoder explicitly models three kinds of node representations: the node latent position vectors $z$, the community membership vectors $s$ and node random factors $\gamma,\delta$. Specifically, the community membership vectors $s$ are used to gate control which of $z$ and $\gamma,\delta$ pass to the next layer. And finally the adjacency matrix is reconstructed by the distance between $z$'s filtered by node influences $\gamma,\delta$. Experiments show better link prediction and community detection performance than plain VGAE and other VGAE methods.
main_review: Strengths:
1. Explicit modeling of community structure, latent positions, and random node factors. 
2. The asymmetric modeling of link existence thus suitable for directed graphs.
3. The learned node embeddings have some interpretability.

Weaknesses:
1. Experiments did not compare with state-of-the-art non-VAE-based GNN methods, nor discussed them. For example, the state-of-the-art link prediction method, SEAL [1], should be compared. It can work on directed graphs too. Merely from Table 5 and results reported from [2], SEAL achieves better performance on Cora and Pubmed than the proposed method.
2. Some larger datasets should be used to evaluate the scalability as well as the effectiveness for large networks. Currently, the largest dataset used contains 15,763 nodes and 171,206 edges. However, people are switching gradually to larger datasets with millions of nodes/edges such as Open Graph Benchmark [3].
3. No ablation study on whether the proposed $s,\gamma,\delta$ are useful.

[1] Zhang, Muhan, and Yixin Chen. "Link prediction based on graph neural networks." Advances in Neural Information Processing Systems 31 (2018): 5165-5175.\
[2] Zhu, Zhaocheng, et al. "Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction." arXiv preprint arXiv:2106.06935 (2021).\
[3] Hu, Weihua, et al. "Open graph benchmark: Datasets for machine learning on graphs." arXiv preprint arXiv:2005.00687 (2020).
summary_of_the_review: Overall, although the paper proposes some interesting ideas by explicitly modeling community structure, latent positions, and random node factors, the paper lacks a discussion and comparison with state-of-the-art non-VAE-based baselines and uses relatively small datasets in the experiments, thus is less convincing in performance. Considering the abstract states "The experimental results on real-world graphs demonstrate that our proposed model achieves the state-of-the-art performances on link prediction and community detection tasks", I cannot recommend an accept given that stronger baselines clearly exist.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a new directed graph representation model. The proposed model is developed from a Bayesian viewpoint and is implemented in the framework of the variational autoencoder. Additionally, the authors justified the interpretability of the model. They indicate that the proposed model quantitatively captures the influence of each node on its neighbors. Experimental results show that the proposed model performs well in commonly-used datasets. 
main_review: Pros:
The paper is well-written and easy to read. The details of the model and related derivations are provided.

From a technical viewpoint, the proposed model can be viewed as a new member of the probabilistic directed graph models. Its learning algorithm is based on (amortized) variational inference and the implementation is compatible with neural networks. Overall, the methodology is solid and interpretable. 

I like section 5, which provides some useful insights into the model.


Cons:
It seems that the proposed method is not as good as GGVAE on some datasets. It would be nice if the authors can explain this phenomenon to some degree.

In Figure 2, the authors should show the t-SNE figures of GGVAE and VGAE.

Besides node embeddings z’s, the authors introduced another kind of latent code, i.e., s’s to indicate the community structure of a graph. It is worth doing an ablation study to demonstrate the necessity of this latent code.

The influence of the number of layers on the performance of the model should be discussed.

summary_of_the_review: Overall, the methodology of this work is solid, which is a reasonable combination of several solid probabilistic models. The learning method is easy to implement. Although the novelty of the whole method is not very strong, the experimental results show its feasibility to some degree.
review_title: null

Meta-review: This to me looks like quality work not yet adequately developed, and thus is borderline work.  The authors seem to have achieved a good result:  equalling SotA SEAL (although, one reviewer did preliminary experiments and could not match this) with a sophisticated algorithm using a variety of Bayesian tricks, a more scalable algorithm, and one potentially adapted to further tasks.  However, not all of these impressive feats are adequately demonstrated in this paper, though many had parts included in the rewrite.  So I'd say the paper needs a rewrite and more focussed experimental work to broaden the presentation of empirial performance, for instance to node classification.
I certaintly appreciated the use of IBP and Dirichlet models within the system, so would love to see the work further developed.
The reviewers agreed in several aspects:  (1) more experimental work, for instance on better and larger benchmark data, (2) better presentation and discussion of the theory, (3) better discussion of the motivation for the model (as per reviewer D8S8), and oftentimes linked to the ablation study to support this, which you have done some of (4) additional connections to recent related work in graph representation learning on link prediction works
The authors have done a good job or addressing many of the reviewers concerns, ultimately lifting the paper from Reject to Borderline Negative, but I think more work is needed.
Decision: Reject