Title: On the Power of Edge Independent Graph Models
Authors: Sudhanshu Chanpuriya
Cameron N Musco
Konstantinos Sotiropoulos
Charalampos Tsourakakis
Venue: NeurIPS 2021
Keywords: graph generative models
random graphs
edge independent models
node embeddings
triangle density

Abstract: Why do many modern neural-network-based graph generative models fail to reproduce typical real-world network characteristics, such as high triangle density?  In this work we study the limitations  of $edge\ independent\ random\ graph\ models$, in which  each edge is added to the graph independently with some probability. Such models include both the classic Erdos-Renyi and stochastic block models, as well as  modern generative models such as NetGAN, variational graph autoencoders, and CELL. 
 We prove that subject to a $bounded\  overlap$ condition, which ensures that the model does not simply memorize a single graph, edge independent models are inherently limited in their ability to generate graphs with high triangle and other subgraph densities. Notably, such high densities are known to appear in real-world social networks and other graphs. We complement our negative results with a simple generative model that balances overlap and accuracy, performing comparably to more complex models in reconstructing many graph statistics. 

TL;DR: We show that "edge-independent" graph generative models face an inherent tradeoff between memorization and the ability to generate realistic triangle dense graphs.

Review discussion:
**Originality**
The work provides a valuable insight to the field and addresses open questions on how well graph models can model certain structural properties. The theoretical results use the notion of overlap, which is intuitively defined and appears to be tied to the analysis of structural properties in an original manner, allowing for theoretical proofs that are easy to understand.
**Quality**
All theorems appear correct (though I did not check Theorem 6 in detail) and are potentially useful starting points for later work.
My main concern with the paper is that the theoretical results are not explicitly validated in the experiments:
1) It would be nice to see the theoretical bounds on the number of triangles reflected in Figures 1 and 2. Considering that the algorithms have a controllable overlap, a fixed volume and that the bound on the number of triangles is independent of the model, including the bound would only result in one additional curve per subplot. It would provide more information on how important the bound actually is in practice.
2) All algorithms are edge-independent models. It would be interesting to at least run one baseline that is not edge-independent.
I further have a minor remark:
3) The quality could be improved in Section 3 by building further upon prior work on models where $P_{ij} = \sigma(l_i + l_j)$. Such parameters should arguably be better defined in graph learning literature, but they have been referred to as 'popularity' [1] or 'parameters modeling degree heterogeneity' [2] in the context of latent space models. The authors may be interested in the Maximum Entropy distribution subject to the constraint that the expected degree sequence (or row/column sums in the adjacency matrix) should equal the empirical degree sequence [3]. Finding this distribution is a convex optimization problem, and the solution is a model of the form $P_{ij} = \sigma(l_i + l_j)$ if its parameters were optimized with Maximum Likelihood Estimation.
**Clarity**
The paper is a joy to read and communicates the result in a surprisingly clear manner.
I do question if it is useful to divide by $V(P)$ in the definition of the overlap. If this division were not present, then the overlap definition would be simpler, and the bound in Theorem 1 would no longer contain the $V(P)$ factor.
**Significance**
The technical contributions appear highly valuable to the literature on graph generative models, where matching structural properties is key. The results may further have a positive impact on other graph-related topics such as link prediction or community detection. Very similar graph models are used there to make predictions, and this paper suggests that those models may also fail to make predictions that correspond with the actual structure of the graph.
[1] Goldenberg, A., Zheng, A. X., Fienberg, S. E., & Airoldi, E. M. (2010). A survey of statistical network models.
[2] Ma, Z., Ma, Z., & Yuan, H. (2020). Universal Latent Space Model Fitting for Large Networks with Edge Covariates. J. Mach. Learn. Res., 21, 4-1.
[3] De Bie, T. (2011). Maximum entropy models and subjective interestingness: an application to tiles in binary databases. Data Mining and Knowledge Discovery, 23(3), 407-446.
**UPDATE**
I thank the authors for their responses. My score remains unchanged and I hope to see the paper accepted and inspire future work.
No
Thank you for the thorough and positive review. We address the helpful suggestions made below.
1. This is a good suggestion. We note that in many cases, the models studied will be far from achieving the theoretical upper bound. This is because none of the algorithms focuses just on maximizing triangle density or replicating some single statistic, but on trying to replicate many aspects of the input graph at the same time.
However, plotting the theoretical bounds would still be a useful point of comparison.
2. This is also a good suggestion, especially as we believe that non-independent models should be able to outperform the edge-independent models in replicating certain graph statistics, such as triangle densities.
3. Thank you for this suggestion. We were not aware of this prior work using $P_{ij} = \sigma(l_i + l_j)$ and agree that it should at least be cited, and that perhaps we can directly build on existing results. And thanks very much for the citation to [3]. We had actually thought about using entropy in place of overlap as a metric, and this result seems very relevant to our work.
I liked this paper. It is generally clearly written, pedagogical, and quite complete. To my knowledge, the theoretical contributions are novel (at least under this form), and, if quite simple, relatively well-put into context.
My main comment would be about a quite significant blind spot concerning edge-independent models as presented here, which may be usual in this (particular corner of the) literature, but I feel that it deserved to be clearly mentioned as limitations.
Indeed, in many generative models, *the matrix P itself is random*, and edges are drawn independently conditionally on P. Think for instance of any latent-position models with random latent variables such as those studied in the graphon literature (in which motifs density is a key quantity, by the way), but also any GAN/VAE model with random input in the generator such as Variational Graph Encoder for instance, unlike what is claimed by the authors. Depending on the distribution of P, allowing random P may significantly decrease the overlap. In return, an additional difficulty is that graphs may then need to be considered up to isomorphism.
Nevertheless, I feel that, with minimal modifications (mostly taking double expectation everywhere, first on P then conditionally on P, and defining a notion of overlap that takes into account the variance of the distribution of Ps), all the theory presented could be adapted to random P, which would include many more models and significantly increase its impact.
On the other hand, the resulting bounds might already exist in the graphon literature, which studied extensively motifs density.
In conclusion, I would have no real problem accepting the paper as is, but I feel that the theory could be significantly extended to include most "real" edge-independent models with latent variables. At minima, these limitations must be mentioned, including the fact that most GAN models have other sources of randomness that are not taken into account here.
Typo: the expectation is missing in the equation at the bottom of page 3 (middle)
**==== Edit after rebuttal ====**
I thank the authors for their response. It seems we agree on the current limitations of this work, and I am confident that they will describe them and future directions in the final version of the paper. I keep my score as is.
No
Thanks for the thoughtful review, and for the very helpful points brought up.
*On P itself being random*: This is a very good point, and we will clarify discussion of this.
We first note that many 'real' graph generative models are truly edge independent -- and do not have random P. E.g., the CELL baseline discussed in the paper or Variational graph auto-encoders of Kipf and Welling. For these methods, given an input graph which we seek to generate similar graphs to, P is fixed by a deterministic algorithm. Other methods such as NetGAN (closely related to CELL) or GraphVAE do use some stochasticity in training, and thus P is random.
However, it seems that this stochasticity does not affect things much (e.g., NetGAN performs similarly to CELL).
Thus, we agree that it would be interesting to extend our results to cover these methods.
However, in general, if P is allowed to be random, this can include in theory all models, including non-edge-independent ones, which can simply be thought of as generating a random binary P.
Thus, we would have to place some restrictions on the randomness used to generate P if we hope to prove theoretical limitations on the ability to generate triangle (or other subgraph) dense graphs with low overlap.
*On isomorphism*: This is one weakness of the overlap metric. It can be 'gamed' by a non-independent model by simply randomly permuting the node ids before returning the adjacency matrix. We agree that any results which push beyond pure edge-independent models where P is fixed would have to grapple with this.
The authors do thorough theoretical analysis regarding the main points of the paper. The empirical results are also exhaustive given the number of datasets the authors decided to use. My major concern with this paper is its applicability and potential impact.
Specifically the authors themselves claim that maybe non-independent models should be preferable to edge-independent models, which reduces the significance of this current work.
The work would be significantly strengthened if the authors could extend their analysis to non-independent models, a limitation of the work the authors themselves acknowledge.
The paper would be strengthened in case the authors can prove that Algorithm 1 always converges so that it can be more practically applicable. In this work, the authors claim that the above algorithm converges for different test cases which is not a guarantee in general.
==== After reading authors' response and reviews by other reviewers ====
I would like to thank the authors for their response to all reviewers and would like to keep my score as it is. The authors are in agreement with the limitations of the paper I pointed out earlier. I wanted to wish the authors all the very best.
No
"Specifically the authors themselves claim that maybe non-independent models should be preferable to edge-independent models, which reduces the significance of this current work." -- Currently edge independent models are the most common graph models studied in the literature. Our results -- in particular our theoretical upper bounds -- suggest that perhaps non-independent models are more powerful. We believe that this is exactly the relevance of our work. We do not believe our analysis should extend to non-independent models since we believe these models can in fact generate graphs with high triangle and other subgraph densities, while having low overlap.
We agree that it would be very nice to show that Algorithm 1 converges. It may have applicability beyond our specific use case.
Dear authors,
Thank you for your response to my queries/comments as well as the other reviewers. I really appreciate your detailed responses and taking the time to add them.
I do not have any further questions at present. I will review the other reviewers' comments as well as all your responses to them to finalize my evaluation/score. I wish you all the very best.
Warm regards,
iT9Y
This paper studies the limitation of edge independence graph models that it could not generate graphs with many triangles and other subgraph densities, which are common subgraphs in real networks. Motivated by this finding, they propose a method of generating graphs with matching graph statistics. Section 2 investigates the upper bound on subgraphs generated by the edge independent models, which is novel and valid.
Meanwhile, I have several concerns about the proposed baseline edge independence models. I summarized the questions that need authors' address below.
1. This paper mentioned some examples of edge independent models such as ER and SBM. I wonder how your results would look like for another family, latent space models (Hoff 2002), where $P_{i,j} = similarity(Z_i, Z_j)$ and $Z_i$, $Z_j$ are node embeddings of node $i$ and $j$. I believe this family of models would be more flexible than ER or SBM here as node individual embeddings provide more flexibility.
Also this model is known to show transitivity. I understand the upper bound in Theorem 1-7 would stay the same format for all edge independent models, but I wonder would this setting would make the larger bound number larger and therefore increase the power of edge independent model?
2. If certain edge independent models could not match the graph statistics, it is a signal that this model is not a good fit for the graph. Under this case, why we choose stay with the edge independent graph models, and try to modify the graph generating mechanism that matches graph statistics, rather than using the models that can handle triangles or subgraphs or more flexible edge independent models (as mentioned in question 1).
3. Related to question 2, I am wondering matching summary statistics would be a necessary or sufficient criterion when evaluating graph generation. It is possible that two graph have similar # of triangles, degrees, etc but show very different structure.
No
Thank you for the thorough review. We appreciate the questions raised, which we address below.
1. Our upper bounds (Theorems 1,4,6) hold for all edge independent models, including latent space models. Theorems 3,5,7 show that a simple ER graph gives the highest possible bound, for any edge independent model.
So a latent space model can only do worse in terms of the metric we focus on. This is not to say that latent space models are not interesting or cannot be more powerful than ER graphs in other aspects (they certainly are in many cases, e.g., community formation), but they are not more powerful in their ability to generate graphs with high triangle density, and other subgraph densities with lower overlap.
In our experimental section, we compare to two important latent space models -- CELL and TSVD.
2. Related to (1) above -- it is not just that certain edge independent models cannot match graph statistics. Our results prove that in some settings *no edge independent models* can match the graph statistics, if they do not exhibit high overlap. We introduce our baseline edge independent models CCOP and HDOP to help explore the limits of how well edge independent models can perform.
However, we agree that our theoretical results suggest that, in many settings, non-edge independent models may be a better choice.
3. We think that this is a very good point, and it also lies close to extremal combinatorics and flag algebras;
for instance, it is well known that two triangle-free graphs can be very different in structure (e.g., bipartite and Andrásfai graphs).
So far, the literature on graph generative models mostly focuses on matching graph statistics, and thus our work addresses these statistics.
However, we agree that this is limiting.
While beyond the scope of our work, it would be very interesting to develop other metrics to evaluate graph generative models, and then to explore the limitations of edge-independence models in achieving these metrics.

Meta-review:
role: PC
data: comment: The reviewers are generally leaning to accept the paper. They unanimously appreciate the theoretical contribution of the paper regarding edge-indendent graph models and consider the paper well written. Most concerns have been very well addressed by the authors. Possible extensions of the paper to edge-dependent models are interesting but beyond the scope of the paper at hand. Reviewers disagree on the potential impact of the paper and on the relevance of the theoretical results in practical applications. It was mentioned that the paper is highly valuable for research on graph generative models and may have further impact on other research regarding learning on graphs.

Decision: Accept (Poster)