GG-GAN: A Geometric Graph Generative Adversarial Network

1 INTRODUCTION . Learning distributions from empirical observations is a fundamental problem in machine learning and statistics ( Goodfellow et al. , 2014 ; 2016 ; Salakhutdinov , 2015 ; Foster , 2019 ) . A challenging variant entails modeling distributions over graphs—discrete objects with possibly complex relational structure ( Simonovsky & Komodakis , 2018 ; You et al. , 2018 ; De Cao & Kipf , 2018 ; Liao et al. , 2019 ; Niu et al. , 2020 ; Yang et al. , 2019 ) . When successfully trained , deep graph generative models carry the potential to transform a wide range of application domains , for instance by finding novel chemical compounds for drug discovery ( De Cao & Kipf , 2018 ) , designing proteins that do not exist in nature ( Huang et al. , 2016 ) , and automatically synthesizing circuits ( Guo et al. , 2019 ) . By and large , there are four properties that a graph generator g should possess : ( 1 ) Isomorphism consistency : g should assign isomorphic graphs the same probability—a property also referred to as permutation equivariance ( Niu et al. , 2020 ; Yang et al. , 2019 ) . ( 2 ) Expressive power : g should be able to model local and global dependencies between nodes and graph edges , e.g. , going beyond simple degree statistics and learning structural features and motifs . ( 3 ) Scalability : g should be able to synthesize graphs with tens of thousands of vertices . ( 4 ) Novelty : g should produce non-isomorphic graphs that are similar to ( but not necessarily in ) the training set . Property ( 1 ) is important since there exist exponentially many ways to represent the same graph as a vector , inconsistent methods effectively waste a large portion of their capacity in describing different ways to construct the same object . Properties ( 2 ) and ( 3 ) are critical in large-scale contemporary applications that require going beyond simple degree statistics as well as learning structural features and motifs towards simultaneously modeling local and global dependencies between nodes and graph edges ( You et al. , 2018 ) . Property ( 4 ) is natural since common failure modes for graph generators include memorizing the training set and repeatedly generating the same graphs . 1.1 GEOMETRIC GRAPH GENERATION . Aiming to satisfy these properties , we propose a geometric generator that represents graphs spatially by embedding each node in a high-dimensional metric space and then by connecting two nodes if their positions are sufficiently similar . There is precedence to our approach , as spatial representations of graphs have been heavily used to construct simple models of random graphs , such as random geometric graphs , unit-disc graphs , unit-distance graphs , and graphons ( Huson & Sen , 1995 ; Penrose et al. , 2003 ; Bollobás et al. , 2007 ; Lovász , 2012 ; Alon & Kupavskii , 2014 ; Glasscock , 2015 ) . Surprisingly , these classical geometric approaches have so far found limited adoption in the context of deep generative graph models . In fact , it is easy to verify empirically that a direct , naive application of these methods yields poor performance , requiring additional conditioning and other stabilizing procedures to train effectively ( Yang et al. , 2019 ; Serviansky et al. , 2020 ) . Our work precisely bridges this gap , showing how deep geometric graph generation approaches can perform well . Our contributions are two-fold : I . We shed light into the fundamental limits and challenges of geometric graph generators ( Section 2 ) . We derive sufficient conditions for representing graphs spatially and demonstrate that for sparse graphs the embedding dimension can depend only logarithmically on n. We then identify challenges that arise when building powerful and isomorphism-consistent generators . Interestingly , we find that straightforward generators must solve a non-trivial collision avoidance problem for every graph . We present evidence that such generators can not be easily trained even in simplified supervised settings . II . We avoid collisions at generation time , while retaining scalability and consistency ( Section 3 ) . We propose geometric graph generative adversarial networks ( GG-GAN ) with new twists . Our numerical evidence demonstrate that our proposed changes can have considerable impact on graph generation quality , and can capture complex relationships . Furthermore , GG-GAN is significantly faster that SotA ( autoregressive ) models , while also being competitive in terms of captured statistics and novelty . A case in point , our method can generate graphs of 10k nodes in ∼ 0.2 seconds , which is 2 orders of magnitude faster than the fastest autoregressive method within the state of the art . 1.2 RELATION TO EXISTING WORK . We argue that despite the impressive progress so far , no current approach satisfactorily meets the four aforementioned properties . Isomorphism inconsistent methods ( Kipf & Welling , 2016 ; You et al. , 2018 ; Bojchevski et al. , 2018 ; De Cao & Kipf , 2018 ; Liao et al. , 2019 ) tend to memorize the training set in the absence of problem-specific rewards ( De Cao & Kipf , 2018 ) . Autoregressive variants , in particular , possess large expressive power , but are limited in scalability as they construct the graph one node or block at a time ( Bojchevski et al. , 2018 ; Li et al. , 2018 ; You et al. , 2018 ; Yang et al. , 2019 ; Liao et al. , 2019 ) . Clever optimization and exploitation of sparsity can help ( Dai et al. , 2020 ) , but only to a degree , as can be seen in our experiments . The closest methods to ours are ScoreMatch ( Niu et al. , 2020 ) , Set2Graph ( Serviansky et al. , 2020 ) and CondGen ( Yang et al. , 2019 ) , all of which are consistent and non-autoregressive . ScoreMatch samples graphs from a learned score function via annealed Langevin dynamics , an approach that we find works well only for small graphs . CondGen combines a variational autoencoder ( VAE ) with a conditional GAN for improved training stability . NetGAN ( Bojchevski et al. , 2018 ) , while being a pure GAN does not operate on distributions of graphs . Instead , it learns to generate random walks on a single large graph using an autoregressive generator and then assembles the graph from these random walks . TagGen ( Zhou et al. , 2020 ) uses a similar model on temporal interaction graphs , but instead of learning a generator , it trains a transformer critic to score random temporal walks and then samples from them in parallel via rejection sampling and assembling the final graph . Neither model directly captures a graph distribution , but only addresses the distributions of the random walks . Our work alleviates the need for such modifications or indirect modeling , demonstrating that a pure GAN approach suffices when the generator has been set-up appropriately . Set2Graph also adopts a geometric perspective to solve partitioning , Delaunay triangulation , and convex hull problems and not to generate graphs . By identifying and solving the collision avoidance problem , our work improves the efficacy of Set2Graph-type approaches , such as ours , in implicit deep generative modeling . Notation . In the sequel , upper-case letters refer to sets , and bold-face letters denote vectors and matrices . We represent any set X = { x1 , . . . , xn } of n points as a matrix X with each row corresponding to some point xi in arbitrary order . Functions , such as neural networks , are denoted by lower case letters , such as f , g. Every undirected weighted graph G = ( V , E , w ) can be defined in terms of a set of n = |V | nodes , a set of edges E with ( vi , vj ) ∈ E if there exists an edge joining the i-th and j-th nodes , and a weight function w : E → R+ indicating the strength of each connection . A simple graph G = ( V , E ) is an undirected graph without weights . Finally , the symbol Sn refers to the group of permutations on [ n ] = ( 1 , . . . , n ) . All proofs are presented in the appendix . 2 FUNDAMENTALS OF GEOMETRIC GRAPH GENERATION . This section recalls some basics of geometric graph generation , presents sufficient conditions for representing graphs spatially ( Section 2.1 ) , and identifies key challenges ( Section 2.2 ) preventing their wide-spread application . Our observations motivate GG-GAN—presented in Section 3 . 2.1 REPRESENTING GRAPHS SPATIALLY . We can represent graphs simply by placing n nodes in some metric space and then by connecting them if their position satisfies some notion of similarity ( Huson & Sen , 1995 ; Penrose et al. , 2003 ; Bollobás et al. , 2007 ; Lovász , 2012 ; Alon & Kupavskii , 2014 ; Glasscock , 2015 ) . More concretely , let x1 , . . . , xn be n points in Rk . We determine whether graph G = ( V , E ) of n = |V | nodes contains edge ( vi , vj ) ∈ E by testing if κ ( xi , xj ) = 1 , for a similarity function κ . It is not hard to see that this particular representation is complete1 over the space of undirected graphs : every undirected graph can be represented by n points as long as k is sufficiently large . For instance , if xi is chosen to be the row of the matrix square-root of the adjacency matrixA of the graph G , and κ ( xi , xj ) = x > i xj , then the representation is complete if and only if k = n. Surprisingly , the embedding dimension k can favorably depend on the sparsity of the graphs we wish to represent . Indeed , for simple graphs of degree at most ∆ , Maehara and Rödl showed that k = 2∆ dimensions suffice ( Maehara & Rödl , 1990 ) . This theorem unfortunately relies on a discrete and non-differentiable similarity function κ and only holds for graphs whose edges have no weights . As shown next , the aforementioned issues can be overcome by relaxing the node degree dependency : Theorem 1 . Let G be a weighted undirected graph with adjacency matrix A ∈ [ 0 , 1 ] n×n and maximum ( combinatorial ) degree ∆ . For any ∈ ( 0 , 1 ) , there exist points x1 , . . . , xn in Rk with k = O ( ∆2 logn 2 ) , for which |A ( i , j ) − ( ex > i xj − 1 ) | ≤ for all i 6= j . Therefore , any weighted graph can be approximately represented by n points in at most k = O ( ∆2 log n/ 2 ) dimensions and this is possible using the differentiable function κ ( x , y ) = ex > y−1 . 2.2 CHALLENGES OF GEOMETRIC GRAPH GENERATION . In the following , we expose three key challenges inherent to geometric generators . 1A vector representation of a set is complete if and only if there exists a surjective function between the representation vectors and the elements of the set . 2.2.1 THE CURSE OF INDEPENDENCE . The simplest way of constructing a generator is by sampling each point xi independently from a distribution Dx . We refer to this approach as random graph generation ( RGG ) ( Penrose et al. , 2003 ) . It is a direct corollary of our Theorem 1 that RGG can be used to generate any simple graph : Corollary 1 . Fix any weighted undirected graph G = ( V , E ) of degree at most ∆ and letA be its adjacency matrix . For any ∈ ( 0 , 1 ) , there exists Dx supported on Rk with k = O ( ∆2 log n/ 2 ) such that with strictly positive probability , |A ( i , j ) − ( ex > i xj − 1 ) | ≤ for all i 6= j . In a deep learning context , one may think of modeling Dx by passing a Normal distribution through a learned function , such as a multi-layer perceptron ( MLP ) : xi = g ( zi ) , where zi ∼ N ( 0 , Im ) , for every i ∈ [ n ] and with m being the dimension of the latent space . Unfortunately , as it is shown next , such generators can not satisfactorily control the probability with which each set is generated . Proposition 1 . RGG generates any { x1 , . . . , xn } with xi 6= xj with probability at most O ( ne−n ) . Random graph generators are therefore weak since they sample every point in the set independently of the other points : the larger n is , the smaller the probability that some specific points are sampled .

Summary: In general, this paper deals with an interesting and essential problem to generate geometric graphs under several standards. The whole algorithm seems easy to implement or reproduce. It seems with minor modifications to traditional autoregressor based generative graph models, the proposed framework can effectively model isomorphism as well as delivers certain novelty. The idea of the paper is with novelty and some theorems can support the observations.