Title: HiGen: Hierarchical Graph Generative Networks
Venue: NeurIPS 2023
Accepted: no
Keywords: Generative Models

Graph Generative Network

Graph Neural Network
Abstract: Most real-world graphs exhibit a hierarchical structure, which is often overlooked by existing graph generation methods. To address this limitation, we propose a novel graph generative network that captures the hierarchical nature of graphs and successively generates the graph sub-structures in a coarse-to-fine fashion. 
At each level of hierarchy, this model generates communities in parallel, followed by the prediction of cross-edges between communities using a separate model. This modular approach results in a highly scalable graph generative network. Moreover, we model the output distribution of edges in the hierarchical graph with a multinomial distribution and derive a recursive factorization for this distribution, enabling us to generate sub-graphs with integer-valued edge weights in an autoregressive approach. Empirical studies demonstrate that the proposed generative model can effectively  capture both local and global properties of graphs and achieves state-of-the-art performance in terms of graph quality on various benchmarks.

Abstract

Most real-world graphs exhibit a hierarchical structure, which is often overlooked by existing graph generation methods. To address this limitation, we propose a novel graph generative network that captures the hierarchical nature of graphs and successively generates the graph sub-structures in a coarse-to-fine fashion. At each level of hierarchy, this model generates communities in parallel, followed by the prediction of cross-edges between communities using a separate model. This modular approach results in a highly scalable graph generative network. Moreover, we model the output distribution of edges in the hierarchical graph with a multinomial distribution and derive a recursive factorization for this distribution, enabling us to generate sub-graphs with integer-valued edge weights in an autoregressive approach. Empirical studies demonstrate that the proposed generative model can effectively capture both local and global properties of graphs and achieves state-of-the-art performance in terms of graph quality on various benchmarks.

1 Introduction

Graphs play a fundamental role in representing relationships and are widely applicable in various domains. The task of generating graphs from data holds immense value for diverse applications but also poses significant challenges (Dai et al., 2020). Some of the applications include: the exploration of novel molecular and chemical structures (Jin et al., 2020), document generation (Blei et al., 2003), circuit design (Mirhoseini et al., 2021), the analysis and synthesis of realistic data networks, as well as the synthesis of scene graphs in computer (Manolis Savva et al., 2019; Ramakrishnan et al., 2021).

In all the aforementioned domains, a common observation is the presence of locally heterogeneous edge distributions in the graph representing the system, leading to the formation of clusters or communities and hierarchical structures. These clusters represent groups of nodes characterized by a high density of edges within the group and a comparatively lower density of edges connecting the group with the rest of the graph. In a hierarchical structure that arise from graph clustering, the communities in the lower levels capture the local structures and relationships within the graph. These communities provide insights into the fine-grained interactions among nodes. On the other hand, the higher levels of the hierarchy reflect the broader interactions between communities and characterize global properties of the graph. Therefore, in order to generate realistic graphs, it is essential for graph generation models to learn this multi-scale structure, and be able to capture the cross-level relations. While hierarchical multi-resolution generative models were developed for specific data types such as voice (Oord et al., 2016), image (Reed et al., 2017; Karami et al., 2019) and molecular motifs Jin et al. (2020), these methods rely on domain-specific priors that are not applicable to general graphs with unordered nature. To the best of our knowledge, there exists no data-driven generative models specifically designed for generic graphs that can effectively incorporate hierarchical structure.

Graph generative models have been extensively studied in the literature. Classical methods based on 37 random graph theory, such as those proposed in Erdos & Rényi (1960) and Barabási & Albert (1999), 38

can only capture a limited set of hand-engineered graph statistics. Leskovec et al. (2010) leveraged 39 the Kronecker product of matrices but the resulting generative model is very limited in modeling 40 the underlying graph distributions. With recent advances in graph neural networks, a variety of 41 deep neural network models have been introduced that are based on variational autoencoders (VAE) 42 (Kingma & Welling, 2013) or generative adversarial networks (GAN) (Goodfellow et al., 2020). 43 Some examples of such models include (De Cao & Kipf, 2018; Simonovsky & Komodakis, 2018; 44 Kipf & Welling, 2016; Ma et al., 2018; Liu et al., 2019; Bojchevski et al., 2018; Yang et al., 2019) 45 The major challenge in VAE based models is that they rely on heuristics to solve a graph matching 46 problem for aligning the V AE's input and sampled output, limiting them to small graphs. On the other 47 hand, GAN-based methods circumvent the need for graph matching by using a permutation invariant 48 discriminator. However, they can still suffer from convergence issues and have difficulty capturing 49 complex dependencies in graph structures for moderate to large graphs Li et al. (2018); Martinkus 50 et al. (2022). To address these limitations, (Martinkus et al., 2022) recently proposed using spectral 51 conditioning to enhance the expressivity of GAN models in capturing global graph properties. 52

53

54

55

56

57

58

59

60

61

62

63

64

65

On the other hand, autoregressive models approach graph generation as a sequential decision-making process. Following this paradigm, Li et al. (2018) proposed generative model based on GNN but it has high complexity of O ( mn 2 ) . In a distinct approach, GraphRNN (You et al., 2018) modeled graph generation with a two-stage RNN architecture for generating new nodes and their links, respectively. However, traversing all elements of the adjacency matrix in a predefined order results in O ( n 2 ) time complexity making it non-scalable to large graphs. In contrast, GRAN (Liao et al., 2019) employs a graph attention network and generates the adjacency matrix row by row, resulting in a O ( n ) complexity sequential generation process. To improve the scalability of generative models, Dai et al. (2020) proposed an algorithm for sparse graphs that decreases the training complexity to O (log n ) , but at the expense of increasing the generation time complexity to O (( n + m ) log n ) . Despite their improvement in capturing complex statistics of the graphs, autoregressive models highly rely on an appropriate node ordering and do not take into account the community structures of the graphs. Additionally, due to their recursive nature, they are not fully parallelizable.

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

A new family of diffusion model for graphs has emerged recently. Continuous denoising diffusion was developed by Jo et al. (2022), which adds Gaussian noise to the graph adjacency matrix and node features during the diffusion process. However, since continuous noise destroys the sparsity and structural properties of the graph, discrete denoising diffusion models have been developed as a solution in (Haefeli et al., 2022; Vignac et al., 2022). These models progressively edit graphs by adding or removing edges in the diffusion process, and then denoising graph neural networks are trained to reverse the diffusion process. While the denoising diffusion models can offer promising results, their main drawback is the requirement of a long chain of reverse diffusion, which can result in relatively slow sampling.

In his work, we introduce HiGen, a Hi erarchical G raph G en erative Network to address the limitations of existing generative models by incorporating community structures and cross-level interactions. This approach involves generating graphs in a coarse-to-fine manner, where graph generation at each level is conditioned on a higher level (lower resolution) graph. The generation of communities at lower levels is performed in parallel, followed by the prediction of cross-edges between communities using a separate model. This parallelized approach enables high scalability. To capture hierarchical relations, our model allows each node at a given level to depend not only on its neighbouring nodes but also on its corresponding super-node at the higher level. Furthermore, we address the generation of integervalued edge weights of the hierarchical structure by modeling the output distribution of edges using a multinomial distribution. We show that multinomial distribution can be factorized successively, enabling the autoregressive generation of each community. This property makes the proposed architecture well-suited for generating graphs with integer-valued edge weights. Furthermore, by breaking down the graph generation process into the generation of multiple small partitions that are conditionally independent of each other, HiGen reduces its sensitivity to a predefined initial ordering of nodes.

2 Background

A graph G = ( V , E ) is a collection of nodes (vertices) V and edges E with corresponding sizes 91 n = |V| and m = |E| and an adjacency matrix A π for the node ordering π . The node set can 92 be partitioned into c communities (a.k.a. cluster or modules) using a graph partitioning function 93

(a)

(b)

(c)

(d)

(e)

0=1

29

→~binomial

～multinomial

29

8

8

21

7

6

14

5

29

8

8

2

21

7

1

6

l=2

14

0

2

5

2

F : V → { 1 , ..., c } , where each cluster of nodes forms a sub-graph denoted by C i = ( V ( C i ) , E ( C i )) 94 with adjacency matrix A i . The cross-links between neighboring communities form a bipartite 95 graph , denoted by B ij = ( V ( C i ) , V ( C j ) , E ( B ij )) with adjacency matrix A ij . Each community 96 is aggregated to a super-node and each bipartite corresponds to a super-edge linking neighboring 97 communities, which induces a coarser graph at the higher (a.k.a. parent) level. Herein, the levels are 98 indexed by superscripts. Formally, each community at level l , C l i , is mapped to a node at the higher 99 level graph, also called its parent node, v l -1 i := Pa ( C l i ) and each bipartite at level l is represented by 100 an edge in the higher level, also called its parent edge, e l -1 i = Pa ( B l ij ) = ( v l -1 i , v l -1 j ) . The weights 101 of the self edges and the weights of the cross-edges in the parent level are determined by the sum of the 102 weights of the edges within their corresponding community and bipartite, respectively. Therefore, the 103 edges in the induced graphs at the higher levels have integer-valued weights: w l -1 ii = ∑ e ∈ E ( C l i ) w e 104 and w l -1 ij = ∑ e ∈ E ( B l ij ) w e , moreover sum of all edge weights remains constant in all levels so 105 w 0 := ∑ e ∈ E ( G l ) w e = |E| , ∀ l ∈ [0 , ..., L ] . 106

This clustering process continues recursively in a bottom-up approach until a single node graph G 0 is 107 obtained, producing a hierarchical graph , defined by the set of graphs in all levels of abstractions, 108 HG := {G 0 , ...., G L -1 , G L } . This forms a dendrogram tree with G 0 being the root and G L being the 109 final graph that is generated at the leaf level. An HG is visualized in figure 1a. The hierarchical tree 110

structure enables modeling of both local and long-range interactions among nodes, as well as control 111 over the flow of information between them, across multiple levels of abstraction. This is a key aspect 112 of our proposed generative model. 113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

3 Hierarchical Graph Generation

In graph generative networks, the objective is to learn a generative model, p ( G ) given a set of training graphs. This work aims to establish a hierarchical multi-resolution framework for generating graphs in a coarse-to-fine fashion. In this framework, we assume that the graphs do not have node attributes, so the generative model only needs to characterize the graph topology. Given a particular node ordering π , and a hierarchical graph HG := {G 0 , ...., G L -1 , G L } , produced by recursively applying a graph partitioning function, F , we can factorize the generative model using the chain rule of probability as:

In other words, the generative process involves specifying the probability of the graph at each level conditioned on its parent level graph in the hierarchy. This process is iterated recursively until the lowest level, or leaf level, is reached. Here, the distribution of the root p ( G 0 ) = p ( w 0 = w 0 ) can be simply estimated using the empirical distribution of the number of edges |E| of graphs in the training set.

Based on the partitioned structure within each level of HG , the conditional generative probability p ( G l | G l -1 ) can be decomposed into the probability of its communities and bipartite graphs as:

The approximation in this decomposition becomes an equivalence when each community C l i or bipartite graph B l ij is assumed to be independent of all other components in its level conditioned on the parent graph G l -1 . . 1 Since the integer-valued weights of the edges in each level can be modeled by a multinomial distribution, we can leverage the properties of multinomial distribution to prove the conditional independence of the components.

Theorem 3.1. Let the random vector w := [ w e ] e ∈ E ( G l ) denote the set of weights of all edges of G l such that their sum is w 0 = 1 T w . The joint probability of w can be described by a multinomial distribution: w ∼ Mu ( w | w 0 , θ l ) . By observing that the sum of edge weights within each community C l i and bipartite graph B l ij are determined by the weights of their parent edges in the higher level, w l -1 ii and w l -1 ij respectively, we can establish that these components are conditionally independent and each of them follow a multinomial distribution:

where

{

θ

l

ij

[

e

]

∈

[0

,

1]

,

1

θ

l

ij

= 1

|

∀

(

i, j

)

∈

E

(

G

-

Therefore, given the parent graph at a higher level, the generation of graph at its subsequent level can be reduced to generation of its partition and bipartite sub-graphs. As illustrated in figure, this decomposition enables parallel generation of the communities in each level which can be followed by predicting all bipartite sub-graphs in that level at one pass. Each of these sub-graphs corresponds to a block in the adjacency matrix, as visualized in figure 1b, so the proposed hierarchical model generates adjacency matrix in a blocks-wise fashion and constructs the final graph topology.

T

l

1

)

}

are the multinomial model parameters.

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

3.1 Community Generation

Based on the equation (3), the edge weights within each community can be jointly modeled using a multinomial distribution. Our objective is to model the generative probability of communities in each level as an autoregressive process. To accomplish this, we need to factorize the multinomial distribution accordingly. Toward this goal, we present two different approaches in the following.

Lemma 3.2. A random counting vector w ∈ Z E + with a multinomial distribution can be recursively decomposed into a sequence of binomial distributions as follows:

This decomposition is known as a stick-breaking process, where ˆ θ e is the fraction of the remaining probabilities we take away every time and allocate to the e -th component (Linderman et al., 2015).

This lemma enable us to model the generation of a community as an edge-by-edge autoregressive process, similar to existing algorithms such as GraphRNN (You et al., 2018) or DeepGMG (Li et al., 2018) with O ( |V C | 2 ) generation steps. However, inspired by GRAN (Liao et al., 2019), a community can be generated more efficiently by generating one node at a time. This requires decomposing the generative probability of edges in a group-wise form, where the candidate edges between the t -th node and the already generated graph are grouped together. In other words, this model completes the lower triangle adjacency matrix one row at a time conditioned on the already generated sub-graph and the parent-level graph. The following theorem formally derives this decomposition for multinomial distributions.

Theorem 3.3. For a random counting vector w ∈ Z E + with a multinomial distribution Mu ( w | w, θ ) , let's split it into M disjoint groups w = [ u 1 , ..., u M ] where u m ∈ Z E m + , ∑ M m =1 E m = E , and also split the probability vector accordingly as θ = [ θ 1 , ..., θ M ] . Additionally, let's define sum of all variables in the m -th group by a random count variable v m := ∑ E m e =1 u m,e . Then, the multinomial distribution can be factorized as a chain of binomial and multinomial distributions:

Here, the probability of binomial, η v m , is the fraction of the remaining probability mass that is 170 allocated to v m , i.e. the sum of all weights in the m -th group. The vector parameter λ m is the 171 normalized multinomial probabilities of all count variables in the m -th group. Intuitively, this 172 decomposition of multinomial distribution can be viewed as a recursive stick-breaking process where 173 at each step, first a binomial distribution is used to determine how much probability mass to allocate 174 to the current group, and a multinomial distribution is used to distribute that probability mass 175 among the variables in the group. The resulting distribution is equivalent to the original multinomial 176 distribution. 177

178

179

180

181

row of the lower triangle of adjacency matrix ˆ A l i ), where the sum of the candidate edge weights is 182 v t . Based on theorem 3.3, the probability of u t can be characterized by the product of a binomial 183 and a multinomial distribution. This process is illustrated in figure 1c. We further increase the 184

Proof. Refer to appendix A.2 for the proof.

Let ˆ C l i,t denote an already generated sub-graph, at the t -th step, augmented with the set of candidate edges, from the new node, v t ( C l i ) , to its preceding node denoted by ˆ E t ( ˆ C l i,t ) := { ( t, j ) | j < t } . We collect the weights of these edges in the random vector u t := [ w e ] e ∈ ˆ E t ( ˆ C l i,t ) (that is the t -th

expressiveness of the generative network by extending this probability to a mixture model with K 185 mixtures: 186

Where ∆ h ˆ E t ( ˆ C l i,t ) is a | ˆ E t ( ˆ C l i,t ) | × d h dimensional matrix, consisting of the set of edge features { ∆ h ( t,s ) := h t -h s | ∀ ( t, s ) ∈ ˆ E t ( ˆ C l i,t ) } , h ˆ C l i,t is a t × d h matrix of node features in the augmented community graph. The mixture weights are denoted by β l . Here, the node features are learned by GNN models and the graph level representation is obtained by the addpool() aggregation function. In order to produce K × |E t ( C l i ) | dimensional matrix of multinomial probabilities, the MLP l θ () network acts at the edge level, while MLP l η v () and MLP l β () act at the graph level to produce the binomial probabilities and K dimensional arrays for K mixture models, respectively. All of these MLP networks are built by two hidden layers with ReLU() activation functions.

During the generation process of each community C l i , the node features of its parent node h Pa ( C l i ) are used as the context. This context is concatenated to the node and edge feature matrices using the operation [ x || y ] , which concatenates vector y to each row of matrix x . The purpose of this context is to enrich the node and edge features by capturing long-range interactions and encoding the global structure of the graph, which is important for generating local components.

3.2 Bipartite Generation

Once all the communities in level l are generated, the edges of all bipartite graphs at that level can be predicted simultaneously. An augmented graph ˆ G l composed of all the communities, {C l i ∀ i ∈ V ( G l -1 ) } , and the candidate edges of all bipartites, {B l ij ∀ ( i, j ) ∈ E ( G l -1 ) } , is used as the input of a GNN to obtain node and edge features. We similarly extend the multinomial distribution of a bipartite, (12), using a mixture model to express its generative probability:

where the random vector w := [ w e ] e ∈ ˆ E ( B l ij ) is the set of weights of all candidate edges in bipartite B l ij and ∆ h Pa ( B l ij ) are the parent edge features of the bipartite graph.

Node Feature Encoding: To encode node features, we extend GraphGPS proposed by Rampá š ek et al. (2022). GraphGPS combines local message-passing with global attention mechanism and uses positional and structural encoding for nodes and edges to construct a more expressive and a scalable graph transformer (GT) (Dwivedi & Bresson, 2020). To apply GraphGPS on augmented graphs, we use distinct initial edge features to distinguish augmented (candidate) edges from real edges. Furthermore, for bipartite generation, the attention scores in the Transformers of the augmented graph ˆ G l are masked to restrict attention only to connected communities. The details of model architecture are provided in appendix B.

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

4 Related Work

In order to deal with hierarchical structures in molecular graphs, a generative process was proposed by Jin et al. (2020) which recursively selects motifs, the basic building blocks, from a set and predicts their attachment to the emerging molecule. However, this method requires prior domainspecific knowledge and relies on molecule-specific graph motifs. Additionally, the graphs are only abstracted into two levels, and component generation cannot be performed in parallel. In (Kuznetsov & Polykovskiy, 2021), a hierarchical normalizing flow model for molecular graphs was introduced, where new molecules are generated from a single node by recursively dividing each node into two. However, the merging and splitting of pairs of nodes in this model is based on the node's neighborhood, and do not consider the diverse community structure of graphs, therefore the hierarchical generation of this model is inherently limited.

5 Experiments

In our empirical studies, we compare the proposed hierarchical graph generative network against state-of-the-art autoregressive models: GRAN and GraphRNN models, diffusion models: DiGress (Vignac et al., 2022) and GDSS (Jo et al., 2022) and a GAN-based model: SPECTRE (Martinkus et al., 2022), on a range of synthetics and real datasets of various sizes.

Datasets: We used 4 different benchmark graph datasets: (1) the synthetic Stochastic Block Model (SBM) dataset consisting of 200 graphs with 2-5 communities each with 20-40 nodes, used in a previous work (Martinkus et al., 2022); (2) the Protein including 918 protein graphs, each has 100 to 500 nodes representing amino acids that are linked if they are closer than 6 Angstroms (Dobson & Doig, 2003), (3) the Enzyme that has 587 protein graphs of 10-125 nodes, representing protein tertiary structures of the enzymes from the BRENDA database (Schomburg et al., 2004) and (4) the Ego dataset containing 757 3-hop ego networks with 50-300 nodes extracted from the CiteSeer dataset, where nodes represent documents and edges represent citation relationships (Sen et al., 2008).

Graph Partitioning Different algorithms approach the problem of graph partitioning (clustering) using various clustering quality functions. Two commonly used families of such metrics are modularity and cut-based metrics (Tsitsulin et al., 2020). Although optimizing modularity metric is an NP-hard problem, it is well-studied in the literature and several graph partitioning algorithm based on this metric have been proposed. For example, the Louvain algorithm (Blondel et al., 2008) starts with each node as its community and then repeatedly merges communities based on the highest increase in modularity until no further improvement can be made. This heuristic algorithm is computationally efficient and scalable to large graphs for community detection. Moreover, a spectral relaxation of modularity metrics has been proposed in Newman (2006a,b) which results in an analytically solution for graph partitioning. Additionally, an unsupervised GNN-based pooling method inspired by this spectral relaxation was proposed for partitioning graphs with node attributes (Tsitsulin et al., 2020). As the modularity metric is based on the graph structure, it is well-suited for our problem. Therefore, we employed the Louvain algorithm to hierarchically cluster the graph datasets in our experiments and then spliced out the intermediate levels to achieve HGs with uniform depth of L = 2 .

Model Architecture In the experiments, the GNN models consist of 8 layers of GraphGPS layers (Rampá š ek et al., 2022). The input node feature of GNNs is augmented with positional and structural encoding, where the first 8 eigenvectors corresponding to the smallest non-zero eigenvalues of the Laplacian and diagonal of the random-walk matrix up to 8 -steps are used. Each level has its own GNN and output models. The details of the model architecture are presented in Appendix B and C.

We conducted experiments using the proposed hierarchical graph generative network (HiGen) model with two variants for the output distribution of the leaf edges: 1) HiGen : the probability of the community edges' weights at the leaf level are modeled by mixture of Bernoulli, using sigmoid() activation in equation 7, since the leaf levels in our experiments have binary edges weights, while higher levels use mixture of multinomials. 2) HiGen-m : the model uses a mixture of multinomial distributions (6) to describe the output distribution for all levels. In this case, we observed that modeling the probability parameters of edge weights of the leaf level, denoted as λ t,k in (7), by a multihot activation function, defined as σ ( z ) i := sigmoid( z i ) / ∑ K j =1 sigmoid( z j ) where σ : R K → ( K -1) -simplex, provided slightly better performance than the standard softmax() function. However, for

Enzyme

Clus.

↓

0.0025

0.062

0.087

0.061

0.157

0

Deg.

↓

0.0011

0.017

0.054

0.026

0.027

0

.

012

.

038

Orbit

↓

3.7e-4

0.046

0.033

0.009

1.2e-3

7

.

2

Deg.

↓

2.2e-4

0.024

0.032

-

0.011

1

.

9

Clus.

Ego

↓

0.010

0.34

0.17

-

0.063

0

.

049

Spec.

↓

1.4e-3

0.089

0.046

-

0.013

0

.

004

Model

Training set

GraphRNN

GRAN

GDSS

HiGen-m

HiGen e-4

e-3

Orbit

↓

0.012

0.14

0.026

-

0

.

021

0.029

(a)

(b)

(c)

(d)

both HiGen and HiGen-m, the probabilities of the integer-valued edges at the higher levels are still 268 modeled by the standard softmax() function. 2 269

For training, HiGen models used the Adam optimizer Kingma & Ba (2014) with a learning rate of 270 5e-4 and its default settings of β 1 = 0 . 9 , β 2 = 0 . 999 and /epsilon1 =1e-8. 271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

Metrics To evaluate the graph generative models, we adopt the approach proposed in (Liu et al., 2019; Liao et al., 2019), which compares the distributions of four different graph statistics between the ground truth and generated graphs: (1) degree distributions, (2) clustering coefficient distributions, (3) the number of occurrences of all orbits with four nodes, and (4) the spectra of the graphs by computing the eigenvalues of the normalized graph Laplacian. The first three metrics capture local graph statistics, while the spectra represents global structure. The maximum mean discrepancy (MMD) score over these statistics are used as the metrics. While Liu et al. (2019) computed MMD scores using the computationally expensive Gaussian earth mover's distance (EMD) kernel, Liao et al. (2019) proposed using the total variation (TV) distance as an alternative measure. TV distance is much faster and still consistent with the Gaussian EMD kernel. Most recently, O'Bray et al. (2021) suggested using other efficient kernels such as an RBF kernel, or a Laplacian kernel, or a linear kernel. Additionally, Thompson et al. (2022) proposed new evaluation metrics for comparing graph sets using a random-GNN approach where GNNs are employed to extract meaningful graph features. However, in this work, we follow the experimental setup and evaluation metrics of (Liao et al., 2019), except for the enzyme dataset where we use a Gaussian EMD kernel to be consistent with the results reported in (Jo et al., 2022). GNN-based performance metrics of HiGen model are also reported in appendix D.2.

The performance metrics of the proposed HiGen models are reported in Table 1, with generated graph 289 samples presented in Figure 2. The results demonstrate that HiGen effectively captures graph statistics 290 and achieves state-of-the-art on all the benchmarks graphs across various generation metrics. This 291 improvement in both local and global properties of the generated graphs highlights the effectiveness 292 of the hierarchical graph generation approach, which models communities and cross-community 293 interactions separately. The visual comparisons of graph samples generated by the HiGen models, 294 as well as the experimental evaluation of different node ordering and partitioning functions, are 295 presented in Appendix D.2. 296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

6 Discussion

The GRAN model can generate graphs one block of nodes at a time in an autoregressive fashion where the block size is fixed and nodes are assigned to blocks based on an ordering. However, the model's performance deteriorates as the block size increases, since adjacent nodes in an ordering may not be relevant and may belong to different clusters. Additionally, intra-block connections are not modeled separately. In contrast, our proposed method generates blocks of nodes within each community that have strong relationships and then predicts the cross-links between communities using a separate model. As a result, this approach enables the model to capture both local relationships between nodes within a community and global relationships across communities, resulting in improved expressiveness of the graph generative model.

The proposed hierarchical model allows for highly parallelizable training and generation. Specifically, let n c be the size of the largest graph cluster, then, it only requires O ( n c log n ) sequential steps to generate a graph of size n .

Node ordering sensitivity The predefined ordering of dimensions can be crucial for training autoregressive (AR) models Vinyals et al. (2015), and this sensitivity to node orderings is particularly pronounced in autoregressive graph generative model Liao et al. (2019); Chen et al. (2021). However, in the proposed approach, the graph generation process is divided into the generation of multiple small partitions, performed sequentially across the levels, rather than generating the entire graph by a single AR model. Therefore, given an ordering for the parent level, the graph generation depends only on the permutation of the nodes within the graph communities rather than the node ordering of the entire graph. In other words, the proposed method is invariant to a large portion of possible node permutations, and therefore the set of distinctive adjacency matrices is much smaller in HiGen. For example, the node ordering π 1 = [ v 1 , v 2 , v 3 , v 4 ] with clusters V G 1 = { v 1 , v 2 } and V G 2 = { v 3 , v 4 } has a similar hierarchical graph as π 2 = [ v 1 , v 3 , v 2 , v 4 ] , since the node ordering within the communities is preserved at all levels. Formally, let {C l i ∀ i ∈ V G l -1 } be the set of communities at level l produced by a deterministic partitioning function, where n l i = |V ( C l i ) | denotes the size of each partition. The upper bound on the number of distinct node orderings in an HG generated by the proposed process is then reduced to ∏ L l =1 ∏ i n l i ! . 3

7 Conclusion

The proposed HiGen framework generates graphs in a hierarchical and block-wise manner, leveraging 326 the inherent hierarchical structure present in real-world graphs. By decomposing the generation 327 process into separate and parallel generation of communities and bipartite sub-graphs, it combines the 328 benefits of one-shot and AR graph generative models. Experimental results on benchmark datasets 329 demonstrate that HiGen achieves state-of-the-art performance across various generation metrics. The 330 hierarchical and block-wise generation strategy of HiGen enables scaling up graph generative models 331 to large and complex graphs, opening up opportunities to extend it to newer generative paradigms 332 such as diffusion models. 333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

References

Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. science , 286(5439): 509-512, 1999.

David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research , 3(Jan):993-1022, 2003.

Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment , 2008(10):P10008, 2008.

Aleksandar Bojchevski, Oleksandr Shchur, Daniel Zügner, and Stephan Günnemann. Netgan: Generating graphs via random walks. arXiv preprint arXiv:1803.00816 , 2018.

Xiaohui Chen, Xu Han, Jiajing Hu, Francisco JR Ruiz, and Liping Liu. Order matters: Probabilistic modeling of node sequence for graph generation. arXiv preprint arXiv:2106.06189 , 2021.

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 , 2020.

Hanjun Dai, Azade Nazi, Yujia Li, Bo Dai, and Dale Schuurmans. Scalable deep generative modeling for sparse graphs. In International Conference on Machine Learning , pp. 2302-2312. PMLR, 2020.

Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint arXiv:1805.11973 , 2018.

Paul D Dobson and Andrew J Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology , 330(4):771-783, 2003.

Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699 , 2020.

Paul Erdos and Alfréd Rényi. On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci , 5(1):17-60, 1960.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11): 139-144, 2020.

Kilian Konstantin Haefeli, Karolis Martinkus, Nathanaël Perraudin, and Roger Wattenhofer. Diffusion models for graphs benefit from discrete state spaces. arXiv preprint arXiv:2210.01549 , 2022.

Hossein Hajimirsadeghi, Wang Yan, Arash Vahdat, and Greg Mori. Visual recognition by counting instances: A multi-instance cardinality potential kernel. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 2596-2605, 2015.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learning , pp. 4839-4848. PMLR, 2020.

Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In International Conference on Machine Learning , pp. 10362-10383. PMLR, 2022.

Mahdi Karami, Dale Schuurmans, Jascha Sohl-Dickstein, Laurent Dinh, and Daniel Duckworth. Invertible convolutional flow. Advances in Neural Information Processing Systems , 32, 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Maksim Kuznetsov and Daniil Polykovskiy. Molgrow: A graph normalizing flow for hierarchical molecular generation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pp. 8226-8234, 2021.

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

Jure Leskovec, Deepayan Chakrabarti, Jon Kleinberg, Christos Faloutsos, and Zoubin Ghahramani. Kronecker graphs: An approach to modeling networks. Journal of Machine Learning Research , 11(Feb):985-1042, 2010.

Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. arXiv preprint arXiv:1803.03324 , 2018.

Renjie Liao, Yujia Li, Yang Song, Shenlong Wang, Will Hamilton, David K Duvenaud, Raquel Urtasun, and Richard Zemel. Efficient graph generation with graph recurrent attention networks. Advances in neural information processing systems , 32, 2019.

Scott Linderman, Matthew J Johnson, and Ryan P Adams. Dependent multinomial models made easy: Stickbreaking with the pólya-gamma augmentation. Advances in Neural Information Processing Systems , 28, 2015.

Jenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, and Kevin Swersky. Graph normalizing flows, 2019.

Tengfei Ma, Jie Chen, and Cao Xiao. Constrained generation of semantically valid graphs via regularizing variational autoencoders. arXiv preprint arXiv:1809.02630 , 2018.

Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019.

Karolis Martinkus, Andreas Loukas, Nathanaël Perraudin, and Roger Wattenhofer. Spectre: Spectral conditioning helps to overcome the expressivity limits of one-shot graph generators. In International Conference on Machine Learning , pp. 15159-15179. PMLR, 2022.

Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al. A graph placement methodology for fast chip design. Nature , 594(7862):207-212, 2021.

Mark EJ Newman. Finding community structure in networks using the eigenvectors of matrices. Physical review E , 74(3):036104, 2006a.

Mark EJ Newman. Modularity and community structure in networks. Proceedings of the national academy of sciences , 103(23):8577-8582, 2006b.

Leslie O'Bray, Max Horn, Bastian Rieck, and Karsten Borgwardt. Evaluation metrics for graph generative models: Problems, pitfalls, and practical solutions. arXiv preprint arXiv:2106.01098 , 2021.

Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 , 2016.

Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021.

Ladislav Rampá š ek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems , 35:14501-14515, 2022.

Scott Reed, Aäron Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation. In International Conference on Machine Learning , pp. 2912-2921. PMLR, 2017.

Ida Schomburg, Antje Chang, Christian Ebeling, Marion Gremse, Christian Heldt, Gregor Huhn, and Dietmar Schomburg. Brenda, the enzyme database: updates and major new developments. Nucleic acids research , 32 (suppl_1):D431-D433, 2004.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Kyle Siegrist. Probability, Mathematical Statistics, Stochastic Processes . LibreTexts, 2017. URL https://stats.libretexts.org/Bookshelves/Probability_Theory/Probability_ Mathematical_Statistics_and_Stochastic_Processes_(Siegrist) .

433

434

435

436

437

438

439

440

441

442

443

444

445

446

Martin Simonovsky and Nikos Komodakis. GraphVAE: Towards generation of small graphs using variational autoencoders. arXiv preprint arXiv:1802.03480 , 2018.

Rylee Thompson, Boris Knyazev, Elahe Ghalebi, Jungtaek Kim, and Graham W Taylor. On evaluation metrics for graph generative models. arXiv preprint arXiv:2201.09871 , 2022.

Anton Tsitsulin, John Palowitch, Bryan Perozzi, and Emmanuel Müller. Graph clustering with graph neural networks. arXiv preprint arXiv:2006.16904 , 2020.

Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734 , 2022.

Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint arXiv:1511.06391 , 2015.

Carl Yang, Peiye Zhuang, Wenhan Shi, Alan Luu, and Pan Li. Conditional structure generation through graph variational generative adversarial nets. Advances in neural information processing systems , 32, 2019.

Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. In ICML , pp. 5694-5703, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes another auto-regressive-based graph generative model similar to GRAN. The authors propose a hierarchical generation scheme to un-coarse a graph level by level. In each non-leaf level, the abstract graph is weighted both in nodes and edges. A node represents a community, and its weight represents how many edges should be inside the community. An edge is the \"connection\" between two communities, and its weight represents how many edges should exist between the two communities. The weights of each community are generated through a stick-breaking process. And the number of communities is automatically decided by it. The structure within the community is generated using an AR model. Then the edges between communities are generated using GNN.\n\nSTRENGTHS:\n1. The assumption makes sense, and the model decomposition is quite convincing.\n2. This method can indeed improve generation efficiency by only auto-regressively generating the diagonal blocks of the adjacency matrix and using GNN (which has O(M) runtime) to predict the off-block entries.\n3. The method is simple and straightforward.\n\nWEAKNESSES:\nsee questions below\n\nQUESTIONS:\n1. The partition function is heuristics (see experiment) and I think this is very important for training a good model, the author should elaborate more on how to generalize it to the case where L>2.\n2. Optimization detail is missing, I'd like to see how the model is being optimized and what the data format is for every level. Do you need to train a model for every level?\n3. It would be nice to see the runtime analysis since it's one of the motivations mentioned in the introduction.\n4. The experiment only shows model performance for the case where L=2, additional experiments should be included. Otherwise, it's just a small modification from GRAN and will limit the contribution. Also, I suggest the author experiment on even larger graphs and compare the model performance to [1,2,3]\n5. Since the edges of the inter-community are generated in parallel, I am concerned about the expressivity of the model due to the edge independence [2]. It would be nice to see some discussion on it.\n\nSome related works are missing:\n\n[1] Rendsburg, Luca, Holger Heidrich, and Ulrike Von Luxburg. \"Netgan without gan: From random walks to low-rank approximations.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Chanpuriya, Sudhanshu, et al. \"On the power of edge independent graph models.\" Advances in Neural Information Processing Systems 34 (2021): 24418-24429.\n\n[3] Haefeli, Kilian Konstantin, et al. \"Diffusion Models for Graphs Benefit From Discrete State Spaces.\" arXiv preprint arXiv:2210.01549 (2022).\n\n[4] Chen, Xiaohui, et al. \"Efficient and Degree-Guided Graph Generation via Discrete Diffusion Modeling.\" arXiv preprint arXiv:2305.04111 (2023).\n\n[5] Kong, Lingkai, et al. \"Autoregressive Diffusion Model for Graph Generation.\" (2023).\n\nLIMITATIONS:\n1. one possible limitation is the edge independency of the model.\n2. The model has made a strong assumption that the graph should have a community structure, while the experiment datasets are relatively small and may not have such a structure.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes HiGen a hierarchical generative graph model. The model consists of a clustering process (Louvain), followed by a GNN model (GraphGPS) to estimate probabilities. The generative process is separated by communities and bipartite sub-graphs.\n\nSTRENGTHS:\nThe paper seems original. The proposition of a new model is always an important contribution. Even though the new model is the combination of a clustering process and GNN. The combination of both ideas is interesting. \n\nThe theoretical quality of the demonstrations is good. Most of them seem fine and no errors were observed during the revision. \n\nParts of the papers are quite clear. Figure 1 really helps to understand the main idea of the paper. However, there is room for improvement.\n\nThe significance of the paper is high, it seems that this new model is able to reproduce the mean of the distribution quite correctly in comparison to other state-of-the-art methods, as it is shown in the results.\n\nWEAKNESSES:\nThe state of the art can be improved. The paper mentions \"there exists no data-driven generative models specifically designed for generic graphs that can effectively incorporate hierarchical structure.\". Neville et al. focused on this type of work, generating several papers related to hierarchical graph models (doi.org/10.1145/3161885, doi.org/10.1145/2939672.2939808, doi.org/10.1007/s10618-018-0566-x). \n\nParts of the paper are closed related to mKPGM (doi.org/10.1145/3161885). In both cases there is a hierarchical structure, both have the idea of a super-node at the higher level, and the sampling process is also based on a multinomial distribution (doi.org/10.1007/s10618-018-0566-x). Please take a look at the sampling process proposed, because it has similarities to the proposition of this paper, and the authors claimed to sample a network with billions of edges in less than two minutes.\n\nThe paper must state its main contribution. In the beginning, it seems to be the model, but after reading the paper, it seems to be the sampling process. Unfortunately, both of them have different issues.\n\nIf the main contribution is the model, then the paper should improve the modeling of the main network and be fairly compared in the experiment section against other baselines (not just the mean of the distribution). The main models consider $\\ell$ hierarchies, but just two are applied. It is also not clear how the final probabilities are obtained. \n\nIf the main contribution is the sampling process, there are some issues too. The time complexity of the generative model claims to be O(n_c \\log n), but this is not demonstrated. The results of the paper are focused on the modeling of networks, not the sampling process. For example, there are no empirical results about the time complexity, and the largest networks have some thousand nodes, rather than millions.\n\nI understand that the papers follow the experimental setup and evaluation metrics of Liao et al. However, this methodology must be stated in the main paper, otherwise, the experiments of the main paper are not reproducible. \n\nThe results of Table 1 are difficult to read because of the lack of explanations. There are no details on the separation of the data in the main paper. I understand that this is explained in the supplementary material (80% for training and 20% for testing), but it must be considered in the main paper too. Moreover, I do not know if the values are the average over the 20% of the testing graphs or, if you just considered it as a single distribution. In the first case, please add the standard deviation, to see if the difference at statistically significant. \n\nSection 5 claims: \"The results demonstrate that HiGen effectively captures graph statistics\". Considering that, generally speaking, MMD estimates the distance between the means of two distributions, I suggest you change it to \"The results demonstrate that HiGen effectively captures the mean of the graph statistics\". Given the use of MMD, you can not determine if the other part of the distribution are correctly estimated. \n\nThe conclusions state that HiGen \"enables scaling up graph generative models to large and complex graphs\" but this is not demonstrated.\n\nQUESTIONS:\n-What are the similarities and differences between this sampling process and the sampling process proposed at doi.org/10.1007/s10618-018-0566-x? Can you use that sampling process to speed up your generative process?\n-Why did you consider such small networks?\n-Can you, at least, empirically demonstrated the time complexity of your model? \n-how did you compare the final distribution for the MMD? Did you consider the average of each network, or do you make a single distribution considering all networks?\n\nLIMITATIONS:\nNo, the authors did not consider the limitations of the proposed model. For suggestions, please check weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an innovative hierarchical method for graph generation, which employs multiple levels of graph coarsening. This approach begins with the first level, representing the most coarse graph, and progressively expands nodes and edges to form new communities and connections between the newly created nodes. At each level, nodes serve as communities for the subsequent level, and the edge weights, including both inter-community edges and self-loops, dictate the total number of edges within each community in the final graph. Consideration of independence among the generation processes of inter and intra-community edges, conditioned on the graph and edge weights from previous levels enables parallel execution of the steps, resulting in acceleration of the generation process.\n\nSTRENGTHS:\n1. The paper effectively utilizes hierarchical clustering to enhance the graph generation process, capitalizing on the benefits of this technique.\n2. By introducing parallelization in generating distinct clusters at each level, the paper successfully minimizes the number of sequential steps required.\n3. The experimental results presented in the paper demonstrate improvements across multiple datasets.\n4. Paper for the most part is well-written and easy to follow.\n\nWEAKNESSES:\n1. In lines 35-36 paper mentions that this work is the first hierarchical method for generic graphs. I believe [1] is also a hierarchal method for graph generation. I understand that methods are significantly different but still it would be more accurate to highlight the unique aspects of the proposed method and consider including a comparison between the two approaches.\n2. The time complexity analysis provided in the paper focuses solely on the sequential steps, neglecting to consider the computational requirements. It would be valuable to compare the overall computational workload, particularly since the proposed method utilizes the GraphGPS approach, which has a time complexity of $O(n^2)$, in contrast to conventional GNN methods with a complexity of $O(n+m)$. Including such a comparison would provide a more comprehensive analysis.\n3. The paper lacks a study examining the distribution of community sizes during the generation process across different datasets. Addressing this limitation by investigating and reporting the distribution of community sizes would enhance the understanding of the method's behavior and its adaptability to various datasets.\n4. The paper uses a more advanced GNN compared to methods like GRAN, raising the question of how much of the observed progress is solely due to the change in the GNN architecture. Conducting an ablation study specifically focused on the GNN architecture used would provide valuable insights into its individual contribution to the overall performance of the method.\n5. The evaluation metrics commonly employed for graph generative models have their limitations, as discussed in [1] and [2]. It is important to consider these limitations in the evaluation process. The paper mentions the use of random GNNs as an alternative evaluation method, but this approach is only used in the appendix for a few experiments. I would suggest using this in the main body and comparing all models using this metric. [Additionally/Optionally, there are two more recent approaches, one based on contrastive training and another one based on Ricci curvatures that could be incorporated for evaluation purposes.]\n\n[1] Shirzad, H., Hajimirsadeghi, H., Abdi, A. H., & Mori, G. (2022, May). TD-gen: Graph generation using tree decomposition. In International Conference on Artificial Intelligence and Statistics (pp. 5518-5537). PMLR.\n\n[2] O'Bray, Leslie, et al. \"Evaluation metrics for graph generative models: Problems, pitfalls, and practical solutions.\" arXiv preprint arXiv:2106.01098 (2021).\n\nQUESTIONS:\n1. From line 253 it appears that the number of levels is fixed to 2 for all datasets, is this correct? What is the average size of the largest cluster for different datasets used? This information is required for understanding the levels of improvement the model makes.\n2. The formulation of Theorem 3.1 is confusing, particularly in relation to the assumption of Multinomial distributions. It seems that the theorem does not fully consider the fact that some nodes are shared during the process of distributing edges among nodes for both intra and inter-cluster components. To clarify, let's consider the scenario of learning the distribution over d-regular graphs. In the final level, the number of edges associated with a node within its cluster and the sum of edges from intra-cluster connections must be fixed. Therefore, we cannot treat them as independent, even if we know the sum of the number of edges inside a cluster and the number of edges between each two clusters. It would be beneficial to address this concern and provide further clarification on how the theorem accounts for shared nodes during edge distribution.\n3. An analysis of the number of parameters in the model would be valuable information to include in the paper. Given the relatively small number of samples in the datasets used and the utilization of large networks, particularly with each hierarchy level having its own separate GNN comprising 8 layers, the complexity and capacity of the model should be carefully considered. Providing details on the number of parameters, as well as discussing potential implications of model size and dataset size, would enhance the understanding and interpretation of the experimental results.\n\nLIMITATIONS:\nConsidering the assumed independence among the clusters and the cross-edges connecting them, it is evident that there exist certain graph distributions which the model may struggle to learn.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a graph generative model that is analogously structured as the inverse process of graph pooling, where the model first split a single node into a metagraph. This metagraph is further partitioned by utilizing a multinomial scheme, which allows for the division of nodes and edges into intra-community and inter-community connections. The proposed model's performance is evaluated on several benchmarks using various metrics, demonstrating state-of-the-art results.\n\nSTRENGTHS:\n1. The approach of initially generating a graph's skeleton and subsequently refining its details is a novel and intuitively logical motivation for the proposed model.\n\n2. The proposed methods have demonstrated state-of-the-art performance on some widely adopted benchmarks.\n\nWEAKNESSES:\n1. Some important technical aspects in the paper may require additional clarification or more detailed elaboration. Here are the major concerns regarding specific aspects:\n\n    (1) Can the authors please provide more information on the loss function utilized in the model?\n\n    (2) How is the weight on level 0 determined during model inference?\n\n    (3) On line 188, the node embedding matrix $\\mathbf{h}_{\\hat{C}}$ is referenced without being defined. Could the authors please explain how this matrix is generated from the node and edge embeddings of prior levels?\n\n    (4) Could the authors please elaborate on how the graph neural network (GNN) is utilized throughout the entire process?\n\n2. The utilization of notations in the paper has resulted in a significant amount of confusion. There are two main issues that need to be addressed:\n\n    (1) Inconsistent notations caused by reusing the same symbols: One notable example is the letter \"t\" used at lines 187-188, which has multiple interpretations. In $\\hat{C}_{i,t}^l$, \"t\" represents the \"t-th\" step in the stick-breaking process. In $h{(t, s)}$, \"t\" denotes the node that is associated with community \"i\". Moreover, when referring to the node matrix size as \"$t \\times d_h$\", it indicates the total number of nodes in community \"i\". These varying interpretations of the same notation can lead to confusion and should be clearly distinguished or explained consistently throughout the paper.\n\n    (2) Notations used without being defined: An example is the \"r\" symbol in Figure 1 (c). Although it is assumed to represent the acronym for \"remaining (edges),\" its precise definition is not explicitly provided in the paper. To enhance clarity, it would be beneficial to define such notations explicitly or provide a glossary of symbols and their corresponding definitions.\n\n3. In the paper, the specific method for determining the number of mutually exclusive events (i.e., the edges split from the same parent node) when modeling the partition weights using a multinomial distribution is not explicitly mentioned. This aspect requires further clarification or explanation. The paper should provide details on how the number of events is determined, whether it is considered a fixed parameter based on the model's architecture or if it is treated as a latent variable to be inferred during the training process.\n\nQUESTIONS:\n1. Please address my questions in the previous section.\n\n2. At line 188, the paper defines the feature for an edge $(t, s)$ as the difference between node features. This implies that the features for any self-edge would be zero, as the difference between a node's feature and itself would result in a zero value. Regarding Eq(7), it states that the expected partition weight received by each edge is determined by concatenating the edge feature and the parent node feature, which remains constant for all edges split from the same parent node. Considering these points, if all self-edges have identical features, they are expected to receive the same partition weights. Could authors please confirm or repute the above analysis?\n\n3. As a consequence of similar partition weights across the self-edges split from the same parent node, when these self-edges are further partitioned into intra-community connections, it could lead to similar sizes and connection densities across the communities. Does it align with the characteristics of real-world graph generation that the model intends to apply to? In other words, how well is the proposed model in generating graphs whose communities has varying sizes and connection densities.\n\n4. I have some questions regarding the rationale behind introducing theorem 3.3, especially considering the prior introduction of lemma 3.2. It seems to me that theorem 3.3 primarily focuses on consolidating the two levels of split into a single function. I kindly request the authors to provide clarification regarding the significance and necessity of theorem 3.3 in the overall context of the paper.\n\nLIMITATIONS:\n1. Please refer to questions 2 & 3.\n\n2. The model is built upon the assumption that the graph contains underlying communities. While this assumption can aid in generating higher-quality graphs with evident community structures, it may come at the expense of the generation quality for graphs where the community structures are less apparent. It would be intriguing to explore how the quality of generated graphs varies with changes in graph modularity or other community metrics.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}