Micro and Macro Level Graph Modeling for Graph Variational Auto-Encoders
Authors: Kiarash Zahirnia
Oliver Schulte
Parmis Naddaf
Ke Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Graph Generative Model
Node-Level Properties
Graph-Level Properties. Graph Variational Auto-Encoder

Abstract: Generative models for graph data are an important research topic in machine learning. Graph data comprise two levels that are typically analyzed separately: node-level properties such as the existence of a link between a pair of nodes, and global aggregate graph-level statistics, such as motif counts.
This paper proposes a new multi-level framework that jointly models node-level properties and graph-level statistics, as mutually reinforcing sources of information.  We introduce a new micro-macro training objective for graph generation that combines node-level and graph-level losses.  We utilize the micro-macro objective to improve graph generation with a GraphVAE, a well-established model based on graph-level latent variables, that provides fast training and generation time for medium-sized graphs. Our experiments show that adding micro-macro modeling to the GraphVAE model improves graph quality scores up to 2 orders of magnitude on five benchmark datasets, while maintaining the GraphVAE generation speed advantage.

Micro and Macro Level Graph Modeling for Graph Variational Auto-Encoders
Kiarash Zahirnia, Oliver Schulte, Parmis Naddaf, Ke Li School of Computing Science, Simon Fraser University, Canada (kzahirni, oschulte, pnaddaf, keli) @ssu.ca
Abstract
Generative models for graph data are an important research topic in machine learning. Graph data comprise two levels that are typically analyzed separately: node-level properties such the existence of link between a pair of nodes, and global aggregate graph-level statistics, such motif counts. This paper proposes new multi-level framework that jointly models node-level properties and graph- level statistics, as mutually reinforcing sources of information. We introduce a new micro-macro training objective for graph generation that combines node-level and graph-level losses. We utilize the micro-macro objective to improve graph generation with GraphVAE, a well-established model based on graph-level latent variables, that provides fast training and generation time for medium-sized graphs. Our experiments show that adding micro-macro modeling to the Gra model improves graph quality scores up to 2 orders of magnitude on five benchmark datasets, while maintaining the Graph VAE generation speed advantage.
Introduction: multi-level graph modeling
Many datasets contain relational information about entities and their links that can be represented a graph. Deep generative learning on graphs has become popular research topic (20). with applications including molecule design 42). and recommendation 12). It is common in graph analysis to distinguish two levels of information [10] 20]: (1) local node-level properties, such as the existence of a link between two nodes or the attribute of a node [27 37 19), and (2) global graph-level statistics (graph statisties for short) that depend on the entire graph, such as node degree distribution or motif counts. Most deep Graph Generative Models (GGMs) are trained with an objective based on local properties (e-g., maximizing the probabilities of observed edges). Figure [ illustrates the difference between local and global structure. Different edges have different roles in the graph global structure. Some edges play critical role for the comnectivity/commnniy structure, while others are less important. Previous GGM training is generally based on likelihood objective that decomposes into individual edge likelihoods [47 31 I1, which does not discriminate edges with different roles. Graph global level information on the other hand can capture how different edges have different importance in the graph structure (see also Section 4 Whereas the sparsity of typical graphs causes difficulties for training objectives based on local information only [38] 27). graph statistics are typically dense.
This paper proposes a new perspective on the node/graph level dichotomy: principled probabilis- tic framework that incorporates both local and global graph properties. From the framework we derive a novel micro-macn (MM) objective function for training GGM to match both local and global properties. The term micro-macro originates from the philosophy of science and refers to scientific frameworks that treat local ("micro"") resp. global ("macro"") properties of a system equal targets [17]
"Supported by NSERC Canada Discovery Grant R61 16411341
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Original Graph
(b) Generated Graph
(c) Generated Graph
Figure 1: To illustrate the difference between local and global properties. The two right
graphs (Ib), (Ic) score the same in terms of number of edges reconstructed from the left graph (Ta), local node-level property. However the right graph (Te), is structurally more similar to (la). containing the same number of connected components, global graph-level statistic.
Advantages Micro-macro modeling increases graph realism and user control. (1) Compared to objective functions that are based on predicting local properties, matching graph statistics serves as a regularizer that increases the realism of the generated graph structures. This is sufficient to generate realistic graphs with a fast all-at-once adge-paralled model. (2) O'Bray et [35] note that different graph statistics are important for different applications. For example for large payment graph recording economic transactions, a macro economist may be mainly interested in the average price level of goods basket. For a central bank managing a payment system, the total number of transactions may be more important. O' O'Bray al. [35] therefore advise selecting GGM based on the graph statistics of interest in the target application. However, this entails searching through the space of GGMs and their hyperparameters to find a good match with target graph statistics. In our MM framework, the user only needs to specify the target graph statistics and learning will automatically select graph models that match them.
Approach We introduce joint probabilistic model over both local properties and global graph statistics. An MM objective that can be used with graph encoder-decoder models is derived as an ELBO from the joint probability model. Our experiments focus on the GraphVAE (Graph Variational Auto-Entor (etto-Encdder model where the encoder outputs a graph-level latent posterior x (graph embedding) and the decoder maps the graph embedding = soft adjacency matrix representing link probabilities (3)). A graph-level embedding supports modeling graph-level statistics. We show how the recent calibrated Gaussian variational auto-encoder [41] can be adapted to model graph statistics with divergent scales. The implementation and datasets are provided at https
//github... woom//iaraaszaaGrraa//...
the related work, section below. Proposed metrics quantify how similar a set of generated graphs is to a set of observed graphs. Our assessment focuses on evaluation metrics based Graph Neural Networks (GNN-based metrics) 44. To our knowledge, this is the most recent published evaluation method, with state-of-thhearr (SOTA) performance shown in extensive experiments. For Graph VAEs, adding micro-macro modeling improves both the realism and the diversity of the generated graphs, up to an order of magnitude on GNN-based metries. This is sufficient to reach very competitive graph quality compared to SOTA regressiee benchmark models [32 47 [1]. At the same time, one-shot graph generation with Graph VAEs is faster than sampling from auto-regressive models. Our experiments macro modeling, Graph VAEs maintain their generation speed advantage over auto-regressive approaches. While our evaluation focuses on GraphVAES, we discuss how micro-macro modeling can be applied to other GGMs to leverage graeh statistics.
Evaluation Evaluating GGMs has become a research topic of its own 44 35
on medium-sized graphs show that even with the overhead of micro-macro
Contributions. Our main contributions can be summarized as follows.
A new joint probabilistic model over both local graph properties and global graph-level statistics.
Deriving joint ELBO as new micro-macro objective function for training graph encoder- decoder models.
An adaptation of the Graph architecture 43 to learn graph embeddings using the joint micro-macro objective.

---

2
Related work
Graph Statistics as Modeling Targets. To our knowledge, this is the first work to develop a GGM with the objective of matching graph statistics. Work on graph moment matching in network statisties [36 6 has studied theoretical properties of graph moment estimators with increasing graph size.
Maximizing the likelihood of exponential random graph models [2]] such as Markov Logic Networks
is equivalent to maximizing entropy subject to matching set of graph statistics [28]
Graph Statistics for Evaluating Generative Models. The ability of generative models match an empirical graph statistic distribution has been assessed in several research papers, which supports our approach of including them in the training objective. Farnadi al. [16] compare relational models in terms of matching (expected) motff counts. You et al. 47 and Liao et al. [2] use 4 graph statistics to measure the realism of generated graphs.
The question of how to evaluate a GGM has been studied in recent papers uation metric quantifies how similar a set of generated graphs ,I.-.., Gm, is to a set of test graphs G1, G1,..., Gmg- From the graphs we can compute corresponding descriptors (e.g., statistics) -1,..., 2m, and I1 E...... Tmy-. The similarity of two descriptor sets can be quantified using Maximum Mean Distance (MMD). O'Bray al. (35] raise several difficulties with this approach. (1) Given list of graph statistics, we obtain list of MMD scores, rather than a single score that can be used to rank GGMs. (2) The statistics-basee based MMD scores are sensitive to syper-parameters different settings can lead to different model rankings even for a single graph statistic. (3) In perturbation studies where observed graphs are corrupted in a controlled manner, MMD scores do not correlate well with degree of perturbation. Thompson et al. 44 develop a recent proposal to address these issues. They utilize reference embedding network GNN E. The embedder is obtained from pretraining or from random weights and is therefore independent of any of the models under evaluation. Given my generated graphs and my test graphs, provides embeddings .., en, and ej.-... Cmg which can be compared using vector metric such as MMD. Extensive evaluation shows that the GGM scores provided by the reference embedding approach correlate well with perturbation degree and capture the realism and diversity of generated graphs. To our knowledge, the GNN-based approach is the state of the art for evaluating GGMs, so we use it as our main evaluation metric.
33
An eval-
Graph Generation Architecture. Our novel contribution is to develop new kind of objective function, not a new kind of GGM. We therefore follow an AB design where we utilize an existing architecture as is and change only the training objective. Developing GGMs is an on-going topic of research;
overviews can be found in [50 20 Two major GGM groups are all-at-once and auto-regressive
methods 20) All-at-once generation decodes a latent variable z to generate (soft) adjacency
matrix A. Auto-regressive methods 132 25 generate graph incrementally. All-at-once methods are faster at graph generation, but tend to generate less realistic graphs. They can use node-level representations or graph-level representation (embedding). We use a graph-level representation for two reasons: (1) A natural fit with modeling graph-level statistics. (2) It is known from previous work that nove-level representations, while useful for many applications, do not capture enough graph structure to generate realistic graphs. We utilize the well-known Graph architecture for computing graph embeddings
43 20 Ch.9 1ec.
For generative model of graph statistics, we adapt calibrated Gaussian variational auto-encoder AT which has been developed as a generative model for i.i.d. data, but not previously been applied to graph modeling.
3 Data model and micro-macro objective
An attributed graph is pair G = (V, E) comprising finite set of N nodes and edges where each node is assigned an d-dimensional attribute X An attributed graph can be represented by an N x N adjacency matrix A with (0, 1} entries, together with an N d node feature matrix X. Following Ma et al. 33 we view the observed adjacency matrix as sample from an underlying probabilistic adjacency matrix À with AL E [0, 1]. The sampling distribution for independent edges is given by
((A
1 A (1-
(1-Ais) --s)
(I)
i=1==1
3

---

A descriptor function é mäps an adjacency matrix A to a I-dimensional graph statistic such that ((A) R' B35. We use also matrices as graph statistics. We consider only descriptor functions that are permutation-equuiarinn [20 Ch.5]. A graph statistic represents higher-order graph property. Examples include node degree histogram or motif counts; for discussion and further examples see [20] Sec.2.1.2) and Section below.
3.1 Micro-maero objective
We assume finite list of descriptor functions ,1,...,, that define target statistics.
For fixed attribute matrix X, micro-macro (MM) loss of the form
(A = C2 (A) C (F,...., Fm)
where is the training graph and Fas 1, ..... m is random variable defined by applying the u-th descriptor function to the training graph. The hyperparameter controls the balance between capturing micro and macro aspects of the training graph. The micro loss _ decomposes into losses for each node or pair of nodes. Well-known examples include the cross-entropy loss and max-margin loss. this paper we work with negative log- oieliiill losses and variational approximation based on an encoder-decoder architecture:
(A) - -In p(A)As))(()dz
/ (F Fm) =-
I (F.)
In P
(3)
Ful
n=1
(A / (F = ((Falb/ /
using the following notation.
iix specifies latent t-dimensional graph embedding with prior distribution p(z).
A₂ is a probabilistic adjacency matrix computed a trainable deterministic decoder function of graph embedding z. The decoder parameters are denoted as vb. The edge reconstruction probability p( (Al Ag) computed as Equation
The conditional distribution of each graph statistic modeled as a Gaussian / with diagonal variance parameter o² 2.
P (F N(F.))=
The Gaussian mean is computed by applying the u-th descriptor function to the recon- structed (soft) adjacency matrix. Even with a Gaussian conditional distribution, the marginal distribution graph statistics, pby (F.), can in principle fit any distribution 226
Ful is the dimensionality of target statistic F...
Figure 2a shows a generative model diagram for Equations by its dimension is important because their scales can diverge widely. For example, transition matrices have 22 entries, while the number of triangles is a single scalar. The following proposition provides an ELBO for MM loss.
Normalizing each graph statistic
Proposition
Let Lo(A) be a micro-macro loss defined by Equations
Then
m
Lo(A) < -xng.
In d (Az), []]
(5)
F
+(1 + (m) (X,
where (2(=2A, ((=|A, X, Fi,..., Fm) is an approximate posterior distribution with pa- rameters ip.

---

The proof is in the Appendix section 8.1 The basic idea is to combine ELBOs for each individual loss term. Adding hyperparameter to control the relative importance of the KL-divergence in the 3-VAE [2 ), we obtain the following training objective for a set of observed training graphs A A
arg min 2.4.0
E (=
1an--mp(((pp) (np(())s)))
In (A.)) o
(6)
Fal
+BKLIqg (2 A [p(z))
In this objective, the graph statistic reconstruction loss, -7 - F W 0n (A)))
acts as a regularizer with respect to the edge reconstruction loss In A Az).
3.2 Implementation
We implement the micro-macro ELBO (6 utilizing graph neural networks as follows.
The prior p(z) is standard normal distribution.
We limit the encoder input to the observed training graph, so g(z F,,,
Fm)
(, X). Therefore we can use any standard graph encoder with parameters yp. The en- coder deterministically maps an input graph (A, X to a posterior distribution (=/=A, X).
The decoder deterninistically maps latent representation z to a probabilistic graph A with parameters vb. Any standard graph decoder can be used.
For each graph-level statistic, the diagonal co- co-variance parameter is computed using the op- timal o- --V method from the calibrated Gaussian framework 41]. This uses the maximum likelihood estimate given the estimated means with respect to minibatch of size B:
0 6
(d)(())()))) F.)).
MSE a 11)
a p2)
B
[eguree2]ppoce the architecture. The advantages of the calibrated Gaussian model for generating graph statistics are as follows. (1) In experiments on non-relationa data, the calibrated Gaussian model achieves state-of-the-art performance. (2) Learning variance parameter eliminates hyperparameters compared to user-assigned weights for each graph statistic. (3) The variance values can interpreted quantifying the empirical uncertainty of graph feature. For example, if all training graphs exhibit a similar number of triangles, the corresponding variance parameter will be small. Table 6 in the Appendix illustrates this phenomenon in benchmark datasets.
Lg
piz)
z
A
A
A
Fu
Fu
Lg
(e)
(8)
Figure 2: (a) Micro-macro generative model. Observed variables in gray. (b) Graph VAE-MM archi- tecture.
4 Graph statistics
The micro-macro ELBBO 6 requires user-specified input graph deseriptors for computing target graph statistics. In experiments, utilize several types of default statistics for regularizing
5

---

graph embeddings that have the following advantages. (1) Known from prior research to be generally important for graph modeling across different domains. (2) Easy to interpret. (3) Differentiable with respect to the entries in reconstructed soft adjacency matrix. In an application, the default statistics can be combined with other statistics of interest.
Graph descriptors used in network analysis and graph kernels comprise two main groups [00 (I) Summaries of local node-level properties. We utilize a descriptor derived from the degree histogram, and the number of triangles in g2aph.(2) Higher-order proximity relations between nodes. We utilize s-step transition probability matrices, for s I.... with respect to the entries in a reconstructed soft adjacency matrix A, which we view as representing a weighted undirected graph.
p These descriptors are differentiable
Degree Histogram. The degree of node oy is given by d(vz) = Σ3 Auj. We adapt the permutation- invariant differentiable histogram layer (DHL) 45).
The DHL is based on soft assignment of points to bins given bin centers and widths. Binning the soft degrees forms a soft degree histogram of graph A comparable to the hard degree histogram. In detail, the bin centers are the possible (hard) node degrees b O.....N All widths are uniformly set to 0.1 (based on experimentation). Then we have
NV
(b) = Σ a(vi,b))
a(v;, b) = max (0, 1 - 0.1 Id(m)) - bl}-
Thus the membership a(u,, b) of node in a bin b ranges from 0 to and decreases with the difference between the node's 's expected degree and the bin center. The DHL assigns to each bin the sum of nodes membership in the bin. In a naive implementation, the computational cost of finding the d x (b) vector is ( our experiments, we use N parallel processors for near constant time obtain the soft degree histogram.
S-Step Transition Probability Kernel. (*A) is the N x N 8-step transition probability matrix derived from adjacency matrix A such that (A)]a,, is the probability of transition from node i to node in a random walk of steps started from i. The transition matrix (*A can be computed where ((A) is diagonal matrix with D ((A); d(u;).
as PS (AA (D(A)-1A)
The GraRep system learns node representations that reconstruct the s-step transition probabilities, using random walks and matrix factorization (9. The transition probability matrix is usually dense and encodes the connectivity information of the graph. An important difference to the adjacency matrix is that whereas adding or removing does result in substantive change in many transition matrix elements. The descriptor Pa (AA) thus measures the importance of an edge in the graph structure (ef. Figure]] TT. In our experiments, we compute the transition probability matrix exactly with runtime cost of (
edge changes only one adjacency, it can and often
Triangle Count. The number of triangles in a simple graph computed by (ri E, (49),;
with runtime cost of O(N*). The number of triangles is a fundamental graph statistic in network science (5] with many applications in graph mining [. For example it has been used to detect spamming activity and assess content quality in social networks [4) uncover thematic structure in the world-wind world web [1] and for query planning in databases 3 It is also used extensively in
exponential random graph models (1
5 Empirical evaluation
We describe our baseline methods and benchmark datasets, then report comparison results.
5.1 Comparison methods
We examine the effect of micro-macro modeling Graph architecture. Our AB methodology to keep the architecture the same and train the model using the joint MM ELBO (6) that combines both local and global graph properties. We also compare the MM GraphVAE with popular GGMs. As

---

the architectures are not new, we describe them briefly. Our experiments used the public repositories and recommended hyperparameters by original creators; the Appendix contains further details.
GraphVAE. VAE. A popular model that transforms a graph-level latent variable to generate a soft adjacency Ch.9.1.2]. The encoder utilizes Multi-layer Graph Convolutional Network (GCN) followed by a graph level readout and a Fully Connected Layer (FCL) that outputs (), The
matrix 4 20
decoder utilizes network with FCLs that outputs entries of a probabilistic adjacency matrix A. To
evaluate the reconstruction probability, we use BFS ordering of nodes [3 Ch.9.1.2].
VAEphVAE-M.. Graph VAE trained with the micro-macro objective 6 Our new method.
GraphRNN. Auto-regressive method that generates the adjacency matrix incrementally. Each step generates one entry in the GraphRNN design, or one column in the GraphRNN-S design [47]
GRAN. Auto-regressive method that generates block of nodes and associated edges a step [32]] BiGG. Auto-regressive method that leverages graphs sparsity to avoid generating the full adjacency matrix M. To our knowledge BiGG achieves SOTA graph quality.
5.2 Benchmark datasets
Our design closely follows previous experiments on generating realistic graph structures 47 32 We utilize 3 synthetic, and 2 real-world datasets for the main results. The synthetic Grid and Lobster are from previous studies [47 32). Triangle Grid is introduced in this paper. Protein and beblbbb are roal-world datasets from biology with information about proteins and molecules respectively. The Appendix, section 8.2 contains further details, well results for 3 more real-world datasets.
Train/Test Split. Following previous experiments 47
we randomly split the graphs sets into train (70%). validation (10%) and test (20%) sets. We use the same train and test graph sets for all models. To evaluate trained model, we generate T new graphs to compare with the T graphs the test set (cf. Section 2). Appendix Figure Sjillustrates the evaluation process.
Network modeling the micro-level might be used for surveillance of individuals and communities. Therefore ethical network data collection must take into account issues of consent, privacy, and bias. None of the datasets used in this research study contain personally identifiable information, or offensive/harmful content about either individuals or communities. The social impact of our work we expect to be on balance more positive than negative, because our macro-level model enhances the understanding of global network structure, not the targeting of individuals.
5.3 Evaluation
We empirically verify the effectiveness of micro-macro modeling through different evaluation metries: Qualitative and quantitative evaluation of graph quality, generation time, and training time.
we compare the generated graphs by visual inspection. Figure 3 provides a visual comparison of randomly selected test graphs and generated graphs. GrMMhVAE-MM and BiGG achieve the best visual match. The Appendix, section 8.5 provides more examples.
Qualitative Evaluation. Following 47 32
Quantitative Evaluation. We follow the descriptor based approach established in previous works (cf. Section 2). As recommended by O' O'Bray et al. [35,, we include scores computed from a 50/50 split of the data set as ideal score, i.e., lower bound on the GGM scores.
(1) The GNN-based evaluation metrics 44 MMD RBF and Fl PR compare generated and test graph embeddings computed by reference GNN with randomly initialized weights. The reference embeddings are independent any target graph statistics used in Graph VAE-M.. Fl PR stands for "Improved Precision and Recall". Precision is the percentage of generated graph embeddings that fall within the manifold test graph embeddings, while recall is the percentage of test graph embeddings that fall within the manifold of generated graph embeddings. This metric mainly captures the diversity of generated graphs 35 MMD RBF compares generated graph embeddings and test graph embeddings using an RBF kernel. This metric captures the realism of the generated graphs.
(2) btad evaluation compares generated and test graphs using evaluation statistics, namely node degree, clustering coefficient, orbit counts, and the spectra of the graphs 32 47 We add graph diameter, statistic commonly studied in network science (29] related to graph connectivity. The target statisties used to train GraphVAE-M model are distinct from the evaluation statistics
7

---

Grid Lobster Lobster
Gid
CAMM
GragRRNN
GiaphRNNNS
Figure 3:
Visualization of generated graphs. The left-most column 3a shows randomly selected graphs from the test set for each dataset, the other columns show graphs generated by each model from the prior. The generated graphs shown are the two visually most similar samples in the generated set. Graph VAE MMM achieves the best visual match, much better than GraphVAE.
used to measure graph quality (however, node degree and Degree histogram are closely related). For each statistic, MMD computed using total variation (TV) distance 32
Impact on Graph VAE. Table shows very large improvement from micro- modeling for both MMD RBF and FI PR. The improvement in F1 PR ranges from 6%-40% achieving a near perfect score. For MMD RBF the reduction ranges from 0.06 to 0.26 which is significant given the low ideal score. Table shows the results for evaluation graph statistics. Micro-macro modeling reduces MMD by 1-2 orders of magnitude on almost all datasets. In sum, MM modeling provides large improvement in the realism and diversity of graphs generated by a Graph VAE architecture.
Lesion studies in the Appendix investigate the effects of each target statistic in isolation (Table 5). No single statistic has the power of all three combined. We also observe that different target statistics have different importance for different datasets; see Table 6
Table 1: GNN-based evaluation of micro-macro modeling for Graph VAEs. For each dataset we report the MMD RBF and FI PR (see text) between the test set graphs (left-most column in Figure 3 and the generated graphs (see other columns in Figure 3. Values reported are the mean + std. For MMD RBF smaller values better, for Fl PR larger values are better.
Method
MMDD RESP
Triangle Grid
PL
MMD RIST
Loboder
F1
MMD RLF
Grid
PL
N34U kile
<glg-mmlllhhp
Fi
MMD RBR
Pretein PL
50550 splis
=
0.00
0.00
98.58 0.00
16.00
0.00
0:00
96.07
GUjUVAE
160,00
GraphVAE-MM vs. Benchmark GGMs. For benchmarking include GGMs that known to generate realistic graphs. Tables shows the GNN-based quality scores. Other than the most recent BiGG method, Graph achieves much better score. MMD RBF scores, and 3 of F1 PR scores are also better in VAF-MM compared to BiGG. Triangle Grid shows the biggest improvement, which illustrates the usefulness matching triangle counts for this dataset.
Appendix Table 7 shows the benchmark results of statistics-based evaluation; we summarize them here. On synthetic graphs, the Graph VAE-MM scores are superior to competitive with the BiGG and GRAN scores. On the real-world graphs, the Graph BiGG and GRAN scores, and superior the other benchmarks. Given the already strong performance of the auto-regressive models, we conclude that Graph VAE-MM generates high-quality graphs.
scores are competitive with the
Generation Time. The code for all models is run the same system, detailed in Appendix section 8.11 Figure compares Graph VAFs to the fastest auto-regressive methods. The auto-regressive methods require substantially more generation time than GraphVAEs. While MM modeling slows

---

micro-macro Table 2: Statistics-based evaluation of miero- maero modeling for GraphVAE: For a named evaluation graph statistic, each column reports the MMD between the test graphs and the generated graphs.
(a) Synthetic Graphs
Method
Triangle Grid
Lobster
Grid
Deg.
Oxbit
Spect
Diam.
Deg.
Uxbit
Spect
Diam.
Deg.
Oxbit
Spect
Diam.
50/50 split
3e
0,0022
8e
0.004
0.014
0.002
0.002
0.005
0.032
le
2e
0.004
0.014
GraphVAE
0.082
0.442
0.421
0.020
0.152
0.081
0.739
0.372
0.056
0.129
0.062
0.055
0.615
0.018
0.143
GRLMVVAE-MM
0
0,093
0,001
0.013
0.133
2-
0.008
0.017
0.187
5e
0.001
0,014
0.065
(b) Real Graphs
Method
Protein
egbg-molbbbb CHtit.
Deg:
Clus
Spoct.
Diast
Deg.
Cluss
Speet.
50/50 split
de
0.004
De
4e-4
0.003
2:
2x
Be
Se
0.002
VAEE
0.022
0.108
0.577
0.016
0,080
0.028
0.442
0.047
0.015
0.055
GrophAF-MMM
0.006
0.059
0.152
0.007
0.091
0.001
0.005
8c**
0.005
0.018
Table 3: GNN-based comparison with benchmark GGMs. See table T caption. The best result is in bold and the second best is underlined.
Method
MM RRI
Trianale Grid
MMD Lobster
MMD RIE
Grid
oggg motbhbp lita
KM
Protein
uplin
GniphAB-MM MM
$3.58
5.50
0.30 0,00
108,00 0,000
0.13 9.01
97,00
GraghRtNN $ 147
0.72 0:117
13 68 19.44
0.98 = 0.13
72 7.55
0.79 0.08
71.18 2.36
0.45 = 0.12
S1.41 2: 0.71
0.28 di 0:25
72.36 57.3
GiggRRN
0.04 2:0.11
25 80
11.75
0.67 0.04
01.97
0.00
0,90 0
13.22 0.06
1:45 = 0.19
98.94
0.56
0.32 +: 0.14
M5.94 0.06
GRAN
0.85 2:09
23.71
=
0.24
0.84
50.53
4: 12.32
0.00 0.00
78. T1 0.00
0.39
= 0.07
14.00 2.00
0.0
+: 100
UX. 0.700.7
BKK IXT
0.43 4:013
08.08 0.14
0:12 0.00
09.74
9 0.70
0.33 0.00
00.03 63 0.00
0.04 :2 0.00
16.116 8: 0.33
0.15 1000
9K11 9,63
down training, for the Graph the training time is still less than for the auto-regressive methods. The Appendix table provides time measurements for all methods. The Appendix Table 12 presents worst-case complexity bounds of the comparison methods.
provides
5.4 Further experiments on real-world graphs
4 datasets evaluate We conducted further experiments on the MUTAG, PTC 46), and QM9
micro-mact modeling on more real world graphs. The detailed results are in the Appendix section 8.10 we summarize them here. For all datasets, GraphVAE-MN. offers much faster generation than the auto-regressive baselines. On MUTAG and PTC, the improvements from micro-macro modeling are even better than those we reported on the Protein and ogbg-molbbbp datasets. Graph VAE- MM achieves substantive improvement in generation quality over all baselines, except for BiGG on PTC,, which competitive. It was infeasible to train the auto-regressive methods on the QM9 datasets (except for BiGG), so report only the comparison of GraphVAE vs. Graph Micro-macro modeling brings small improvement in graph quality on QM9, which is good given the strong score of Graph VAE on QM9.
Limitations and discussion
We discuss the limitations of our Graph model and options for combining micro-macro modeling with other GGMs.
Strengths and Weaknesses. GraphVAE-MM inherits the strengths of Graph 20
Ch.9.1.2)
expressive power through graph embeddings, and fast generation due to all-at-once parallel edge generation. GraphVAE- a maximum number of nodes before generation. (Smaller graphs can be generated using a mask.) 2) The decoder is an FCL that ouputs N' x N numbers; we need to (implicitly) assume a node ordering to evaluate edge reconstruction probabilities based on the FCL output. The dependence on node ordering common to both Graph VAES and ruto-regressive GGMs, and efficient heuristics have been designed whose effectiveness has been confirmed in experiments, including those reported in this paper. We note that MM modeling makes GGM training less dependent on a node ordering because it uses permutation-invariant invariant graph statistics.
also inherits the known limitations of Graph
We need know

---

Time
40
GenprusE
Generation PerBatch
Tein Time (Scconni
2o0
123
BBD
GrophiAE MM GRAN
GrophiVE
GRAN
BIGG
BR06
Gcterstinn The
60
20
0 0ALLL
GII
Lobrrar
Cnd
Pretur
LAN ALL Rodo
Thangie orid
Lobster
Grdd
Protein
og0g molboop
Dataset
Figure 4: Comparison of benchmark GGMs with Graph VAEs time (right). For visualizing the small generation time of Graph VAE and GraphVAE-MM, we round them up to 0.01, the first and second bar of each dataset respectively.
GAap on generation time (left) and training
Training Time Overhead. Evaluating the Graph VAE-MM ELBO during training incurs computational overhead compared to the Graph VAE ELBO due to computing graph statistics. We note that evaluat- ing the edge reconstruction probability is already expensive and can dominate training time. Table 12 in the Appendix presents worst-case complexity bounds for the training time of our comparison methods, which indicate that scaling Graph VAEs very large graphs challenge. Despite the theoretical worst-case cost of evaluating graph statistics, Figure/Aright shows that the overhead was small in experiments with medium-size graphs, due to parallelization. Also, approximating graph statistics is a promising avenue for significantly reducing the training time even for large graphs significantly 24 15 31 40)
Micro-Macro Modeling for Other GGM Architectures. In principle the MM objective can be applied with auto-regressive architectures well: after complete graph has been generated sequentially, the global loss can be backpropagated through the individual edge generation decisions.
In graph GAN 7 g the generator maps graph latent z to an adjacency matrix, random walk and the discriminator classifies them as real or synthetic. GAN models are not considered as SOTA GGMs [TT] Both random walk and adjacency matrices represent local micro-level information only (9] A straightforward way combine MM modeling with GANs to augment the input to the discriminator with graph statistics computed for both real and generated graphs.
7 Conclusion and future work
Our main idea is to model graph data jointly at two levels: a micro level based on local information (e.g. the existence of a link between two nodes) and a macro level based on aggregate graph statistics. We described principled joint probabilistic model for both micro and macro levels, and derived an ELBO training objective for graph encoder-decoder models. Compared to previous micro level training objectives, the macro statistics regularize graph embeddings to match global graph statistics. To evaluate our model, we described a set of strong default graph statistics (node degree, number of triangles, transition probabilities). We applied the training objective to the Graph VAE architecture, a widely used graph generative model based on latent graph representation. Micro-macro (MM) modeling greatly improved the quality of graphs generated by GraphVAE, match exceed that of benchmark models. MM modeling maintains the speed advantages of edge-parallel all-at-once graph generation. With efficient computation of graph statistics, it provided fast training time as well compared to auto-regressive methods.
Micro-macro modeling opens a number of fruitful avenues for future work. i) Investigating which graph statistics are important for generating which types of graphs. This connects with the rich area of graph kernels 34] that are often based on graph statistics. ii) Investigating which graph statistics are important for particular domains. iii) Developing a micro-macro model for other graph generative architectures, such as auto-regressive and GANs. iv) Evaluating the impact of micro-mact modeling on other graph-level tasks, such as graph classification.
In sum, modeling both global graph statistics and local information enhances the power of graph generation. Compared to using local information only, graph statistics can be used to regularize graph representations to efficiently generate realistic graphs.
10

---

References
[1] Shaikh Arifuzzaman, Maleq Khan, and Madhav Marathe. Patric: A parallel algorithm for counting triangles in massive networks. In ACM international conference on Information & Knowledge Management, pages 529-538, 2013.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064400 2016.
[3] Ziv Bar- Yossef,, Ravi Kumar, and D Sivakumar. Reductions in streaming algorithms, with an application to counting triangles in graphs. In SODA, volume pages 623-632, 2002.
[4] Luca Becchetti, Paolo Boldi, Carlos Castillo, and Aristides Gionis. Efficient semi-streaming algorithms for local triangle counting in massive graphs. In ACM SIGKDD international conference on Knowledge discovery and data mining, pages 16-24, 2008
[5] Suman K Bera and C Seshadhri. How to count triangles, without seeing the whole graph. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 306-316, 2020.
[6] Peter J Bickel, Aiyou Chen, and Elizaveta Levina. The method of moments and degree distributions for network models. The Annals of Statistics, 39(5):2280-25) 23010 2011.
[7] Aleksandar Bojchevski, Oleksandr Shchur, Daniel Zügner, and Stephan Günnemann. Netgan: Generating graphs via random walks. In International Conference on Machine Learning, pages 610-619. PMLR, 2018.
[8] Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. CoRR, abs/1805. 108.0.1..9.3.. 11933, 2018. URL http:://rxiv org/ abaaabas///0 110/0.805.000 11973.
[9] Shaosheng Cao, Wei Lu, and Qiongkai Xu. GraRep: structural information. In ACM international on conference on information and knowledge management, pages 891-900, 2015.
Learning graph representations with global
[10] Deepayan Chakrabarti and Christos Faloutsos. Graph mining: laws, tools, and case studies. Synthesis Lectures on Data Mining and Knowledge Discovery, 7(1):1-207, 2012.
[11] Hanjun Dai, Azade Nazi, Yujia Li, Bo Dai, and Dale Schuurmans. Scalable deep generative modeling for sparse graphs. In International Conference on Machine Learning, pages 2302- 2312. PMLR, 2020.
[12] Tien Huu Do, Due Minh Nguyen, Evaggelia Tsiligianni, Angel Lopez Aguirre, Valerio Panzica La Manna, Frank Pasveer, Wilfried Philips, and Nikos Deligiannis. Matrix completion with variational graph autoencoders: Application in hyperlocal air quality inference. JEEE International Conference Acoustics, Speech and Signal Processing, pages 7535-7539. IEEE, 2019.
[13] Paul D Dobson and Andrew J Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology, 330(4):771-783 2003.
[14] Jean-Pierre Eckmann and Elisha Moses. Curvature of co-links uncovers hidden thematic layers in the world wide web. The national academy of sciences, 99(9):5825-5829, 2002.
[15] Talya Eden, Amit Levi, Dana Ron, and C Seshadhri. Approximately counting triangles in sublinear time. SIAM Journal on Computing, 46(5): 160:-1646666 2017.
[16] Golnoosh Farnadi, Stephen H Bach. Marie-Francine Moens, Lise Getoor, and Martine De Cock. Soft quantification in statistical relational learning. Machine Learning, 106(12):171-1191, 2017.
[17] Roman Frigg and Stephan Hartmann. Models in Science. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Spring 2020 edition, 2020.

---

[18] Nikhil Goyal, Harsh Vardhan Jain, and Sayan Ranu. Graphgen: a scalable approach to domain- agnostic labeled graph generation. Proceedings of The Web Conference 2020. pages 1253- 1263, 2020.
[19] Aditya Grover, Aaron Zweig, and Stefano Ermon. Graphite: Iterative generative modeling of graphs. In International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2434-2444, 2019.
[20] William L Hamilton. Graph representation learning. Synthesis Lectures on Artifical Intelligence
and Machine Learning, 14(3):1-159, 2020.
[21] Jenine K Harris. An introduction to exponential random graph modeling, volume 173. Sage Publications, 2013.
[22] Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In International Conference on Learning Representations. OpenReview.net, 2017.
[23] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118-221333 2020.
[24] Tony Jebara and Risi Kondor. Bhattacharyya expected likelihood kernels. In Bernhard Schölkopf and Manfred K. Warmuth, editors, Computational Learning Theory and Kernel Machines, volume 2777 of Lecture Notes in Computer Science. pages 57-71. Springer, 2003.
[25] Mahmoud Khademi and Oliver Schulte. Deep generative probabilistic graph neural networks for scene graph generation. In Proceedings AAAI, volume 34, pages 11237-11245, 2020.
[26] Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014.
[27] Thomas Kipf and M. Welling. Variational graph auto-encoders. ArXiv, abs/161 1107000 2016.
[28] Ondiej Kuželka, Yuyi Wang, Jesse Davis, and Steven Schockaert. Relational marginal problems: Theory and estimation. In AAAI Conference on Artificial Intelligence, volume 32, 2018.
[29] Jure Leskovec, Deepayan Chakrabarti, Jon Kleinberg, Christos Faloutsos, and Zoubin Ghahra- mani. Kronecker graphs: an approach to modeling networks. Journal of Machine Learning Research, 1 11(2):955-1422 2010.
[30] Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. In International Conference on Learning Representations, 2016.
[31] Renjie Liao, Yujia Li, Yang Song, Shenlong Wang, Will Hamilton, David K Duvenaud, Raquel Urtasun, and Richard Zemel. Efficient graph generation with graph recurrent attention networks. Advances in Neural Information Processing Systems, 32:4257-420 2019.
[32] Renjie Liao, Yujia Li. Yang Song, Shenlong Wang. William L. Hamilton, David Duvenaud, Raquel Urtasun, and Richard S. Zemel. Efficient graph generation with graph recurrent attention networks.
Advances in Neural Information Processing Systems, pages 4257-4267 2019.
[33] Tengfei Ma, Jie Chen, and Cao Xiao. Constrained generation of semantically valid graphs via regularizing variational autoencoders. In Advances in Neural Information Processing Systems, pages 7113-7124, 2018.
[34] Giannis Nikolentzos, Giannis Siglidis, and Michalis Vazirgiannis. Graph kernels: A survey. J. Artif. Intell. Res., 11:943--127,, 2021.
[35] Leslie O' O'Bray, Max Horn, Bastian Rieck, and Karsten Borgwardt. Evaluation metrics for graph generative models: Problems, pitfalls, and practical solutions. International Conference on Learning Representations, 2022.
12

---

[36] Peter Orbanz and Daniel M Roy. Bayesian models of graphs, arrays and other exchangeable random structures. IEEE transactions on pattern analysis and machine intelligence. 37(2): 437-461, 2014.
[37] Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Adversarially regularized graph autoencoder for graph embedding. In International Joint Conference Artificial Intelligence, 2018.
[38] Heng Qiao, Daniela Oliveira, and Dapeng Wu. A novel criterion of reconstruction--ont based anomaly detection for sparse- pparse-binary data. In IEEE Global Communications Conference, pages 1-6. IEEE, 2020.
[39] Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
[40] Bruno Ribeiro and Don Towsley. On the estimation accuracy of degree distributions from graph sampling. In IEEE Conference on Decision and Control, pages 5240-5247. IEEE, 2012.
[41] Oleh Rybkin, Kostas Daniilidis, and Sergey Levine. Simple and effective training with calibrated decoders. In International Conference on Machine Learning, pages 9179-9189. PMLR, 2021.
[42] Bidisha Samanta, Abir De, Gourhari Jana, Vicenç Gómez, Pratim Kumar Chattaraj, Niloy Ganguly, and Manuel Gomez-Rodriguee NEVAE: deep generative model for molecular graphs. J. Mach. Learn. Res., 21:114:1-114:33, 2020.
[43] Martin Simonovsky and Nikos Komodakis. Graph VAE: Towards generation of small graphs using variational autoencoders. In Artificial Neural Networks and Machine Learning, volume 11139 of Lecture Notes Computer Science, pages 412 42-4 2018.
[44] Rylee Thompson, Boris Knyazev, Elahe Ghalebi, Jungtaek Kim, and Graham W Taylor. On evaluation metrics for graph generative models. In International Conference on Learning Representations, 2022.
[45] Zhe Wang, Hongsheng Li, Wanli Ouyang, and Xiaogang Wang. Learnable histogram: Statistical context features for deep neural networks. In European Conference Computer Vision, pages 246-262. Springer, 2016.
[46] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2018.
[47] Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. GraphRNN: Generating realistic graphs with deep auto-regressive models. In International Conference on Machine Learning, pages
PMLR, 2018.
[48] Kiarash Zahimiaa, Oliver Schulte, Ke Li, Ankita Sakhuja, and Parmis Nadaf. Deep learning of latent edge types from relational data. In 35th Canadian Conference on Artificial Intelligence, 2022.
[49] Zhen Zhang, Mianzhi Wang, Yijian Xiang, Yan Huang, and Arye Nehorai. Retgk: Graph kernels based on return probabilities of random walks. In Advances in Neural Information Processing Systems, volume 31, pages 3968-3978, 2018.
[50] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section for details.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section (5. and 83 for details.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section for details.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section and Appendix for details.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Code, datasets and library requirements can be found at https://gi http//gttttuut..t/// com/
kiarashza/ Brarshzz//GrapVAAE-MM
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sections 5.2 and Appendix for all experiment settings. We also added all details, including models, generated samples, training logs and library requirements in our public repository.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] The focus of this paper is on the GNN-based metries, tables [ [3] containing error bar information. For all other compared metrics, the proposed approach typically results in 1-2 orders of magnitude of improvement. This evidence seems to be sufficient without error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]] See Section 8.11 for machine details.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Reported datasets and codes are open-source and publicly available. We cited them clearly in Tables and Context
(b) Did you mention the license of the assets? [Yes] All codes and datasets used in this study are publicly available and clearly mentioned in the Section 5.22 and [..] Our implementation is publicly released under MIT license mentioned in repository.
(c) Did you include any new assets either in the supplemental material as URL? Yes] We included our model implementation and it is publicly available at hhtpps
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] As mentioned in Section 5.1 we used the original papers' public repository: hence no consent was needed to curate this study. Also all used datasets are open source and publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] As mentioned in Section [.2] None of the datasets used for this research study contain any personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] This point is not applicable for this research study.
14

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if upplicable? [N/A] This point is not applicable for this research study.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] No such participants were used for this research study.
15


Reviews:
[{"review_id": "qdKNOG0RoIy", "reviewer": "Reviewer_Uc7B", "summary": "This paper jointly models micro and macro level graph information for graph generation. A principled joint probabilistic model for both levels is proposed and an ELBO training objective is derived for graph encoder-decoder models. Extensive experiments and visualization results validate the efficacy of adding micro-macro modelling to GraphVAE models for graph generation.", "questions": "The proposed MM objective function is applied on GraphVAE for an AB design and its effectiveness is showed for graph generation. Do you think if the benefits of micro-macro modeling would generalise to other models or other graph tasks? Some related discussions in this regard are necessary.", "limitations": "The authors have adequately discussed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is well motivated and the idea of utilizing node-level properties and graph-level statistics to constrain graph generation seems reasonable.\n2. The design of micro-macro (MM) loss is clear and theoretically solid.\n3. The authors have done a thorough analysis of the proposed model and validated its effectiveness through qualitative and quantitative evaluation. The main claims are supported by the experimental results.\n\nWeaknesses:\nMy main concern is that the proposed objective function is only applied on GraphVAE following an AB design. Although the experimental results are satisfactory on graph generation, it remains unclear whether the benefits of micro-macro modeling would generalise to other models. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aj7J4lEPx-N", "reviewer": "Reviewer_dtRG", "summary": "The authors of this paper newly presented a function that can reflect graph statistics in the graph generative model. They have shown various experiments and visualizations proving graph statistics are well-reflected. In addition, designing an objective function to reflect different graph statistics simply is a significant contribution.", "questions": "1. As a result of examining the paper or code, it seems that node feature or edge feature is not generated. In the case of using the Real-world dataset, it is considered important to generate not only the structure of the graph but also the node/edge feature well. Can you suggest an experiment related to this part?  \n\n2. As described in the paper, graph statistics are reflected using the calibrated Gaussian framework. There are two questions about this.  \n        a) Are graph statistics used for performance evaluation or reflected in objective function actually following Gaussian distribution? For example, if you think about 'degrees', it seems that the skew distribution is easy to appear due to the nature of the graph dataset. If so, it seems necessary to verify whether 'degrees' can be assumed to follow Gaussian distribution.  \n        b) Can graph statistics be reflected in other ways without utilizing the calibrated Gaussian framework? Is there an experiment that can support the evidence using the Calibrated Gaussian framework?        \n\n3. Figure 1 shows 'connected components' as an example to be improved in this paper. It would be nice to show that the proposed model has improved this part well.\n", "limitations": "The positive social impacts presented in this paper include molecular presentation and medical discovery. However, since the model proposed by real-world dataset shows weak performance, it is seen as an important limitation.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: (Yes) \nThe proposed method seems to be original in that the authors proposed a new but simple VAE-based objective function to reflect graph statistics.\n\nQuality: (Neutral) \nSince the purpose of this study is to generate graphs that reflect graph statistics, theoretical support and experiments for the purpose are well shown. However, the performance on real-world datasets such as the molecule is marginal. In particular, when only one graph statistic is used, the performance degradation is greater than that of GraphVAE, which needs clarification. Since it shows good performance only when all three statistics presented in the paper are written, it is necessary to explain why these three combinations were selected and what synergy they show.\n\nClarity: (Yes) \nThere was no difficulty in understanding what the paper was trying to say, and it shows sufficient proof of the formula. I think it would be easier to understand if the architecture overview was attached. I suggest adding a picture of the structure for the reader's clear understanding.\n\nSignificance: (Neutral) \nThis model seems to have particular strengths in experiments using Synthetic datasets. In addition, it seems to be a good contribution that it showed a higher performance improvement compared to GraphVAE. However, as discussed in the paper, performance in real-world datasets seems to be more important to contribute in practical areas such as molecule and medical discovery. However, the experimental results presented in the paper do not support this.  Additional experiments will be needed to show that graphs are well generated using the QM9 dataset shown in GraphVAE.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "AKFpBkPsL4q", "reviewer": "Reviewer_KhAw", "summary": "The contributions of this paper was to model graph data jointly at two levels: a micro level based on local\ninformation and a macro level based on aggregate graph statistics.", "questions": "1. How the proposed model is different from the GAN-based model? ", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Positives:\n\n1. The idea of this work is interesting and novel, which trys to use probabilistic model to explore the local and global graph statistics.\n\n2. The performance of this work is very good, comparing to the existing GraphVAE. And the code is available.\n\nNegathive:\n1. The scalability of this work may be a challenge, the compexity of the descriptors is either O(N^2) or  O(N^3). Also the algorithm reuqires pre-define a graph descriptors to compute the graph statistics.\n\n2. The algorithm part is straightforwad. Basically, it designs a MM loss into one unified fraemwork. It seems that many GAN-based models can achieve the similar function. Any discussion?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "3g2iLFWAbLb", "reviewer": "Reviewer_BpMN", "summary": "This paper studies the problem of graph generation, and proposes a new model using both micro and macro level supervision information in GraphVAE architecture. Fitting adjacency matrix is the micro supervision, and three kinds of graph statistics, i.e., degree histogram, number of triangles, and higher-order proximity relations, are adopted as macro supervision. The object consists of ELBOs modeling micro-macro loss and a KL-divergence between the prior and the approximate posterior of hidden representation.  The proposed model is validated on 3 synthetic, and 2 real-world graph datasets. The experimental results the proposed model generates graphs with a lower discrepancy between generated and test graph embeddings than graphs generated by competitors in terms of MMD RBF and F1 PR.", "questions": "Q1.  How the graph generation task benefits from fitting graph statistics. And what kind of graph statistics should be chosen as targets？\n\nQ2. How to form descriptor functions with respect to vector label histogram and triangle count?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong points:\n\nS1. The macro objective of fitting graph statistics in graph generation is novel to me. \n\nS2. The paper proposes a general micro-macro ELBO as the objective, and then implements the ELBO by graph neural networks.\n\nS3. The experimental results show the proposed model outperforms the competitors.\n\nWeak points:\n\nW1. It is not clear how the graph generation task benefits from fitting graph statistics. In other words, what is the limitation to only fitting adjacency matrix in graph generation? From this line, I have a concern about what kind of graph statistics should be chosen as targets？\nThis paper selects three graph statistics, but does not present an explanation for this selection. \n\nW2. The efficiency. Both calculating and fitting graph statistics bring new computing costs, e.g., the complexity is $O(n^3)$ to compute the transition probability matrix.\n\nW3. It is not clear how to form descriptor functions with respect to vector label histogram and triangle count. And how to guarantee the descriptor functions are differentiable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept