SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This section compares GenStat with parametric and Deep GGMs. Our design closely follows previous studies on generating realistic graphs [6, 88, 89]. We report qualitative and quantitative evaluations of the generated graphs' quality. 1) We compare the performance of GenStat with popular statistics-based GGMs. 2) We compare GenStat with popular deep GGMs. These models require access to all node interactions, so this is not an apple-to-apples comparison. It measures how much graph generation quality is lost when generation is based on aggregated local statistics, rather than node interactions. 3) We compare GGMs trained under different Edge LDP budgets. 4) We compare GGMs in terms of their benchmark effectiveness for GNNs on link prediction as a downstream task. 5) We also compare deep GGMs and GenStat in terms of generation and training time.
Comparison methods. We compare GenStat to statistics-based and deep adjacency-based GGMs. Statistics-based baselines. Statistics-based GGMs that have been used with LDP guarantees [78] include the Chung-Lu Model [5] and the Block Two-Level Erdos-Renyi Model (BTER) [67]. We also evaluate the Stochastic Block Model (SBM) [1] and the Erdos-Renyi model [18], which have been employed in previous comparisons with deep GGMs [88]. See Appendix Section 7.4 for further details.
Deep adjacency-based baselines include GraphVAE-MM [89], BiGG [6], GRAN [43] and GraphRNN [88]. To our knowledge, these are the SOTA models for generating realistic graphs.
Datasets. Following previous studies, we use real and synthetic datasets [6, 88, 89]. As the datasets are not new, we describe them briefly. We evaluate our model on 3 synthetic datasets: the Lobster trees (Lobster), Grid [88] and Triangle-Grid [89] datasets, all of which consist of graphs with regular structures. We also evaluate our models on 5 real datasets: ogbg-molbbbpl (ogbg-mol) [27], Protein [12], IMDb [84], PTC [73] and MUTAG [8]. We randomly partitioned the datasets into train (70%), validation (10%), and test (20%) sets [6, 88, 89]. See Appendix Section 7.5 for detail.
Evaluating the generated graphs. Evaluating the sample quality of GGMs requires a comparison between two sets of graphs — the generated graphs and the (held-out) test sets [6, 88, 89]. Qualitative evaluation compares the generated graphs by visual inspection.
Quantitative approaches compare the distance between the distribution of test graphs and the generated graphs. We use GNN-based [68, 72] and statistics-based [88] metrics to measure the fidelity (realism) and the diversity of generated graphs. GNN-based metrics extract graph representations with a reference GNN that is independent of graph statistics. The reference GNN is either randomly initialized (Random-GNN) [72] or contrastively trained (Pretrained-GNN) [68]. Evaluation metrics then compute the discrepancy between the test set representations and the generated set representations; we report F1 PR and MMD RBF. Statistic-based evaluation metrics compute the MMD between the test set and generated set with respect to structural properties (orbit counts, degree coefficients, clustering coefficients and diameter) [43]. For all GNN-based metrics, we used 10 GNNs with different random initializations and reported mean  $\pm$  standard deviation across different GNNs [72]. Following O'Bray et al. [55] we report scores computed from a 50/50 split of the data sets as the ideal score. As in previous work, all models are trained with one random weight initialization to keep the training time feasible [6, 42, 88, 89]. Each trained model is used to generate  $S^{\prime}$  new graphs to compare them with the  $S^{\prime}$  graphs in the test set. See Appendix Section 7.6 for further detail.
4.1 Comparison of GenStat with statistics-based GGMs on graph realism.
We compare GenStat to non-neural baseline models that are widely used in network science. Qualitative evaluation. Figure 3 is a visual comparison of randomly selected test and generated graphs graphs for statistics-based GGMs. The GenStat graphs are visually much more similar to the test samples than the baseline graphs. For example, instead of Lobster trees, baselines often generate samples that contain multiple cycles. IMDb graphs exhibit a community structure where multiple small communities are interconnected through a central node. For instance, the left IMDb test sample in Figure 3 consists of six communities. None of the baselines was able to generate a graph with high modularity. The Protein samples generated by the baselines exhibit a higher density compared to the test samples. The GenStat repository contains the complete collection of generated graphs.
Quantitative evaluation. The quantitative evaluation confirms that the quality of graphs generated by GenStat is substantially higher. Table 1 reports the GNN-based MMD RBF scores. GenStat achieved better MMD RBF scores than the baseline GGMs in 12 out of 16 cases. Notably, GenStat generated samples have MMD RBF scores up to ten times smaller than the baselines for the Random-GNN and Pretrained-GNN approaches. Appendix Table 3 reports the GNN-based F1 PR. The F1 PR of generated graphs by GenStat is substantially higher than those of the baselines. For example, on Protein, the GenStat F1 PR score is  $83.72\%$  vs.  $71.08\%$  for the next best method with RandomGNN, and  $79.95\%$  vs.  $23.76\%$  for Pretrained-GNN. Appendix Table 4 shows the results for the statistics-based evaluation metrics. GenStat generates graphs with up to 1-2 orders of magnitude better statistics-based MMDs, at least in one of the reported MMDs, on almost all datasets.
4.2 Comparison of GenStat with deep adjacency-based GGMs on graph realism
Qualitative evaluation. Appendix Figure 7 provides a visual comparison of the graphs generated by GenStat and deep SOTA adjacency-based GGMs. On the real-world datasets, the GRAN graphs are less realistic than the GenStat graphs. For the other comparison methods, the quality of their graphs is visually indistinguishable from those generated by GenStat.
Quantitative evaluation. Table 2 compares GenStat with adjacency-based GGMs in terms of the GGN-based MMD RBF score of the generated graphs. Although GenStat is limited to graph statistics, for real-world datasets, it ranks among the top two models in 9 out of 10 reported MMD RBF scores. On the synthetic datasets the GenStat scores are competitive with the GraphRNN and GRAN scores but do not reach the level of BiGG and GraphVAE-MM. The difference is that the synthetic datasets
Figure 3: Visualization of generated graphs by the statistics-based GGMs. The top row shows randomly selected graphs from the test set for each dataset, with varying structural characteristics. The other rows show graphs generated by each model. The generated graphs shown are the two visually most similar samples in the generated set.
Table 1: Comparison of GenStat with statistics-based GGMs for the GNN-Based MMD RBF score (lower is better). The best result is in bold and the second best is underlined.
Dataset	Descriptor	50/50 split	GenStat*	BTER	Chung-Lu	Erdos-Roni	SBM
ogbg-mol	Random-GNN	0.00 ± 0.00	0.05 ± 0.01	0.21 ± 0.12	0.05 ± 0.00	0.12 ± 0.07	0.10 ± 0.03
Pretrained-GNN	0.00 ± 0.00	0.01 ± 0.01	0.07 ± 0.00	0.07 ± 0.00	0.22 ± 0.02	0.21 ± 0.03
Protein	Random-GNN	0.00 ± 0.00	0.00 ± 0.00	0.06 ± 0.02	0.64 ± 0.40	0.59 ± 0.36	0.27 ± 0.13
Pretrained-GNN	0.01 ± 0.00	0.00 ± 0.00	0.21 ± 0.04	0.60 ± 0.16	0.59 ± 0.07	0.50 ± 0.04
IMDb	Random-GNN	0.00 ± 0.00	0.05 ± 0.03	0.20 ± 0.15	0.23 ± 0.15	0.16 ± 0.14	0.12 ± 0.03
Pretrained-GNN	0.01 ± 0.00	0.08 ± 0.02	0.07 ± 0.02	0.27 ± 0.07	0.16 ± 0.06	0.20 ± 0.09
PTC	Random-GNN	0.01 ± 0.00	0.00 ± 0.00	0.04 ± 0.00	0.09 ± 0.00	0.10 ± 0.04	0.19 ± 0.04
Pretrained-GNN	0.02 ± 0.00	0.00 ± 0.00	0.08 ± 0.01	0.12 ± 0.01	0.17 ± 0.02	0.18 ± 0.02
MUTAG	Random-GNN	0.00 ± 0.00	0.05 ± 0.05	0.25 ± 0.05	0.27 ± 0.07	0.37 ± 0.21	0.27 ± 0.16
Pretrained-GNN	0.01 ± 0.00	0.00 ± 0.00	0.18 ± 0.02	0.24 ± 0.03	0.41 ± 0.03	0.35 ± 0.06
Lobster	Random-GNN	0.03 ± 0.00	0.35 ± 0.10	0.15 ± 0.04	0.14 ± 0.02	0.44 ± 0.10	0.22 ± 0.08
Pretrained-GNN	0.10 ± 0.00	0.04 ± 0.03	0.17 ± 0.01	0.11 ± 0.06	0.32 ± 0.03	0.27 ± 0.03
Grid	Random-GNN	0.02 ± 0.00	0.53 ± 0.28	0.41 ± 0.28	0.59 ± 0.45	0.40 ± 0.27	0.61 ± 0.36
Pretrained-GNN	0.10 ± 0.00	1.17 ± 0.10	1.21 ± 0.13	1.39 ± 0.09	1.21 ± 0.13	1.25 ± 0.09
Triangle Grid	Random-GNN	0.00 ± 0.00	0.33 ± 0.07	0.38 ± 0.29	1.18 ± 0.29	1.10 ± 0.27	0.78 ± 0.34
Pretrained-GNN	0.03 ± 0.00	1.11 ± 0.13	0.80 ± 0.17	1.30 ± 0.14	1.20 ± 0.12	1.20 ± 0.11
exhibit highly regular local structure (e.g., a grid pattern). We observed similar results for the F1 PR and statistics-based scores (Appendix Tables 5 and 6). Apparently the global graph statistics used by GenStat are insufficient for expressing strict local constraints.
In sum, our deep statistics-based GGM GenStat outperforms previous parametric statistics-based GGMs by orders of magnitude. In an apples-to-oranges comparison with deep GGMs trained on complete adjacency matrices, it is competitive or superior on real-world datasets, but not on synthetic datasets with highly regular local structures such as grid patterns.
4.3 Graph realism and edge local differential privacy
We examine training both statistics-based and adjacency-based GGMs with an Edge LDP guarantee. For adjacency-based GGMs, the randomized neighbour list (RNL) approach [58] can be used. Each node randomly flips each bit in its neighbour list with probability  $\frac{1}{1 + e^{\epsilon}}$  and sends the perturbed neighbour list to the untrusted curator. A adjacency-based GGM can then be trained on the collected perturbed adjacency matrix. For an  $\epsilon$ -Edge LDP guarantee, GenStat was trained on local ego-graph (node-level) statistics, each perturbed by the Laplace mechanism with variance  $\frac{1}{(M\epsilon)^2}$  [14].
Table 2: Comparison of GenStat with adjacency-based GGMs for the GNN-Based MMD RBF score (lower is better). The best result is in bold and the second best is underlined.
Dataset	Descriptor	50/50 split	GenStat*	GraphVAE-MM	BiGG	GRAN	GraphRNN-S	GraphRNN
ogbg-mol	Random-GNN	0.00 ± 0.00	0.05 ± 0.01	0.01 ± 0.00	0.04 ± 0.00	0.44 ± 0.02	0.44 ± 0.15	1.53 ± 0.03
Pretrained-GNN	0.00 ± 0.00	0.01 ± 0.01	0.04 ± 0.00	0.03 ± 0.00	0.25 ± 0.03	0.54 ± 0.04	0.86 ± 0.09
Protein	Random-GNN	0.00 ± 0.00	0.00 ± 0.01	0.06 ± 0.01	0.17 ± 0.07	0.05 ± 0.02	0.56 ± 0.18	1.43 ± 0.32
Pretrained-GNN	0.01 ± 0.00	0.00 ± 0.00	0.17 ± 0.01	0.11 ± 0.00	0.11 ± 0.02	0.97 ± 0.06	1.73 ± 0.21
IMDb	Random-GNN	0.00 ± 0.00	0.05 ± 0.03	0.08 ± 0.02	0.02 ± 0.00	0.02 ± 0.00	1.45 ± 0.03	0.99 ± 0.06
Pretrained-GNN	0.01 ± 0.00	0.08 ± 0.02	0.09 ± 0.03	0.03 ± 0.00	0.42 ± 0.27	1.2 ± 0.23	0.75 ± 0.12
PTC	Random-GNN	0.01 ± 0.00	0.00 ± 0.00	0.03 ± 0.00	0.03 ± 0.00	0.14 ± 0.03	0.67 ± 0.11	0.81 ± 0.16
Pretrained-GNN	0.02 ± 0.00	0.00 ± 0.00	0.06 ± 0.00	0.03 ± 0.00	0.18 ± 0.03	0.53 ± 0.03	0.51 ± 0.11
MUTAG	Random-GNN	0.00 ± 0.00	0.05 ± 0.05	0.09 ± 0.04	0.03 ± 0.00	0.09 ± 0.00	0.53 ± 0.12	1.08 ± 0.05
Pretrained-GNN	0.01 ± 0.00	0.00 ± 0.00	0.13 ± 0.03	0.09 ± 0.01	0.10 ± 0.01	0.50 ± 0.14	0.31 ± 0.05
Lobster	Random-GNN	0.03 ± 0.00	0.35 ± 0.10	0.09 ± 0.00	0.11 ± 0.00	0.16 ± 0.07	0.86 ± 0.08	0.62 ± 0.03
Pretrained-GNN	0.10 ± 0.00	0.04 ± 0.03	0.10 ± 0.00	0.11 ± 0.00	0.29 ± 0.03	0.74 ± 0.07	0.22 ± 0.01
Grid	Random-GNN	0.02 ± 0.00	0.53 ± 0.28	0.14 ± 0.01	0.35 ± 0.00	0.40 ± 0.00	0.92 ± 0.05	1.04 ± 0.09
Pretrained-GNN	0.10 ± 0.00	1.17 ± 0.10	0.35 ± 0.14	0.29 ± 0.10	0.45 ± 0.09	1.27 ± 0.07	1.06 ± 0.05
Triangle Grid	Random-GNN	0.00 ± 0.00	0.33 ± 0.07	0.18 ± 0.00	0.38 ± 0.10	0.31 ± 0.18	0.79 ± 0.14	0.94 ± 0.08
Pretrained-GNN	0.03 ± 0.00	1.11 ± 0.13	0.15 ± 0.02	0.34 ± 0.07	0.42 ± 0.27	0.88 ± 0.16	0.96 ± 0.10
We compare the quality of graphs generated by GenStat and BiGG, the best adjacency-based GGM, against ground-truth test graphs, when each is trained under differential privacy. We also compare perturbed adjacency matrices with the test graphs. This helps us understand the optimal score that can be achieved by an adjacency-based GGM trained on the perturbed adjacency matrices. Graph statistics used in the GenStat model for this experiment are triangle histogram and degree histogram. Figure 4 compares the quality scores under the edge  $\epsilon$ -LDP guarantee, for  $\epsilon \in \{0.1, 0.5, 1, 2, 3, 4\}$ , using the three datasets on which BiGG showed the biggest advantage over GenStat when trained without perturbations (Section 4.2); see the Appendix for other datasets. GenStat generates more realistic graphs for almost all  $\epsilon$ -privacy budgets, as indicated by much lower MMD-RBF scores.
(b) Lobster
Figure 4: Comparison of GenStat (statistics-based GGM) with BiGG (adjacency-based GGM) under  $\epsilon$ -Edge LDP guarantee, in terms of the Random GNN-Based MMD RBF score. A lower score is better. The lower bound  $\epsilon = 0$  ensures perfect privacy.
(c) Grid
4.4 Benchmark effectiveness
A use case for statistics-based GGMs is generating privacy-controlled synthetic graphs for benchmarking the performance of GNNs on downstream tasks, when privacy concerns limit access to the original adjacency matrices in a data collector. The idea behind benchmark effectiveness is that "performance rankings among  $m$  GNN models on generated graphs should be similar to the rankings among the same  $m$  GNN models on the original graphs" [87]. The methodology is to evaluate each GNN model on the downstream task twice: first on the original dataset, and second on a synthetic dataset generated by the GGM. The benchmark effectiveness of the GGM is then measured by the correlation between the GNNs' task scores from the original and the task scores from the synthetic data. Following [87], we benchmark GNNs on link prediction; see Section 7.8 for further detail. Table 8 compares the benchmark effectiveness of GenStat with that of BiGG, the SOTA adjacency-based GGM. On 4 out of 8 datasets, the GenStat benchmark effectiveness is competitive with or superior to the benchmark effectiveness of BiGG. For the other datasets, the benchmark effectiveness of BiGG is better, but the GenStat graphs still show a substantive correlation (at least 0.5046 for Pearson correlation). For most of the datasets, the MSE of GenStat is smaller than that of BiGG. In our opinion, the benchmark effectiveness of GenStat for link prediction is impressive,
especially considering that the model does not observe specific links during training time, unlike BiGG.
4.5 Generation and training time
Evaluating the edge reconstruction probability is expensive and tends to dominate the training time of adjacency-based GGMs. Training statistics-based methods is therefore generally faster than training adjacency-based methods. Specifically, the training time of GenStat is up to two orders of magnitude lower than that of the fastest auto-regressive model, and up to 5 times lower than that of GraphVAE-MM. In terms of generation time, both GraphVAE-MM and GenStat are much faster than auto-regressive methods, because they generate graphs all-at-once rather than incrementally. Tables 9 and 10 in the Appendix give a detailed comparison of the train and generation time of deep GGMs.
5 Discussion and limitations
Graph generative models have a potential attack surface that reveals sensitive information about individuals. Our model contributes to an effective analysis of network structure while maintaining privacy guarantees and minimizing access to the sensitive information of individuals. We expect the social impact of our work to be positive. We discuss the limitations of the GenStat +VAE design.
Attributed and heterogeneous graphs. Following [6, 88, 89], the reported research studies graph generation issues with relatively simple homogeneous graphs. GenStat can be extended to attributed/heterogeneous graphs, where nodes/edges possess attributes, including potentially sensitive ones. The graph descriptors can be defined as functions of both feature matrices and edge tensors.
Computational complexity. In our implementation, we used matrix multiplication with  $O(N^3)$  complexity to exactly compute the descriptors  $\Phi(\cdot)$ . Approximating graph statistics [16, 32, 59] and exploiting the sparsity of real-word graphs are promising avenues for scaling to large graphs.
Neural network design. Following [70, 89], we used FCNN decoders to generate probabilistic adjacency matrices. The all-at-once parallel edge generation of FCNN decoders enables fast training and generation time [24, Ch.9.1.2]. However, they require specifying a maximum number of nodes, and do not scale to large graphs. These limitations in our current system can be addressed with more scalable decoders, e.g. graph transformers [76].
A statistics-based graph generative model (GGM) is trained on graph statistics that summarize the graph, rather than a complete adjacency matrix. Non-neural parametric models for statistics-based graph generation have been developed in network science for decades. We have described a new GenStat framework, which to our knowledge is the first deep GGM architecture based on statistics. Our main motivation for statistics-based graph generation is to avoid requiring sensitive information from individual network participants, especially in the decentralized setting without a single trusted data curator. We show that if local node-level statistics are collected from individuals under edge local differential privacy (LDP), applying GenStat to the statistics preserves LDP. In empirical evaluation on eight datasets, graphs generated by GenStat were substantially more realistic than those generated by previous statistics-based methods (e.g., by an order of magnitude on the standard MMD RBF quality metric). On real-world datasets, GenStat graphs show competitive quality to SOTA GGMs that are based on the entire adjacency matrix, even though GenStat sees only summary statistics. We also show that because GenStat learns on compressed graph information, training time is much faster than with adjacency-based methods.
A valuable direction for future work is extending statistics-based graph generation to attributed and heterogeneous graphs. There are several directions for scaling GenStat architectures to large graphs, such as approximating expected graph statistics, leveraging graph sparsity, and using more scalable decoders. In terms of application tasks, it would be useful to evaluate how well GenStat supports learning for downstream tasks (link prediction, node classification) with an LDP guarantee [87].
In sum, statistics-based graph generation is fast and effective in generating high-quality graphs. It offers a strong option for graph learning that respects individuals privacy.