SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kiarash Zahirnia $^{1}$ , Yaochen Hu $^{2}$ , Mark Coates $^{3}$ , Oliver Schulte $^{1*}$
<sup>1</sup>Simon Fraser University, <sup>2</sup>Huawei Noah's Ark Lab, <sup>3</sup>McGill University
kzahirni@sfu.ca, yaochen.hu@huawei.com, mark.coates@mcgill.ca, oschulte@sfu.ca
We describe a new setting for learning a deep graph generative model (GGM) from aggregate graph statistics, rather than from the graph adjacency matrix. Matching the statistics of observed training graphs is the main approach for learning traditional GGMs (e.g., BTER, Chung-Lu, and Erdos-Renyi models). Privacy researchers have proposed learning from graph statistics as a way to protect privacy. We develop an architecture for training a deep GGM to match statistics while preserving local differential privacy guarantees. Empirical evaluation on 8 datasets indicates that our deep GGM generates more realistic graphs than the traditional non-neural GGMs when both are learned from graph statistics only. We also compare our deep GGM trained on statistics only, to state-of-the-art deep GGMs that are trained on the entire adjacency matrix. The results show that graph statistics are often sufficient to build a competitive deep GGM that generates realistic graphs while protecting local privacy.
Graph generative models (GGMs) have produced many insights into fundamental processes in domains including biology, engineering, and social sciences. Current deep GGMs are based on training data with complete adjacency matrices [6, 43, 88, 89]. This paper presents GenStat, a new deep GGM architecture for a setting where the available graph data are summarized by graph statistics, not a complete adjacency matrix. We refer to this setting as statistics-based graph generation. Previous work in network analysis has introduced several parametric non-neural models that support statistics-based graph generation, such as the Chung-Lu model [5], and the BTER model [67]. The parametric models tend to generate less realistic graphs due to lack of expressive power.
Our main motivation for generating graphs from statistics is privacy preservation. There is a direct tension between releasing real graphs to the research community and privacy concerns of the individual entities (graph nodes) [63, 85]. A promising proposal to address it is to release synthetic graphs that preserve the original graph properties while guaranteeing a user-specified level of privacy [20, 41]. Benchmarking Graph Neural Networks (GNNs) is a use-case of GGMs with a privacy guarantee. Synthetic graphs generated by GGMs enable GNN research without information leakage [87].
Statistics-based graph generation supports the challenging use case of decentralized graphs [86], where privacy concerns rule out collecting adjacency matrices in a central repository. Common examples are social graphs [58], e.g., users' connections through their phone contact lists, face-to-face interactions, sexual and friendship networks, or distributed social networks, e.g., Mastodon [58]. A solution is to have each entity perturb its data and send the perturbed data to a curator [86]. However, collecting raw data locally, such as an entity's neighbour list, requires heavy noise injection to satisfy privacy, and may result in a dense, distorted graph with low utility. An alternative, used widely in industry including Google, Apple, and Microsoft [11, 19, 71, 78], is to collect, from each entity's ego-graph, local node-level graph statistics [24][Ch.2], with a guarantee of Local Differential
*Supported by NSERC Canada Discovery Grant R611341
Privacy (LDP) as a strong privacy measure [58]. The perturbed node-level statistics are aggregated to summarize important graph properties [58, 86, 91]. The idea is that coarser-grained information requires much less noise to satisfy LDP [58]. The final step is to generate representative synthetic graphs for public release. Previous works use a parametric statistics-based GGM [34, 35, 48, 58, 63]. Our GenStat system is a neural alternative that generates substantially more realistic graphs. Figure 1 summarizes the privacy-preserving workflow.
Figure 1: Generating realistic-looking graphs with Local Differential Privacy. See text for detail.
Evaluation. We evaluate the realism of the generated graph structures following previous work [6, 41, 58, 68, 72, 88]. Experiments on 8 datasets with diverse characteristics demonstrate the effectiveness of GenStat. Compared to parametric statistics-based methods, GenStat generates graphs that are up to 10 times more realistic on Maximum Mean Discrepancy (MMD) metrics. Compared to state-of-the-art GGMs (that require access to all adjacencies), the GenStat graphs reach very competitive graph quality, especially on real graph datasets, such as datasets of molecules and chemical compounds. Thus in many domains, graph statistics are sufficient for learning to generate realistic graphs. We also evaluate the effectiveness of GenStat for benchmarking GNNs on link prediction as a downstream task and we show GenStat is highly effective. Moreover, graph learning from statistics is much faster than from adjacency matrices. The implementation and datasets are provided at GenStat repository https://github.com/kiarashza/GenStat.git and explained in the Appendix Section 7.9.
- We introduce GenStat, a novel deep graph generative architecture that requires access only to graph-level statistics. To our knowledge, GenStat is the first deep GGM that does not require observations of individual nodes or edges.
- We show that given training statistics collected with a local differential privacy guarantee, GenStat also satisfies local differential privacy, while generating high-quality graphs.
- We identify permutation-invariant differentiable graph statistics that are based on aggregating node-level information and support realistic graph generation across multiple diverse domains.
Realistic graph generation has been studied extensively for decades, leading to the development of both parametric and neural approaches [24][Ch.8]. Figure 5 in the Appendix positions GenStat in the GGM landscape. The objective of previous parametric statistics-based GGMs such as [1, 5, 18, 21, 40, 56, 67] is to generate graphs with similar statistical properties to observed graphs [46]. The expressive power of their parameter space is limited. More recent machine learning models, including Deep GGMs [6, 88, 89], have a higher capacity to generate realistic graphs [6]. However, to our knowledge, they all assume access to a complete adjacency matrix. Such access is incompatible with privacy concerns since adjacency matrices reveal the entities with which an entity has interacted. In contrast, GenStat learns a deep GGM using only observed graph statistics and generates realistic graphs for a variety of domains.
Graph statistics. Zahirnia et al. [89] recently introduced a joint probabilistic micro-macro model over both adjacencies and graph statistics and showed that matching graph-level statistics is an effective regularizer for the edge reconstruction loss, which assumes access to all adjacencies. GenStat operates on permutation-invariant graph statistics that can be collected locally and privately, whereas some of the statistics used in [89] require all adjacencies.
Statistical databases are database systems that present only aggregate statistics (e.g., sample mean and count) for a subset of the entities represented in the database and ensure that sensitive information is safeguarded while still enabling valuable statistical analysis and research. An example is the database maintained by the U.S. Census Bureau [39]. Statistical databases also protect privacy through aggregate statistics and can be analyzed using the GenStat framework.
Permutation-invariance is a fundamental property of graph-structured data [24], which suggests that a GGM should produce the same output irrespective of the node ordering employed in the training set. It is satisfied by many GGMs [23, 28, 33, 54, 76], but not all [6, 88, 89]. Since GenStat takes as input permutation-invariant statistics, its training and the generative graph distribution are permutation-invariant as well (Section 3.3).
Differential privacy (DP) [15] enables the extraction of useful information about a population while providing strong privacy guarantees for individuals. Local differential privacy (LDP) [36] is a stronger guarantee for the decentralized setting [86]. Each entity perturbs its sensitive information before transmitting it to an untrusted curator. Collecting decentralized network statistics under LDP has been deployed by major technology companies, including Google, Apple, and Microsoft [11, 19, 71]. For example, Google's RAPPOR [19] collects randomized statistics to enable analysis of popular web destinations without revealing individual browsing habits.
Graphs and privacy. Privacy-preserving techniques have been proposed for the release and generation of graph data [41, 50, 51, 63]. Yang et al. [85] and Yoon et al. [87] leverage DP to enforce privacy constraints on deep GGMs. Yoon et al. [87] advocate using generated graphs to replace original graphs in GNN research. Therefore the GGM should generate effective benchmark graphs, meaning that GNNs show similar task performance as on the original source graphs. In the decentralized setting, entities perturb sensitive information locally to support the generation and release of synthetic graphs [58, 78, 86, 91]. Theoretical analysis shows that perturbing local adjacencies impairs graph quality too much for synthetic graphs to be useful [58]. Our experiments confirm this for SOTA deep GGMs (Section 4.3). Therefore, recent studies train parametric GGMs with perturbed local ego-graph statistics [58, 78] to achieve LDP. To our knowledge, GenStat is the first deep GGM that can generate synthetic graphs under LDP.
Graph Anonymization is a procedure that disguises or modifies information in graphs, making it anonymous. Anonymization techniques have two main limitations [50, 63]. These techniques 1) are mainly applicable in a centralized setting and 2) typically protect only against specific known attacks. Narayanan and Shmatikov [51] describe a de-anonymization algorithm for an anonymized binary adjacency matrix that effectively re-identifies the nodes in real word graphs, Twitter and Flicker.
Federated Learning is a privacy-preserving paradigm for building models from separate data sources [90]. [20, 25, 80] used federated learning to learn GNNs on graphs from multiple data sources. These studies assume that each source has trusted access to a sufficiently large subgraph to locally train an accurate GNN [80], and they do not target the construction of a GGM.
3 Problem definition and method
Given a set of observed graphs  $\hat{G} = \{\mathcal{G}^1,\dots,\mathcal{G}^S\}$ , with variable number of nodes, sampled from a data distribution  $p(G)$ , the goal of GGMs is to learn a model that can generate similar synthetic graphs [6, 88, 89]. We introduce GenStat, a statistics-based architecture for probabilistic GGMs.
GenStat assumes that a training graph  $\mathcal{G}^i$  is summarized by a set of observed  $M$  graph statistics, denoted as  $\mathcal{I}^i = \{\mathcal{I}_m^i\}_{m=1}^M$ , where the statistic  $\mathcal{I}_m$  is a vector of dimension  $D_m$ . Each statistic is computed by a descriptor function  $\phi_m : [0,1]^{n \times n} \to \mathbb{R}^{+D_m}$  that maps an adjacency matrix representing a graph with  $n$  nodes to a vector; we write  $\phi_m(\mathbf{A}^i) = \mathcal{I}_m^i$  and  $\Phi(\mathbf{A}^i) = \mathcal{I}^i$ . Permutation-invariant statistics satisfy  $\Phi(\mathbf{A}^i) = \Phi(\mathbf{A}_\pi^i)$  for all adjacency matrices obtained from  $\mathbf{A}^i$  through a permutation  $\pi$  [10, 52, Ch. 7]. Section 3.5 defines the statistics in our experiments, namely histograms of triangles, neighborhood sizes, random walks and graph size.
3.1 Probabilistic model of graph statistics
Given a distribution  $p(\mathbf{A})$  over adjacency matrices, we let  $\mathcal{I}$  denote the random variable defined by applying the descriptor functions to a random graph  $\mathbf{A}$  (i.e.,  $\mathcal{I} = \{\phi_m(\mathbf{A})\}_{m=1}^M$ ). Following Ma et al.
[45], we view the latent adjacency matrix  $\mathbf{A}$  as a sample from an underlying probabilistic adjacency matrix  $\tilde{\mathbf{A}}$  with  $\tilde{\mathbf{A}}_{u,v} \in [0,1]$  specifying independent link probabilities. We define the mixture model
$$
p (\mathbf {A} | \tilde {\mathbf {A}}) = \prod_ {u = 1} ^ {n} \prod_ {v = 1} ^ {n} \tilde {\mathbf {A}} _ {u, v} ^ {\mathbf {A} _ {u, v}} (1 - \tilde {\mathbf {A}} _ {u, v}) ^ {1 - \mathbf {A} _ {u, v}} \tag {1}
$$
$$
p (\mathbf {A}) = \int p (\mathbf {A} | \tilde {\mathbf {A}}) p (\tilde {\mathbf {A}}) d \tilde {\mathbf {A}} \qquad p (\tilde {\mathbf {A}}) = \int p (\tilde {\mathbf {A}} | Z) p (Z) d Z,
$$
where  $Z_{1 \times d}$  is a graph latent representation [24][Sec. 9.1] with associated prior  $p(Z)$ . The marginal distribution over graph statistics is:
$$
p _ {\theta} (\mathcal {I}) = \int \int p _ {\theta} (\mathcal {I} | \tilde {\mathbf {A}}) p _ {\theta} (\tilde {\mathbf {A}} | Z) p (Z) d \tilde {\mathbf {A}} d Z. \tag {2}
$$
We assume graph statistics are independent given the probabilistic matrix  $\mathbf{A}$  and model the conditional distribution of each graph statistic as a Gaussian with diagonal variance parameter  $\sigma_m^2$ ,
$$
p _ {\theta} \left(\mathcal {I} _ {1}, \dots , \mathcal {I} _ {M} \mid \tilde {\mathbf {A}}\right) = \prod_ {m = 1} ^ {M} \mathcal {N} \left(\mathcal {I} _ {m} \mid \phi_ {m} (\tilde {\mathbf {A}}), \sigma_ {m} ^ {2} I\right). \tag {3}
$$
Given the conditional distribution Equation (3), the marginal distribution Equation (2) is a mixture of Gaussians. Figure 2a illustrates the probabilistic graphical model of GenStat.
(b)
Figure 2: (a) The proposed latent variable model. The diagram shows the dependency of graph statistics  $\{\mathcal{I}_m^i\}_{m = 1}^M$  on the latent probabilistic adjacency  $\tilde{\mathbf{A}}$  and graph-level latent representation  $\mathbf{Z}$ . (b) The model's overall architecture.
3.2 Training and variational lower bound
For an i.i.d. sample of graphs with associated statistics,  $\mathcal{I}^1,\ldots ,\mathcal{I}^S$ , the marginal log-likelihood is the sum of the marginal log-likelihoods of the individual graphs,  $\log p_{\theta}(\mathcal{I}^{1},\dots ,\mathcal{I}^{S}) = \sum_{i}\log p_{\theta}(\mathcal{I}^{i})$ . The double integral in Equation (2) is generally intractable. We approximate the mixture integral with a Variational Auto-Encoder (VAE) to minimize the negative ELBO as an approximation of the negative log-likelihood for graph statistics:
Proposition 1. Let  $p_{\theta}$  be the marginal likelihood defined in Equation (2). Then
$$
\begin{array}{l} - \ln p _ {\theta} (\mathcal {T} ^ {i}) = - \ln p _ {\theta} (\mathcal {T} _ {1} ^ {i}, \dots , \mathcal {T} _ {M} ^ {i}) \leq E _ {Z \sim q _ {\theta} (Z | \mathcal {T} _ {1} ^ {i}, \dots , \mathcal {T} _ {M} ^ {i})} \big [ - \ln \int p _ {\theta} (\mathcal {T} _ {1} ^ {i}, \dots , \mathcal {T} _ {M} ^ {i} | \tilde {\mathbf {A}}) p _ {\theta} (\tilde {\mathbf {A}} | Z) d \tilde {\mathbf {A}} \big ] \\ + K L \left(q _ {\theta} \left(Z \mid \mathcal {I} _ {1} ^ {i}, \dots , \mathcal {I} _ {M} ^ {i}\right) | | p (Z)\right). \tag {4} \\ \end{array}
$$
The proof is in the Appendix Section 7.1.1. We approximate the inner integral with a Monte Carlo estimate of the expectation of  $p_{\theta}(\mathcal{I}_1^i,\ldots ,\mathcal{I}_M^i |\tilde{\mathbf{A}})$  w.r.t.  $p_{\theta}(\tilde{\mathbf{A}} |Z)$  as follows:
$$
\begin{array}{l} \int p _ {\theta} (\mathcal {I} _ {1} ^ {i}, \ldots , \mathcal {I} _ {M} ^ {i} | \tilde {\mathbf {A}}) p _ {\theta} (\tilde {\mathbf {A}} | Z) d \tilde {\mathbf {A}} = E _ {p _ {\theta} (\tilde {\mathbf {A}} | Z)} [ p _ {\theta} (\mathcal {I} _ {1} ^ {i}, \ldots , \mathcal {I} _ {M} ^ {i} | \tilde {\mathbf {A}}) ] \\ \approx \frac {1}{T} \sum_ {t = 1} ^ {T} p _ {\theta} \left(\mathcal {I} _ {1} ^ {i}, \dots , \mathcal {I} _ {M} ^ {i} \mid \tilde {\mathbf {A}} ^ {t}\right) \quad \text {w h e r e} \quad \tilde {\mathbf {A}} ^ {t} \sim p _ {\theta} (\tilde {\mathbf {A}} | Z), \tag {5} \\ \end{array}
$$
where the conditional distribution of the underlying probabilistic adjacency matrix can be modelled as a Beta distribution [44].
We implement the GenStat objective (4) with an Auto-Encoder in which fully connected neural networks (FCNNs) are used to jointly learn  $q_{\theta}(Z|\mathcal{I}_1^i,\dots,\mathcal{I}_M^i)$  and  $p_{\theta}(\tilde{\mathbf{A}} |Z)$  with learning parameters  $\theta$ . Figure 2b illustrates the VAE design of GenStat. The prior  $p(Z)$  is a standard normal distribution. The variational posterior  $q_{\theta}(Z|\mathcal{I}_1^i,\dots,\mathcal{I}_M^i)$  is a factored Gaussian with vector mean and variance. See Appendix Section 7.2 for the neural network design and more details concerning implementation.
3.3 Permutation-invariant graph generation
Given permutation-invariant statistics, GenStat training is permutation-invariant in the sense that the gradient updates are permutation-invariant [76]. Because training is permutation-invariant, the model distribution  $p_{\theta}(\mathbf{A})$  from Equation (1) can be made permutation-invariant by applying a uniform random permutation to the output of the decoder. In addition to generating synthetic graphs, a major use of statistics-based generative models is to define an inference distribution to support prediction tasks like link prediction and graph classification [60]. These inference models apply to graph distributions that assign the same probability to graphs with the same statistics, which is appropriate when the graph model is based on statistics only. Accordingly, we define the following permutation-invariant inference distribution for a GenStat model:
$$
P _ {\theta} (\mathbf {A}) = p _ {\theta} \left(\boldsymbol {\Phi} (\mathbf {A})\right) / C _ {\boldsymbol {\Phi} (\mathbf {A})}, \tag {6}
$$
where the distribution  $p_{\theta}(\mathcal{I})$  over statistics follows Equation (2) and  $C_{\Phi (\mathbf{A})}$  is the number of graphs that generate the same statistics as  $\mathbf{A}$  (i.e.,  $C_{\Phi (\mathbf{A})} = |\{\mathbf{A}' : \Phi (\mathbf{A}') = \Phi (\mathbf{A})\}|$ ).
The following statement summarizes these observations. Appendix Section 7.1.2 provides a proof.
Observation 1. Suppose that a GGM parameterized by  $\theta$  is trained with the GenStat architecture and permutation-invariant descriptor functions  $\Phi$ . Then the following hold.
1. The gradient updates of  $\theta$  given a training graph  $A^i$  are permutation-invariant.
2. The model distribution  $p_{\theta}(\mathbf{A})$  is permutation-invariant if the generated adjacency matrix is computed by applying a random permutation to the GenStat output.
3. The inference distribution  $P_{\theta}(\mathbf{A})$  in Equation (6) is permutation-invariant.
3.4 GenStat privacy analysis
This section shows that training the GenStat architecture on statistics collected under an LDP guarantee, also satisfies LDP. We employ a graph LDP concept known as Edge LDP [58, 78], which guarantees plausible deniability for the inclusion or removal of a particular edge associated with an individual (node). The neighbor list of  $u$  is an  $n$ -dimensional vector  $l_{u} = [l_{u1},\dots,l_{un}]$ , with  $l_{uv}\in \{0,1\}$  and  $l_{uv} = 1$  iff  $u$  and  $v$  are connected.
Definition 1 (ε-Edge LDP). A randomized algorithm  $R$  satisfies  $\epsilon$ -Edge LDP if and only if, for any two neighbour lists  $l$  and  $l'$ , such that  $l$  and  $l'$  only differ in one bit, and for any output value  $s \in \text{range}(R)$ , the following inequality holds,
$$
p (R (l) = s) \leq e ^ {\epsilon} p \left(R \left(l ^ {\prime}\right) = s\right). \tag {7}
$$
An example of a randomized algorithm  $R$  is a perturbed node degree: A user calculates her true node degree, adds noise with a perturbation mechanism (e.g., the Laplace mechanism [13]), and sends the resulting noisy degree to a data curator. The curator aggregates the perturbed node degrees as local graph statistics that can be used to train statistics-based GGMs, such as GenStat. The following proposition, with proof in Appendix Section 7.1.3, captures the LDP-preserving properties of GenStat.
Proposition 2. Let  $R = (R_{1} \ldots R_{M})$  be a set of independent randomized algorithms, outputting perturbed node-level statistics, such that algorithm  $R_{m}$  satisfies  $\epsilon_{m}$ -Edge LDP. Then a GenStat GGM trained on the outputs of  $R$  satisfies  $\sum_{m=1}^{M} \epsilon_{m}$ -Edge LDP.
3.5 Observed graph statistics
Graph statistics used in a GenStat model are computed by permutation-invariant descriptor functions  $\phi(\tilde{\mathbf{A}})$  that can be applied to probabilistic as well as binary adjacency matrices, and are differentiable with respect to entries in the latent adjacency matrix  $\tilde{\mathbf{A}}$ . To support local privacy, we use graph statistics that aggregate node-level statistics:
$$
F (\mathcal {G}) = A G G \left(\left\{f \left(\mathcal {G} _ {u}\right): u \in \{1, 2, \dots , n \} \right\}\right), \tag {8}
$$
where  $\mathcal{G}_u$  denotes the node  $k$ -hop ego network of node  $u$  in graph  $\mathcal{G}$  with  $n$  nodes, the local statistic  $f(\cdot)$  returns a real-valued vector for an ego-graph, and  $AGG(\cdot)$  is a permutation invariant function (e.g., average, sum, histogram) that takes as input the set of statistics (e.g., node degree) of the ego-graphs and summarizes them into a vector representation. When  $\mathbf{A}$  represents the graph  $\mathcal{G}$ , we have that  $\phi(\mathbf{A}) = F(\mathcal{G})$  is differentiable if the  $f(\cdot)$  and  $AGG(\cdot)$  functions are differentiable.
The representation of graphs based on local ego-graph properties has been widely studied [22, 64, 79, 83]. Recent studies proposed algorithms for estimating  $k$ -hop-based graph structural properties under (L)DP [7, 17, 29, 30, 61, 66, 78, 81]. Our experiments use three histogram-based  $k$ -hop graph statistics: 1) triangle histogram; 2)  $k$ -HOP neighbors histogram; 3) histogram of random walks, and graph size. We use a differentiable soft histogram [77]; see Section 7.3. The histogram function  $h(\cdot)$  transforms a vector of counts into a real-valued soft histogram. We explain each graph statistic next.
Triangle histogram. A triangle histogram counts the number of nodes that participate in a given number of triangles. The descriptor function is  $h\left(\frac{1}{2}\left(\mathbf{A}^{3}\right)_{u,u}\right)$ .
$k$ -HOP neighbors histogram for  $k = 1,2,3,4$ . The  $k$ -HOP neighbor histogram counts the number of nodes that have a given number of  $k$ -HOP neighbors. The 1-HOP neighbour histogram is equivalent to the degree histogram. The  $k$ -Hop descriptor function is  $h(\sum (\min (\sum_k\mathbf{A}^k,1))$  where  $\min (\mathbf{A},1)_{u,v} = \min (\mathbf{A}_{u,v},1)$  and  $\sum (\mathbf{A})_u = \sum_v\mathbf{A}_{u,v}$ .
Random walk histogram of length  $r$  for  $r = 2,3$ . The number of walks of length  $r$  between node  $u$  and  $v$  is given by  $(\mathbf{A}^r)_{u,v}$ . The random walk histogram of length  $r$  counts the number of node pairs with a given number of walks of length  $r$  connecting them. It is defined by  $h(\mathbf{A}^r)$  [65].
Graph size. The size of a graph is its number of edges given by  $\frac{1}{2}\sum_v\sum_u\mathbf{A}_{v,u}$ .
This paper uses these four statistics in the experiments as default statistics and demonstrates that the default statistics are capable of modelling graphs with a wide range of structural characteristics; see Section 4. The default statistics are known from prior research to be generally important for graph modelling across different domains and are easy to interpret [24]. Different graph statistics are important for different applications [55]. The default statistics can be combined with other statistics of interest in a specific application.