SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hyuna Cho Minjae Jeong Sooyeon Jeon Sungsoo Ahn Won Hwa Kim POSTECH, South Korea
{hyunacho, minjaetidtid, jsuyeon, sungsoo.ahn, wonhwa}@postech.ac.kr
Successful graph generation depends on the accurate estimation of the joint distribution of graph components such as nodes and edges from training data. While recent deep neural networks have demonstrated sampling of realistic graphs together with diffusion models, however, they still suffer from oversmoothing problems which are inherited from conventional graph convolution and thus high-frequency characteristics of nodes and edges become intractable. To overcome such issues and generate graphs with high fidelity, this paper introduces a novel approach that captures the dependency between nodes and edges at multiple resolutions in the spectral space. By modeling the joint distribution of node and edge signals in a shared graph wavelet space, together with a score-based diffusion model, we propose a Wavelet Graph Diffusion Model (Wave-GD) which lets us sample synthetic graphs with real-like frequency characteristics of nodes and edges. Experimental results on four representative benchmark datasets validate the superiority of the Wave-GD over existing approaches, highlighting its potential for a wide range of applications that involve graph data.
The flexible representation of graphs lets us investigate a variety of real-world phenomena such as social networks [8, 39], traffic flows [43, 45], molecule structure [18, 19, 34, 44], and brain networks [2, 22, 28]. The rich expressive power of graphs inherits from embodying various combinations of nodes and edges within an irregular structure. However, the analysis of graph data is challenging due to the arbitrary and heterogeneous structure, as it becomes difficult to characterize robust patterns both within a graph and across different graphs for a downstream machine learning task.
The same issue arises in graph generation, where the objective is to learn the underlying graph distribution from training samples and thereby synthetic graphs with similar characteristics to the training data can be generated, i.e., sampled. For this, accurately estimating the joint distribution of the measurements from node features and graph structures is essential. Recent graph generative methods [13, 35, 37] have been successful in sampling realistic node and edge features, however, their generated samples often fail to embed the true relationships especially given in sparse connections between nodes. This problem often occurs as existing methods derive separate embeddings of nodes and edges considering them as different entities [37]. Moreover, successive layers of graph convolution [13, 35] aggravate this issue by overly averaging signals on graphs, such that local details (i.e., high-frequency characteristics) become intractable.
Key ideas. Notice that the edge signals, e.g., edge weights and connectivity, should be coherent to node signals as the edges should properly explain the relationships between the nodes. Based on this homophily assumption, to address the aforementioned issues, we propose a novel generative model that precisely captures the dependency between nodes and edges in multiple resolutions on local-to-global structures. This is realized via spectral graph wavelet transform (SGWT) [9], which
(b)
Figure 1: An example of SGWT with diverse scales that impacts the graph representation. (a) An undirected graph with an adjacency matrix  $\mathbf{A}$ , (b) Disconnectivity between nodes with dashed lines for  $\mathbf{A}$ . (c) Result from graph transform of  $\mathbf{A}$  with a band-pass kernel  $k(s) = sxe^{-s.x}$  at scale  $s = 1$  and (d) corresponding disconnectivity to (c). (e) Result from graph transform of  $\mathbf{A}$  using the same band-pass kernel at  $s = 0.1$ . (f) Corresponding disconnectivity for (e). The edge thickness denotes the magnitude of connectivity, which is controlled by a scale parameter. Compared to  $\mathbf{A}$ , the sparse connection (orange) is significantly strengthened via SGWT at  $s = 1$ , and disconnectivities (blue) are also emphasized at the same scale.
enables the filtering of graph signals at a specific level of granularity. We transform the node and edge signals into a common graph wavelet space and obtain their underlying cross-characteristics, i.e., spectral coherence, via their inner product at different scales. We expect that sparse connections between nodes (i.e., higher variation) can be captured with high fidelity as high-frequency components in the spectral space. Fig. 1 shows an example of such an effect with wavelet transforms on edges, which accentuates certain connectivity and disconnectivity over a template graph. These accentuated (dis-)connectivities at specific scales can be characterized by a generative model with multiresolution wavelet filtering. Also, we further show that the scale-specific node and edge features computed in the spectral space can be easily and jointly obtained in a spatial graph convolutional form by applying wavelet filtering to either nodes or edges for efficiency.
Proposed Framework. We leverage this spectral coherence to model the joint distribution of graph node and edge features embedded in a common graph wavelet space with a score-based diffusion model [36]. As opposed to conventional approaches [13] where the scores for the joint distribution of nodes and edges are modeled with graph multi-head attention (GMH) [1] involving graph convolutions of the raw node and edge signals, our model calculates the scores over multiple scales of nodes and edges to capture different frequency characteristics. With diffusion and reverse denoising processes in multi-resolution, our Wavelet Graph Diffusion Model (Wave-GD) accurately estimates the joint distribution of the node and edge features preserving their graph frequency characteristics.
Contributions. To this end, we summarize our main contributions as follows: 1) To obtain representative graph characteristics of different fidelity, we propose a novel concept to capture spectral coherence between nodes and edges in multiple scales in a common graph wavelet space. 2) Furthermore, we show that spectral coherence can be obtained in the graph domain as a spatial graph convolutional formulation, which allows us to perform the SGWT only on the edge signals for efficiency. 3) We propose a training scheme on the multi-resolution at which the optimal joint frequency characteristics of graph nodes and edges can be captured from training samples. The Wave-GD was tested on four representative benchmarks for graph generation which demonstrated outperforming results over state-of-the-art baselines. Extensive experiments on real-world, synthetic, and biochemical graph datasets validate the generality of the method, suggesting its potential for diverse applications with graph samples.
Deep graph generative models aim to learn the underlying distribution of node and edge signals from the training set. They can be categorized into (1) autoregressive models that construct graphs by the iterative generation of a node or an edge signal at each step or (2) one-shot models that simultaneously update all the signals at once.
Autoregressive models. Autoregressive models are widely used for graph generation since they are based on a straightforward way to iteratively update the graph structure conditioned on an incomplete graph generated so far. Prior works have parameterized the autoregressive models using recurrent neural network (RNN) [29, 41, 42], variational autoencoder (VAE) [11, 12], and normalizing flow [20, 34]. However, they suffer from high complexity since the required number of updates to generate the graph grows with the size of the graph. Furthermore, their generation is sensitive to the order of nodes being generated, i.e., autoregressive models may assign different probabilities for the construction of the same graph executed in a different way.
One-shot models. One-shot models alleviate the issues of autoregressive models by simultaneous generation of node and edge signals. This potentially reduces the complexity of generation since the number of generation steps no longer depends on the graph size. Furthermore, their generation is often permutation invariant and assigns the same probability for generating a graph regardless of the permutation. Researchers have considered parameterizing these models using generative adversarial network (GAN) [4], VAE [21], normalizing flow models [23, 44], and diffusion models [13, 25, 37].
Diffusion models. In particular, diffusion models are recently catching attention due to their extraordinary capability in learning the joint distributions of graph components such as nodes and edges. They are based on (1) defining a forward diffusion process to sequentially corrupt a graph using noise distributions and (2) training a reverse diffusion process parameterized by a graph neural network to reconstruct the original graph. Existing works mainly differ by defining the diffusion processes in the continuous space [13] or the discrete space [10, 25, 37].
3 Generating Graphs with Robust Spectral Characteristics
3.1 Preliminary: Graph Wavelet Transform
Let  $G = (\mathbf{X}, \mathbf{A})$  be an undirected graph, where  $\mathbf{X} \in \mathbb{R}^{N \times F}$  represents  $F$ -dimensional node features for  $N$  nodes and  $\mathbf{A} \in \mathbb{R}^{N \times N}$  is an adjacency matrix representing connectivity among the nodes. The graph Laplacian is defined as  $\mathbf{L} = \mathbf{D} - \mathbf{A}$ , where  $\mathbf{D}$  is a diagonal degree matrix. As the  $\mathbf{L}$  is real and positive semi-definite, it has a set of orthonormal eigenvectors  $\mathbf{U} = [u_1, u_2, \dots, u_N]$  and corresponding non-negative real eigenvalues  $\boldsymbol{\Lambda} = \text{diag}(\lambda_1, \dots, \lambda_N)$ . Decomposing this Laplacian as  $\mathbf{L} = \mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^T$ , the connectivity, complexity, and spectral properties of the graph are characterized. Specifically, its eigenvectors and eigenvalues identify the frequencies and mode of vibrations of a graph. Eigenvectors with small eigenvalues represent large connected components in a graph structure and slow-varying signals among nodes. On the other hand, eigenvectors associated with large eigenvalues indicate the sparse and disconnected signals in the graph [22, 26, 40].
To capture localized characteristics of signals on arbitrary graph structures, the spectral graph wavelet transform [9] is defined by constructing a set of graph wavelet bases  $\psi_{s} = (\psi_{s_{1}},\psi_{s_{2}},\dots,\psi_{s_{J}})$  with  $J$  scales where  $\psi_s = \mathbf{U}k(s\pmb{\Lambda})\mathbf{U}^T$ . The  $\psi_{s}$  is a realization of a kernel function  $k(\cdot)$  in the spectral domain localized with  $\delta_{n}$  in the graph space, which captures the local characteristics of the graph at each node. The scale  $s$  selects a specific bandwidth in the spectral space which corresponds to the range of locality in the original graph space. The choice of wavelet basis, i.e., mother wavelet, depends on the shape of  $k(\cdot)$ , which may vary depending on a target task such as smoothing or band-pass filtering. In this work, we used both types of kernel functions: (1) a band-pass kernel  $k(s) = sxe^{-sx}$  to capture both low and high frequency of graphs and (2) a low-pass kernel  $k(s) = e^{-sx}$  to capture cluster-like and smoothed features from graphs.
Using the bases  $\psi_s$ , graph wavelet transform projects a signal  $x$  (e.g., node feature or connectivity) from the graph domain into the spectral domain as
$$
W _ {x} (s) = <   \psi_ {s}, x > = \psi_ {s} x, \tag {1}
$$
which yields a wavelet coefficient  $W_{x}(s)$ . This wavelet coefficient is a scale-filtered signal defined in the wavelet space. When admissibility condition on  $k(\cdot)$  is satisfied [24], the inverse transform completely reconstructs the signal  $x$  by projecting  $W_{x}(s)$  back to the graph domain as follows:
$$
x = \frac {1}{C _ {k}} \int_ {0} ^ {\infty} \psi_ {s} \cdot W _ {x} (s) \frac {d s}{s} \tag {2}
$$
with an admissibility constant  $C_k = \int_0^\infty \frac{k(x)^2}{x} dx < \infty$ .
3.2 Adaptive Multi-Resolution Representation of Nodes and Edges
To capture both fine and coarse graph features from an irregular graph structure, the filtering operation is performed over multiple scales. The multi-resolution representations of graph nodes and edges are derived by decomposing the graph signals  $\mathbf{X}$  and  $\mathbf{A}$  with scales  $\{s_i\}_{i=1}^J$  using (1), which result in  $\{W_{\mathbf{X}}(s_i)\}_{i=1}^J$  and  $\{W_{\mathbf{A}}(s_i)\}_{i=1}^J$  in the spectral domain. These are wavelet coefficients that contain different levels of local and global details of graph node and edge signals in a multi-resolution fashion.
To obtain the multi-resolution representations of these graph signals in the graph space, the inverse transform in the Eq. 2 is performed on the filtered coefficients using the bases  $\psi_s$  at the same scales.
(a) Diffusion process scheme of Wave-GD.
(b) Overview of the model architecture.
Figure 2: (a) Schematic diagram of Wave-GD with multi-resolution diffusion. Note that the scale-filtering is applied to the  $\mathbf{A}_0$ , not to the  $\bar{\mathbf{A}}_t$ . (b) Overview of Wave-GD that jointly estimates partial scores of node features  $\mathbf{X}$  and edges  $\mathbf{A}$  along with scale-filtered edges  $\mathbf{A}^s$ . We first perform multi-resolution filtering on the given edges  $\mathbf{A}_0$  at  $t = 0$  using graph wavelet basis  $\psi_s$  with trainable scale  $s$ . Forward diffusion processes smoothly transform  $\mathbf{X}$ ,  $\mathbf{A}$ , and  $\mathbf{A}^s$  into noises in parallel and they are reconstructed by solving reverse-time SDEs. During the reconstruction, the knowledge across multi-resolution is shared in the score-based model  $\mathcal{A}_{\phi,t}$ . Also, the information in separate edge and node components is effectively entangled by obtaining their spectral coherence at various resolutions.
As the Eq. 2 is the superposition of multi-resolution representation of  $x$  over scales at  $s \in [0, \infty)$ , it allows us to define a scale-filtered signal in the graph space at specific  $s$  as
$$
x ^ {s} = \psi_ {s} \cdot W _ {x} (s) = \mathbf {U} k ^ {2} (s \boldsymbol {\Lambda}) \mathbf {U} ^ {T} x, \tag {3}
$$
where  $k^2(s\Lambda) = \text{diag}(k^2(s\lambda_1), \ldots, k^2(s\lambda_N))$  is a diagonal matrix [14, 22]. Therefore, the  $\mathbf{X}$  and  $\mathbf{A}$  at multiple resolutions in the spatial domain are defined as  $\{\mathbf{X}^{s_i}\}_{i=1}^J = \{\mathbf{U}k^2(s_i\Lambda)\mathbf{U}^T\mathbf{X}\}_{i=1}^J$  and  $\{\mathbf{A}^{s_i}\}_{i=1}^J = \{\mathbf{U}k^2(s_i\Lambda)\mathbf{U}^T\mathbf{A}\}_{i=1}^J$ , respectively. Within our proposed framework, we make these scales trainable such that the local-to-global characteristics can be obtained adaptively.
3.3 Refining Spectral Coherence between Nodes and Edges in Multi-resolution Fashion
As the graph features  $\mathbf{X}$  and  $\mathbf{A}$  should be coherent, i.e., connectivities should explain the relationship among features from different nodes, we hypothesize that they should share similarities even when observed from different resolutions. To capture the key components that are shared on both edges and nodes at multi-resolution, we measure the underlying cross-characteristics between nodes and edges in each scale in the spectral space. We quantify their coherence as a dot product as  $W_{\mathbf{A}}(s)\cdot W_{\mathbf{X}}(s)$  and use this spectral coherence as a feature that represents a graph at a specific resolution  $s$ , which is used to estimate the score function in multiple resolutions later in Section 3.5.
Lemma 1. Let  $G$  be an undirected graph, with a node signal  $\mathbf{X}$  and a symmetric adjacency matrix  $\mathbf{A}$ . Given wavelet coefficients  $W_{\mathbf{A}}(s)$  and  $W_{\mathbf{X}}(s)$ , which are spectral representations of  $\mathbf{X}$  and  $\mathbf{A}$  at a specific scale  $s$ , a dot product of these wavelet coefficients is equivalent to the product between the scale-filtered edge signals and the node signals in the graph space, i.e.,  $W_{\mathbf{A}}(s) \cdot W_{\mathbf{X}}(s) = \mathbf{A}^{s}\mathbf{X}$ .
By Lemma 1, we can efficiently extract multi-resolution coherence by applying the filtering only on the edges. Using the  $J$  number of  $\mathbf{A}^s$  along with the given data  $G = (\mathbf{A},\mathbf{X})$ , we introduce our simple yet effective graph generative framework through the stochastic differential equations (SDEs) in Sec. 3.4 and explain how Lemma 1 is used in the generative model in Sec. 3.5. The proof of Lemma 1 is given in Appendix A.
3.4 Wave-GD: Wavelet Graph Diffusion Model via Multi-resolution
Now we describe our graph generative model which captures the dependency of node and edge on both local and global structures with adaptive spectral filtering. The overall description of Wave-GD is given in Fig. 2, which demonstrates (a) multi-resolution diffusion strategy and (b) overall model architecture. In particular, we enhance the existing diffusion models for graphs [13] using the spectral coherence introduced in Lemma 1. The building blocks of our model are as follows: 1) the forward
diffusion to corrupt the graph distribution in a continuous domain, 2) the reverse diffusion to recover the original graph, and 3) the training objective for our model.
Multi-resolution forward diffusion process. Building upon the work of GDSS [13], we propose a new diffusion process that sequentially corrupts a graph structure at different levels of resolution based on the Ito SDE [32]. As the forward diffusion process is modeled by a SDE which describes a process of random variables,  $\{\mathbf{X}_t,\{\mathbf{A}_t^{s_i}\}_{i = 0}^J\}$  with  $\mathbf{A}_t^{s_0} = \mathbf{A}_t$  at timestep  $t\in [0,T]$  are diffused as follows:
$$
d \mathbf {X} _ {t} = \mathbf {f} _ {\mathbf {X}} \left(\mathbf {X} _ {t}, t\right) d t + \sigma_ {\mathbf {X}, t} d \mathbf {w} _ {\mathbf {X}},
$$
$$
d \mathbf {A} _ {t} ^ {s _ {i}} = \mathbf {f} _ {\mathbf {A} ^ {s _ {i}}} \left(\mathbf {A} _ {t} ^ {s _ {i}}, t\right) d t + \sigma_ {\mathbf {A} ^ {s _ {i}}, t} d \mathbf {w} _ {\mathbf {A} ^ {s _ {i}}}, \tag {4}
$$
where  $\mathbf{f}_{\mathbf{X}}: \mathbb{R}^{N\times F}\rightarrow \mathbb{R}^{N\times F}$  and  $\mathbf{f}_{\mathbf{A}^{s_i}}:\mathbb{R}^{N\times N}\to \mathbb{R}^{N\times N}$  are drift functions,  $\sigma_{\mathbf{X},t}$  and  $\sigma_{\mathbf{A}^{s_i},t}$  are scalar diffusion coefficients, and  $\mathbf{w}_{\mathbf{X}}$  and  $\mathbf{w}_{\mathbf{A}^{s_i}}$  are standard Wiener processes for  $\mathbf{X}_t$  and  $\{\mathbf{A}_t^{s_i}\}_{i = 0}^J$ , respectively. As this diffusion process is simulated at varying levels of granularity in a multi-resolution scheme, it is likely that certain frequency characteristics are accentuated and the highlighted information will be preserved from corruption. In other words, at an arbitrary timestep  $t$ , the scale-filtered edges  $\{\mathbf{A}_t^{s_i}\}_{i = 1}^J$  are highly likely to contain richer information of edges for score estimation than the given  $\mathbf{A}_t$  as their edge signals are adaptively controlled and highlighted by trainable scales.
Learning the reverse diffusion process. Let  $G_{t} = (\mathbf{X}_{t},\mathbf{A}_{t})$  be a given graph at timestep  $t$  and  $\{G_t^{s_i} = (\mathbf{X}_t,\mathbf{A}_t^{s_i})\}_{i = 0}^J$  be a set of graphs for multiple scales at the same timestep. Given the multi-resolution forward diffusion process, we generate graphs from solving the associated SDE backward in time. To be specific, the reverse diffusion process can be formulated as follows:
$$
d \mathbf {X} _ {t} = \left[ \mathbf {f} _ {\mathbf {X}} \left(\mathbf {X} _ {t}, t\right) - \sigma_ {\mathbf {X}, t} ^ {2} \nabla_ {\mathbf {X} _ {t}} \log p _ {t} (G _ {t}) \right] d \bar {t} + \sigma_ {\mathbf {X}, t} d \bar {\mathbf {w}} _ {\mathbf {X}},
$$
$$
d \mathbf {A} _ {t} ^ {s _ {i}} = \left[ \mathbf {f} _ {\mathbf {A} ^ {s _ {i}}} \left(\mathbf {A} _ {t} ^ {s _ {i}}, t\right) - \sigma_ {\mathbf {A} ^ {s _ {i}}, t} ^ {2} \nabla_ {\mathbf {A} _ {t} ^ {s _ {i}}} \log p _ {t} \left(G _ {t} ^ {s _ {i}}\right) \right] d \bar {t} + \sigma_ {\mathbf {A} ^ {s _ {i}}, t} d \bar {\mathbf {w}} _ {\mathbf {A} ^ {s _ {i}}}, \tag {5}
$$
where  $p_t(G_t)$  and  $p_t(G_t^{s_i})$  denote the distribution of the forward diffusion process evaluated at time  $t$ . This reverse-time SDE recovers the original data distribution  $p_0(\mathbf{X}_0, \{\mathbf{A}_0^{s_i}\}_{i=0}^J)$  with the standard reverse-time Wiener processes  $\bar{\mathbf{w}}_{\mathbf{X}}$  and  $\bar{\mathbf{w}}_{\mathbf{A}^{s_i}}$  in which the direction of time is reversed as  $d\bar{t}$ .
However, since the score functions of the joint distribution, i.e.,  $\nabla_{\mathbf{X}_t}\log p_t(G_t)$  and  $\nabla_{\mathbf{A}_t^{s_i}}\log p_t(G_t^{s_i})$ , are unknown, we train a neural network to estimate them using the denoising score matching loss [38]:
$$
L _ {\mathbf {X}} = \min  _ {\theta} \mathbb {E} _ {t} \left[ \lambda_ {\mathbf {X}} (t) \mathbb {E} _ {G _ {0}, G _ {t} \mid G _ {0}} \left[ \| \mathcal {X} _ {\theta , t} \left(G _ {t}\right) - \nabla_ {\mathbf {X} _ {t}} \log p _ {0 t} \left(\mathbf {X} _ {t} \mid \mathbf {X} _ {0}\right) \| _ {2} ^ {2} \right] \right], \tag {6}
$$
$$
\begin{array}{l} L _ {\mathbf {A}} = \min  _ {\phi , \mathbf {s}} \mathbb {E} _ {t} \left[ \lambda_ {\mathbf {A}} (t) \mathbb {E} _ {G _ {0}, G _ {t} | G _ {0}} [ \| \mathcal {A} _ {\phi , t} (G _ {t}) - \nabla_ {\mathbf {A} _ {t}} \log p _ {0 t} (\mathbf {A} _ {t} | \mathbf {A} _ {0}) \| _ {2} ^ {2} ] \right. \\ \left. + \lambda_ {\mathbf {A} ^ {s _ {i}}} (t) \sum_ {i = 1} ^ {J} \mathbb {E} _ {G _ {0} ^ {s _ {i}}, G _ {t} ^ {s _ {i}} \mid G _ {0} ^ {s _ {i}}} \left[ \| \mathcal {A} _ {\phi , t} \left(G _ {t} ^ {s _ {i}}\right) - \nabla_ {\mathbf {A} _ {t} ^ {s _ {i}}} \log p _ {0 t} \left(\mathbf {A} _ {t} ^ {s _ {i}} \mid \mathbf {A} _ {0} ^ {s _ {i}}\right) \| _ {2} ^ {2} \right] \right], \tag {7} \\ \end{array}
$$
where  $p_{0t}(\cdot)$  denotes the transition distribution from  $p_0$  to  $p_t$ ,  $\mathcal{X}_{\theta ,t},\mathcal{A}_{\phi ,t}$  are score-based models which are graph neural networks trained to approximate the true score functions. The  $\lambda_{\mathbf{X}}(t),\lambda_{\mathbf{A}}(t),\lambda_{\mathbf{A}^{s_i}}(t)$  are time-varying scaling factors for individual loss terms and are defined as in [36].
Both our method and GDSS perform denoising score-matching to the partial scores of  $X_{t}$  and  $A_{t}$ . However, our method additionally models the joint probability space of  $X_{t}$  and  $A_{t}^{s_{i}}$  via SDEs. This is realized by the loss in Eq. 7, and this operation allows a model to flexibly estimate the complex dependency between nodes and edges with multi-resolution SGWT. Also, note that the scales of SWGT  $\{s_i\}_{i = 1}^J$  in Eq. 7 are trainable so that multi-level granularities that characterize a graph distribution can be adaptively captured during training.
Remark. While we solve the reverse SDEs for  $\{\mathbf{X}_t, \{\mathbf{A}_t^{s_i}\}_{i=0}^J\}$  in the training, we only need to solve the reverse diffusion processes (5) only for the node signal  $\mathbf{X}$  and the non-filtered graph  $\mathbf{A} = \mathbf{A}^{s_0}$  in the sampling phase. In other words, we do not generate  $\{\mathbf{A}^{s_i}\}_{i=1}^J$  in the sampling phase. Also note that the parameters of the networks  $\mathcal{A}_{\phi,t}$  are shared across different scales so that they can learn generalizable representations over different resolutions. As the shared parameters in the score function do not overfit to a specific scale, it also promotes stable training.
3.5 Score-based Joint Density Estimation with Multi-Resolution Coherence