SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, we describe the score-based models  $\mathcal{X}_{\theta ,t}$  and  $\mathcal{A}_{\phi ,t}$  to better capture the spectral coherence over different resolutions. In particular, we revise the graph multi-head attention (GMH) [1] layer to better incorporate the spectral coherence between the edge and the node attributes. In the original GMH layer, node and edge attributes were encoded into query  $Q$  ,key  $K$  ,and value  $V$  as follows:
$$
Q = \mathbf {A} _ {t} \mathbf {X} _ {t} W _ {Q}, \quad K = \mathbf {A} _ {t} \mathbf {X} _ {t} W _ {K}, \quad V = \mathbf {A} _ {t} \mathbf {X} _ {t} W _ {V}, \tag {8}
$$
where  $W_{Q}, W_{K}, W_{V}$  are trainable weights. This layer is repeated for each  $\{s_i\}_{i=1}^J$  as follows:
$$
Q ^ {s _ {i}} = \mathbf {A} _ {t} ^ {s _ {i}} \mathbf {X} _ {t} W _ {Q}, \quad K ^ {s _ {i}} = \mathbf {A} _ {t} ^ {s _ {i}} \mathbf {X} _ {t} W _ {K}, \quad V ^ {s _ {i}} = \mathbf {A} _ {t} ^ {s _ {i}} \mathbf {X} _ {t} W _ {V}, \tag {9}
$$
which incorporate spectral coherence at a specific scale  $s_i$  as given in Lemma 1. Using  $Q^{s_i}$ ,  $K^{s_i}$ ,  $V^{s_i}$  as the new query, key, and value for the GMH layer, one can incorporate the spectral coherence when training  $W$ 's. Consequently, the score-based models consist of multiple GMH layers followed by a multi-layer perceptron (MLP). We used the original GMH layer for the parameterization of  $\mathcal{X}_{\theta,t}$ , and the modified GMH layer is used for  $\mathcal{A}_{\phi,t}$ . Note that this parameterization preserves the permutation invariance of the original GMH layer, hence our score-based models are also permutation invariant.
In this section, we quantitatively and qualitatively evaluate our method in comparison to various recent graph generative methods on four benchmark datasets. Ablation studies are also introduced to empirically analyze the roles of individual components within Wave-GD.
We evaluated Wave-GD on four public datasets with varying sizes and characteristics, demonstrating its robustness and generalizability to generate high-quality graphs for diverse graph domains. Ego-small [33] consists of 200 small real sub-graphs from the CiteSeer network dataset with  $4 \leq N \leq 18$ . Community-small consists of 100 randomly generated synthetic graphs with  $12 \leq N \leq 20$ . The graphs are constructed by two equal-sized communities and each community is generated by E-R model [5] and  $0.05N$  inter-community edges are added with uniform probability as in previous works [17, 25]. Grid consists of randomly generated 100 standard 2D grid graphs with  $100 \leq N \leq 400$ . As all nodes are arranged in a regular lattice, the maximum number of edges per node is 4. QM9 [31] is a molecular dataset with 133,885 molecules that are represented by attributed graphs with  $1 \leq N \leq 9$ , four node types, and three edge types.
For all datasets, we used  $80\%$  of the whole data as a train set and the rest  $20\%$  as a test set with the same split as in [13, 42]. As in [13], we sampled 10,000 molecules from the QM9 dataset, and for the remaining generic datasets, we sampled an equal number of data as the number of test data in each respective dataset. Also, the predictor-corrector sampler (PC-sampling) proposed in [36] was used to sample data with 1000 predictor and 1000 corrector steps.
We set two wavelet kernels  $k(s)$ : one is a low-pass filter  $k(s) = e^{-sx}$  that captures signals in the low-frequency and the other  $k(s) = sxe^{-sx}$  is a band-pass filter which is 0 at the origin. To obtain the combined effect of the low-pass and band-pass filters using a total of  $J$  scales, edges at one scale are filtered using the low-pass filter. For the remaining  $J - 1$  scales, the band-pass filter is applied. This approach enables selective extraction and processing of different frequency components across the spectral domain. The number of scales  $J$  was set to 11, 6, 4, and 6 for the Ego-small, Community-small, Grid, and QM9 datasets, respectively. More detailed experimental settings are given in Appendix B.
4.3 Experiment on Generic Graphs
To assess the versatility of Wave-GD, we used both real-world and synthetic datasets with varying sizes and characteristics. Experimental results on these diverse generic datasets demonstrate the robustness of Wave-GD in sampling graphs of various characteristics.
Baselines and metrics. For experiments on generic graph datasets (i.e., Ego-small, Community-small, and Grid), we compared the performance of our proposed method with the following autoregressive and one-shot graph generation methods: DeepGMG [16], GraphRNN [42], GraphAF [34],
Table 1: Generation results on generic graph datasets. MMD statistics between sampled graphs and test data are given, where smaller values indicate that the generated samples are more similar to the test set. The baseline results are from [13, 20, 25]. Mean from three replicates are reported, and their standard deviations are reported in Appendix C.
Method	Ego-small	Community-small	Grid
Degree	Cluster	Orbit	Avg.	Degree	Cluster	Orbit	Avg.	Degree	Cluster	Orbit	Avg.
Autoreg.	DeepGMG [16]	0.040	0.100	0.020	0.053	0.220	0.950	0.400	0.523	-	-	-	-
GraphRNN [42]	0.090	0.220	0.003	0.104	0.080	0.120	0.040	0.080	0.064	0.043	0.021	0.043
GraphAF [34]	0.030	0.110	0.001	0.047	0.180	0.200	0.020	0.133	-	-	-	-
GraphDF [20]	0.040	0.130	0.010	0.060	0.060	0.120	0.030	0.070	-	-	-	-
One-shot	GraphVAE [35]	0.130	0.170	0.050	0.117	0.350	0.980	0.540	0.623	1.619	0.0	0.919	0.846
GNF [17]	0.030	0.100	0.001	0.044	0.200	0.200	0.110	0.170	-	-	-	-
EDP-GNN [25]	0.052	0.093	0.007	0.051	0.053	0.144	0.026	0.074	0.455	0.238	0.328	0.340
GDSS [13]	0.021	0.024	0.007	0.017	0.045	0.086	0.007	0.046	0.111	0.005	0.070	0.062
DiGress [37]†	0.017	0.021	0.010	0.016	0.028	0.115	0.009	0.050	-	-	-	-
Wave-GD (Ours)	0.012	0.010	0.005	0.009	0.007	0.058	0.002	0.022	0.144	0.004	0.021	0.056
- results not reported in the original manuscripts, †: results obtained with the official public code.
Table 2: Converged scales of generic graph datasets.
Dataset	J	low-pass	band-pass
Ego-small	11	45.4	46.8	46.6	43.7	40.6	38.1	37.0	31.3	29.0	25.6	20.0
Community-small	6	32.8	44.7	31.0	27.5	21.4	16.5	-	-	-	-	-
Grid	4	35.8	25.4	18.1	12.0	-	-	-	-	-	-	-
Table 3: Generation results on generic graph datasets using a larger number of samples. We compared 1024 generated samples with the given test set and evaluated their MMDs.
Method	Ego-small	Community-small
Degree	Cluster	Orbit	Avg.	Degree	Cluster	Orbit	Avg.
GraphRNN [42]	0.040	0.050	0.060	0.050	0.030	0.010	0.010	0.017
GNF [17]	0.010	0.030	0.001	0.014	0.120	0.150	0.020	0.097
EDP-GNN [25]	0.010	0.025	0.003	0.013	0.006	0.127	0.018	0.050
GDSS [13]	0.023	0.020	0.005	0.016	0.029	0.068	0.004	0.034
Wave-GD (Ours)	0.010	0.018	0.005	0.011	0.016	0.077	0.001	0.031
GraphDF [20], GraphVAE [35], GNF [17], EDP-GNN [25], GDSS [13] and DiGress [37]. To evaluate the generation quality of the generated graphs, we used Maximum Mean Discrepancy (MMD) [7] to statistically measure similarities between probability distributions of sampled data and the real test data. Following prior works [13,42], we compared distributions of three different graph statistics-degree, clustering coefficient, and orbit counts-to compute MMD between generated and test graphs.
Main results. Table 1 shows the comparisons of degree, clustering, orbit, and their averaged MMDs between the graphs generated from baselines and our method. The results show that our model significantly outperforms all baseline models for all metrics on Ego-small and Community-small datasets. Specifically, on the Ego-small dataset, Wave-GD demonstrated far smaller average MMDs by a margin of 0.108 over GraphVAE and at least 0.007 lower than DiGress. For the Community-small data, our method exhibited up to 0.601 lower MMD compared to GraphVAE and at least 0.024 better than GDSS. For the Grid dataset, our method outperformed all baselines in terms of orbit MMD and outperformed all one-shot baselines for the averaged MMD. Note that, while one-shot models struggle to generate large graphs due to their intricate and complex structures, our method achieved comparable results to GraphRNN on large grid graphs (with  $100 \leq N \leq 400$ ). This highlights the effectiveness of our approach in learning graphs with varying sizes and structures through multi-resolution learning.
Table 2 shows converged scales for different datasets. Interestingly, the scales of the large Grid dataset (with  $100 \leq N \leq 400$  nodes) were generally smaller compared to the Ego-small ( $4 \leq N \leq 18$ ) and Community-small ( $12 \leq N \leq 20$ ) datasets. Specifically, the converged scales of Ego-small ranged within [20, 47] and the scales of Grid ranged within [12, 36]. Note that the smaller scales capture local graph features with higher frequencies. Therefore, the results demonstrate that capturing local and detailed graph representations is more critical when dealing with complex and large graphs, in
Figure 3: Visualization of training samples and generated graphs on Ego-small (top), Community-small (middle), and QM9 (bottom). The generated molecules from the QM9 dataset (bottom panel) are represented with the pairwise Tanimoto similarity based on the Morgan fingerprints [15], which show the one-to-one similarity to the real training samples at the same location in the upper panel.
(a) Degree
(b) Clustering
Figure 4: Comparison of the MMDs between GDSS and Wave-GD on the Ego-small. The sample quality of Wave-GD is stable and converges faster for a) Degree, b) Clustering, c) Orbit, and d) their average.
(c) Orbit
(d) Average
comparison to smaller graphs. In other words, for the Ego-small and Community-small datasets, features in relatively low frequency (e.g., cluster-related features) were captured with larger scales.
Experiments with larger sample size. To provide a more definitive assessment of the generated sample quality, we conducted additional experiments by generating a larger number of samples compared to the test data, as in previous works [13, 17, 25]. For this, we sampled 1024 graphs for evaluating the MMD metrics for the Ego-small and Community-small datasets. As shown in Table 3, the lower MMDs of Wave-GD demonstrate superior performance to the baselines on Ego-small. For the Community-small dataset, GraphRNN performed better than ours on the averaged MMD, however, our method still surpasses recent score-based models [13, 25] and normalizing flow model [17] with at most 0.066 margin. Along with these quantitative results, we present the actual visualization of the generated samples from Wave-GD in Fig. 3, which are highly similar to the training samples.
Stability analysis. In addition to MMDs, we present a stability analysis of Wave-GD in Fig. 4. The figure presents a comparison of the averaged MMDs between GDSS and our method on the Ego-small dataset. Specifically, between 2000-5000 epochs, while the difference between the maximum and minimum values of the averaged MMD of GDSS is 0.016, Wave-GD shows better stability with a difference of  $\sim 0.0018$  at the same epochs, indicating that our method can much more robustly generate high-quality samples.
4.4 Experiment on Molecular Graphs
In addition to the generic graphs, we evaluated Wave-GD on a molecular benchmark dataset, which contains complex chemical bonds between atoms (i.e., nodes) and edges. By evaluating Wave-GD on
Table 4: Generation results on QM9 molecular dataset. The baseline results are taken from [13, 20, 34] and all results are presented without validity correction. We report the mean of three runs and their standard deviations. The best performance is marked in bold and the second-best result is indicated by an underline.
Method	Validity (%) ↑	Novelty (%) ↑	Uniq. (%) ↑	Avg. (%) ↑	NSPDK ↓	FCD ↓	time (s) ↓
AR	GraphAF [34]	67	88.83	94.51	83.45	0.020 ± 3e-3	5.27 ± .4	2.52e3
GraphDF [20]	82.67	98.10	97.62	92.80	0.063 ± 1e-3	10.82 ± .0	5.35e4
One-shot	MoFlow [44]	91.36 ± 1.2	94.72 ± .8	98.65 ± .6	94.91	0.017 ± 3e-3	4.47 ± .6	4.60e0
EDP-GNN [25]	47.52 ± 3.6	86.58 ± 1.9	99.25 ± .1	77.78	0.005 ± 1e-3	2.68 ± .2	4.40e3
GraphEBM [18]	8.22 ± 2.2	97.01 ± .2	97.90 ± .1	67.71	0.030 ± 4e-3	6.14 ± .4	3.71e1
GDSS [13]	95.72 ± 1.9	86.27 ± 2.3	98.46 ± .6	93.48	0.003 ± .0	2.90 ± .3	1.14e2
DiGress [37]†	99.37 ± .0	35.31 ± .0	96.74 ± .0	77.14	1.4e-4 ± .0	0.07 ± .0	6.14e2
Wave-GD (Ours)	96.95 ± .3	89.79 ± .5	98.87 ± .2	95.20	0.003 ± .0	3.85 ± .1	1.21e2
$\dagger$  : results obtained with the official public code.
the molecular graph generation task, we validated its ability to capture the complex dependencies between nodes and edges. Notably, Wave-GD successfully generated plausible graphs while adhering to the chemical valency rule, ensuring the realistic representation of molecular structures.
Baselines and evaluation metrics. In this experiment, we compared the results from seven baselines and Wave-GD on a real molecular QM9 dataset. Among the baselines, two are autoregressive flow-based models, namely GraphAF [34] and GraphDF [20], which employ discrete latent variables. Five others are one-shot generative models, which include MoFlow [44], GraphEBM [18], EDP-GNN [25], GDSS [13], and DiGress [37]. Following the standard procedure as in [20, 34], the molecule data were kekulized by the RDKit library [15] and hydrogen atoms were removed.
We evaluated the quality of generated molecular samples with six metrics as in [13]: 1) Validity is the fraction of valid molecules that hold chemical rules. In our experiment, we did not apply any valency correction or edge resampling. 2) Novelty measures the proportion of valid molecules that are not present in the training set. 3) Uniqueness is the proportion of generated molecules that are not duplicated. 4) Neighborhood subgraph pairwise distance kernel (NSPDK) [3] is the MMD between the generated molecules and test molecules which considers both the nodes and edges for evaluation. 5) Fréchet ChemNet Distance (FCD) [30] evaluates the distance between distributions of the training and generated sets using the activations of the ChemNet layer. 6) Time quantifies the duration required to generate 10,000 molecules in the RDKit molecule format. In addition to these metrics, we also report the average values of validity, novelty, and uniqueness to present a unified evaluation over various aspects of the generated graphs.
Results. As shown in Table 4, our model outperformed all baselines on the average of validity, novelty, and uniqueness and showed the second-best results on validity, uniqueness, and NSPDK. To be specific, our method showed at most  $27.49\% p$  gain over GraphEBM and  $18.06\% p$  margin over DiGress in terms of the averaged statistic. Note that, DiGress performed better than our model in NSPDK and FCD, however, its lower novelty  $(35.31\%)$  inherently leads to lower FCD and NSPDK scores. As the molecular generation task aims to create previously unseen molecular structures, novelty may be a more critical measure on which Wave-GD performed better.
In general, Wave-GD outperformed most of the baselines, excelling not only in generation quality but also in generation speed. It was faster than all autoregressive models, showing  $442\times$  speed up compared to GraphDF and  $21\times$  speed up compared to GraphAF. Moreover, our method requires a shorter generation time than discrete-time diffusion models. For example, our method was  $36\times$  faster than EDP-GNN and  $5\times$  faster than DiGress. These results demonstrate that employing a continuous-time diffusion process for converting graphs into noise and vice versa is considerably more efficient compared to the discrete-step noise perturbation utilized in EDP-GNN and DiGress.
Ablation study on oversmoothing. In Fig. 5, we present the effect of the number of GMH layers (i.e., depth  $D$ ) on the quality of generated samples. With larger  $D$ , while the sample quality of GDSS is decreased with high averaged MMDs (blue line) due to the oversmoothing, the average MMD of Wave-GD (yellow line) is consistently low representing high fidelity in the generated samples. This is because Wave-GD captures diverse graph frequency characteristics by adaptive multi-resolution filtering, which allows the model to preserve fine-grained details even in deeper layers.
Community-small	J	Deg. ↓	Clus. ↓	Orb. ↓	Avg. ↓
3	0.011	0.072	0.003	0.028
4	0.017	0.070	0.007	0.031
5	0.006	0.065	0.002	0.024
6	0.007	0.058	0.002	0.022
7	0.005	0.069	0.009	0.027
QM9	J	Val. ↑	Uniq. ↑	Nov. ↑	Avg. ↑
4	97.73	79.21	92.68	89.87
5	96.82	88.68	99.07	94.85
6	96.70	88.95	98.91	94.85
7	96.36	88.10	98.82	94.42
Table 5: Ablation studies on the number of scales.
Figure 5: Comparison between Wave-GD (yellow line) and GDSS (blue line) for GMH depths  $D$  on Community-small. Bar graphs are MMDs of Wave-GD.
Ablation study on the number of scales. In Table 5, we report degree, cluster, orbit MMDs, and their averaged value on the Community-small by changing the numbers of scales  $J$ . Also, validity, uniqueness, novelty, and their averaged value are presented on the QM9 dataset for multiple  $J$ 's. All experiments were replicated three times and the averaged results are reported. We observed that the averaged MMD slightly depends on the number of scales, however, the overall quality of generated samples is still better than existing baselines in most settings.
Although Wave-GD is robust to varying numbers of scales as shown in Table 5, the choice of  $J$  differs across datasets which should be carefully chosen by a user. Also, as Wave-GD performs multi-resolution diffusion, it requires larger resources than conventional models during training. However, the diffusion processes are performed in parallel so there is no overhead in time. Moreover, as sampling is performed without multi-resolution filtering, the generation process is still quite efficient.
In this paper, we proposed Wave-GD which leverages spectral dependencies between node and edge signals to better characterize their joint distributions via a score-based diffusion model. By capturing their multi-resolution coherence, the model is able to generate graphs of high-fidelity preserving frequency characteristics of the graphs from the training samples. Extensive validation on various datasets and superior performance of Wave-GD highlights its significant potential for various application domains for graph modeling and generation.
Acknowledgement. This research was supported by NRF-2022R1A2C2092336 (40%), IITP-2022-0-00290 (40%), IITP-2019-0-01906 (AI Graduate Program at POSTECH, 10%), IITP-2022-2020-0-01461 (ITRC, 10%) from South Korea, and NSF IIS CRII 1948510 from the U.S.
[1] Baek, J., Kang, M., et al.: Accurate learning of graph representations with graph multiset pooling. International Conference on Learning Representations (2021)
[2] Cho, H., Wu, G., et al.: Mixing temporal graphs with MLP for longitudinal brain connectome analysis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 776-786. Springer (2023)
[3] Costa, F., De Grave, K.: Fast neighborhood subgraph pairwise distance kernel. In: International Conference on Machine Learning. pp. 255-262 (2010)
[4] De Cao, N., Kipf, T.: MolGAN: An implicit generative model for small molecular graphs. arXiv preprint arXiv:1805.11973 (2018)
[5] Erdős, P., Rényi, A.: On random graphs i. Publicationes Mathematicae (Debrecen) 6, 290-297 (1959)
[6] Grenander, U., Miller, M.I.: Representations of knowledge in complex systems. Journal of the Royal Statistical Society: Series B (Methodological) 56(4), 549-581 (1994)
[30] Preuer, K., Renz, P., et al.: Fréchet ChemNet distance: a metric for generative models for molecules in drug discovery. Journal of Chemical Information and Modeling 58(9), 1736-1741 (2018)
[31] Ramakrishnan, R., Dral, P.O., et al.: Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data 1(1), 1-7 (2014)
[32] Rogers, L.C.G., Williams, D.: Diffusions, Markov processes and martingales: Volume 2, Itô calculus, vol. 2. Cambridge University Press (2000)
[33] Sen, P., Namata, G., et al.: Collective classification in network data. AI Magazine 29(3), 93-93 (2008)
[34] Shi, C., Xu, M., et al.: GraphAF: a flow-based autoregressive model for molecular graph generation. International Conference on Learning Representations (2020)
[35] Simonovsky, M., Komodakis, N.: GraphVAE: Towards generation of small graphs using variational autoencoders. In: International Conference on Artificial Neural Networks. pp. 412-422. Springer (2018)
[36] Song, Y., Sohl-Dickstein, J., et al.: Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations (2021)
[37] Vignac, C., Krawczuk, I., et al.: DiGress: Discrete denoising diffusion for graph generation. International Conference on Learning Representations (2023)
[38] Vincent, P.: A connection between score matching and denoising autoencoders. Neural Computation 23(7), 1661-1674 (2011)
[39] Wang, H., Wang, J., et al.: GraphGAN: Graph representation learning with generative adversarial nets. In: Association for the Advancement of Artificial Intelligence. vol. 32 (2018)
[40] Xu, B., Shen, H., et al.: Graph wavelet neural network. International Conference on Machine Learning (2019)
[41] You, J., Liu, B., et al.: Graph convolutional policy network for goal-directed molecular graph generation. Advances in Neural Information Processing Systems 31 (2018)
[42] You, J., Ying, R., et al.: GraphRNN: Generating realistic graphs with deep auto-regressive models. In: International Conference on Machine Learning. pp. 5708-5717. PMLR (2018)
[43] Yu, J.J.Q., Gu, J.: Real-time traffic speed estimation with graph convolutional generative autoencoder. IEEE Transactions on Intelligent Transportation Systems 20(10), 3940-3951 (2019)
[44] Zang, C., Wang, F.: MoFlow: an invertible flow model for generating molecular graphs. In: ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 617-626 (2020)
[45] Zhang, Q., Chang, J., et al.: Spatio-temporal graph structure learning for traffic forecasting. In: Association for the Advancement of Artificial Intelligence. vol. 34, pp. 1177-1185 (2020)
In this appendix, we present 1) the proof of Lemma 1 given in the main paper, 2) additional implementation details of Wave-GD, 3) standard deviations of MMDs for generic datasets, 4) additional ablation studies on the number of scales on Ego-small and Grid datasets, 5) holistic architecture of two score-based models  $\mathcal{X}_{\theta}$  and  $\mathcal{A}_{\phi}$ , and 6) qualitative comparison of generated samples for all datasets which were not included in the main manuscript due to the page limit.
Lemma 1. Let  $G$  be an undirected graph, with a node signal  $\mathbf{X}$  and a symmetric adjacency matrix  $\mathbf{A}$ . Given wavelet coefficients  $W_{\mathbf{A}}(s)$  and  $W_{\mathbf{X}}(s)$ , which are spectral representations of  $\mathbf{X}$  and  $\mathbf{A}$  at a specific scale  $s$ , a dot product of these wavelet coefficients is equivalent to the product between the scale-filtered edge signals and the node signals in the graph space, i.e.,  $W_{\mathbf{A}}(s)\cdot W_{\mathbf{X}}(s) = \mathbf{A}^{s}\mathbf{X}$ .
Proof. With  $\psi_s = \mathbf{U}k(s\pmb{\Lambda})\mathbf{U}^T$  as the basis, the wavelet coefficients of the edge and node signals are given as follows:
$$
W _ {\mathbf {A}} (s) = \psi_ {s} \cdot \mathbf {A} = \mathbf {U} k (s \boldsymbol {\Lambda}) \mathbf {U} ^ {T} \mathbf {A} \tag {10}
$$
$$
W _ {\mathbf {X}} (s) = \psi_ {s} \cdot \mathbf {X} = \mathbf {U} k (s \boldsymbol {\Lambda}) \mathbf {U} ^ {T} \mathbf {X}.
$$
As the  $k(s\Lambda)$  is a diagonal matrix and  $\mathbf{U}$  is a set of orthonormal eigenvectors, the dot product of  $W_{\mathbf{A}}(s)$  and  $W_{\mathbf{X}}(s)$  is computed as