SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Synthetic data. In this experiment, we create a  $2 \times n$  non-attributed cyclic grid. Each node is assigned either a binary label, with neighboring nodes having different labels. We consider two scenarios: one where the known labels are randomly scattered throughout the graph, and another where they are clustered in a local region. Examples of both scenarios are illustrated in Figures 4(a) and 4(b). These two scenarios verify the capability for capturing both short and long-range dependencies between node labels. The data statistics are described in Appendix D.
We compare DPM-SNC with conventional GNN that ignores the label dependencies and other structured node classification methods: GMNN [7],  $\mathrm{G}^3\mathrm{NN}$  [8], and CLGNN [13]. We describe the detailed experimental setup in Appendix E.1. We report the performance using five different random seeds.
(a) Scattered
Figure 4: Illustration of two scenarios. Green highlights known labels.
(b) Localized
Table 1: The transductive node classification accuracy on synthetic data. Bold numbers indicate the best performance.
Method	Scattered	Localized
GCN [3]	50.1±0.6	51.3±1.4
+ GMNN [7]	64.2±3.2	48.2±4.8
+ G3NN [8]	50.1±0.6	51.3±1.4
+ CLGNN [13]	87.0±2.4	53.4±2.1
+ DPM-SNC	98.5±0.7	90.9±3.4
Table 2: The transductive node classification performance. N-Acc. and Sub-Acc. denote the node-level and subgraph-level accuracy, respectively. Bold numbers indicate the best performance among the structured prediction methods using the same GNN.
Method	Pubmed	Cora	CiteSeer	Photo	Computer
N-Acc.	Sub-Acc.	N-Acc.	Sub-Acc.	N-Acc.	Sub-Acc.	N-Acc.	Sub-Acc.	N-Acc.	Sub-Acc.
LP [42]	69.1±0.0	45.7±0.0	68.1±0.0	46.9±0.0	46.1±0.0	29.8±0.0	81.0±2.0	37.2±1.7	69.9±2.9	15.1±1.1
C&S [43]	77.3±0.0	57.1±0.0	80.2±0.0	60.1±0.0	69.5±0.0	47.9±0.0	90.1±1.4	48.5±2.5	81.6±0.9	24.5±1.3
BPN [44]	78.2±1.5	48.4±2.9	82.5±0.9	63.8±0.9	73.3±0.7	51.9±1.1	89.0±1.0	44.1±1.4	80.4±1.4	22.3±1.5
PTA [45]	80.1±0.2	55.2±0.4	82.9±0.4	62.6±0.8	71.3±0.4	51.4±0.7	91.1±1.5	51.0±1.5	81.6±1.7	26.3±1.0
GCN [3]	79.7±0.3	55.8±0.6	81.4±0.8	59.3±1.1	70.9±0.8	49.8±0.6	91.0±1.2	52.0±1.0	82.4±1.5	27.0±1.5
+LPA [12]	79.6±0.6	53.5±0.9	81.7±0.7	60.3±1.5	71.0±0.6	50.2±1.0	91.3±1.2	52.9±2.0	83.7±1.4	28.5±2.4
+GMNN [7]	82.6±1.0	58.1±1.4	82.6±0.9	61.8±1.3	72.8±0.7	52.0±0.8	91.2±1.2	54.3±1.4	82.0±1.0	28.0±1.6
+G3NN [8]	80.9±0.7	56.9±1.1	82.5±0.4	62.3±0.8	73.9±0.7	53.1±1.0	90.7±1.1	53.0±2.0	82.1±1.2	28.1±2.1
+CLGNN [13]	81.7±0.5	57.8±0.7	81.9±0.5	61.8±0.8	72.0±0.7	51.6±0.9	91.1±1.0	53.4±1.8	83.3±1.2	28.5±1.4
+DPM-SNC	83.0±0.9	59.2±1.2	83.2±0.5	63.1±0.9	74.4±0.5	53.6±0.6	92.2±0.8	55.3±2.1	84.1±1.3	29.7±1.8
GAT [21]	79.1±0.5	55.8±0.5	81.5±0.6	61.3±0.9	71.0±0.8	50.8±1.0	90.8±1.0	50.8±1.9	83.1±1.6	27.8±2.2
+LPA [12]	78.7±1.1	56.0±1.2	81.5±0.9	60.7±0.8	71.3±0.9	50.1±0.9	91.3±0.8	52.7±2.1	84.4±1.0	29.4±2.6
+GMNN [7]	79.6±0.8	57.0±0.7	82.3±0.7	62.2±0.8	71.7±0.9	51.4±0.9	91.4±1.0	53.1±1.6	83.3±2.0	29.1±1.8
+G3NN [8]	77.9±0.4	55.9±0.5	82.7±1.3	62.7±1.3	74.0±0.8	53.7±0.5	91.5±0.9	52.6±2.2	83.1±1.7	28.8±2.4
+CLGNN [13]	80.0±0.6	57.5±1.2	81.8±0.6	61.5±0.9	72.1±0.8	52.1±0.8	90.6±0.8	51.9±1.8	82.6±1.2	28.4±1.8
+DPM-SNC	81.7±0.8	59.0±1.1	83.8±0.7	63.8±0.7	74.3±0.7	54.0±0.9	92.0±0.8	54.0±2.4	84.2±1.2	30.0±2.0
GCNII [46]	82.0±0.8	57.2±1.1	84.0±0.6	63.4±0.8	72.9±0.5	52.1±0.7	91.2±1.2	53.2±1.5	82.5±1.4	26.6±1.3
+DPM-SNC	83.8±0.7	61.6±0.9	85.3±0.6	65.8±0.7	74.1±0.5	54.1±0.9	92.8±1.1	54.2±1.2	84.4±1.8	29.2±1.1
The results, presented in Table 1, demonstrate that our method significantly improves accuracy compared to the baselines when labeled nodes are scattered. This highlights the superiority of DPM-SNC in considering label dependencies. Furthermore, our method also excels in the localized labeled nodes scenario, while the other baselines fail. These results can be attributed to the capabilities of DPMs for capturing long-range dependencies through iterative reverse diffusion steps.
Homophilic graph. In this experiment, we consider five datasets: Pubmed, Cora, and Citeseer [22]; Photo and Computer [23]. For all datasets, 20 nodes per class are used for training, and the remaining nodes are used for validation and testing. Detailed data statistics are in Appendix D.
We compare DPM-SNC with conventional GNN, and structured prediction baselines. We compare with label propagation-based methods: LP [42], C&S [43], BPN [44], and PTA [45]. We also consider  $\mathrm{G}^3\mathrm{NN}$ , GMNN, and CLGNN. We describe the detailed experimental setup in Appendix E.2. As the backbone network, we consider GCN and GAT [21]. We further evaluate our algorithm with the recently developed GCNII [46]. For all the datasets, we evaluate the node-level accuracy. We also report the subgraph-level accuracy, which measures the ratio of nodes with all neighboring nodes being correctly classified. The performance is measured using ten different random seeds.
The results are presented in Table 2. Our method outperforms the structured prediction-specialized baselines in both node-label and subgraph-level accuracy. These results highlight the superiority of DPM-SNC in solving real-world node classification problems. Furthermore, even when we combine with GCNII [46], DPM-SNC consistently improves performance regardless of the backbone network.
Table 3: The transductive node classification accuracy on heterophilic graphs. Bold numbers indicate the best score.
Empire	Rating
H2GCN [47]	60.11±0.52	36.47±0.23
CPGNN [48]	63.96±0.62	39.79±0.77
GPR-GNN [49]	64.85±0.27	44.88±0.34
FSGNN [50]	79.92±0.56	52.74±0.83
GloGNN [51]	59.63±0.69	36.89±0.14
FAGCN [52]	65.22±0.56	44.12±0.30
GBK-GNN [53]	74.57±0.47	45.98±0.71
JacobiConv [54]	71.14±0.42	43.55±0.48
GCN [3]	73.69±0.74	48.70±0.63
SAGE [4]	85.74±0.67	53.63±0.39
GAT [21]	80.87±0.30	49.09±0.63
GAT-sep [24]	88.75±0.41	52.70±0.62
GT [55]	86.51±0.73	51.17±0.66
GT-sep [24]	87.32±0.39	52.18±0.80
DPM-SNC	89.52±0.46	54.66±0.39
Heterophilic graph. To validate whether our framework can also consider heterophily dependencies, we consider recently proposed heterophilic graph datasets: Empire and Ratings [24], where most heterophily-specific GNNs fail to solve. In Table 3, we compare our method with six GNNs:
Table 4: The inductive node classification performance. N-Acc., G-Acc., and F1 denote the node-level accuracy, graph-level accuracy, and micro-F1 score, respectively. Bold numbers indicate the best performance among the structured prediction methods using the same GNN.
Method	Pubmed	Cora	Citeseer	PPI
N-Acc.	G-Acc.	N-Acc.	G-Acc.	N-Acc.	G-Acc.	F1
GCN [3]	80.25±0.42	54.58±0.51	83.36±0.43	59.67±0.51	76.37±0.35	49.84±0.47	99.15±0.03
+G3NN [8]	80.32±0.30	53.93±0.71	83.60±0.25	59.78±0.47	76.34±0.37	50.76±0.47	99.33±0.02
+CLGNN [13]	80.22±0.45	53.98±0.54	83.45±0.34	60.24±0.38	75.71±0.40	50.51±0.38	99.22±0.04
+SPN [9]	80.78±0.34	54.91±0.40	83.85±0.60	60.35±0.57	76.25±0.48	51.02±1.06	99.35±0.02
+DPM-SNC	80.58±0.41	55.16±0.43	84.09±0.27	60.88±0.36	77.01±0.49	51.44±0.56	99.46±0.02
GAT [21]	80.10±0.45	54.38±0.54	79.71±1.41	56.66±1.40	74.91±0.22	49.87±0.44	99.54±0.01
+G3NN [8]	79.88±0.62	54.66±0.29	81.19±0.45	58.68±0.38	75.45±0.26	50.86±0.46	99.56±0.01
+CLGNN [13]	80.23±0.40	54.51±0.36	81.38±0.55	58.81±0.61	75.45±0.36	50.66±0.45	99.55±0.01
+SPN [9]	79.95±0.34	54.82±0.33	81.61±0.31	59.17±0.31	75.41±0.35	51.04±0.53	99.46±0.02
+DPM-SNC	80.26±0.37	54.26±0.47	81.79±0.46	59.55±0.49	76.46±0.60	52.05±0.71	99.63±0.01
GCN, SAGE [4], GAT, GAT-sep [24], GT [55], and GT-sep [24]. We also compare with eight heterophily-specific GNNs:  $\mathrm{H}_2\mathrm{GCN}$  [47], CPGNN [48], GPR-GNN [49], FSGNN [50], GloGNN [51], FAGCN [52], GBK-GNN [53], and JacobiConv [54]. We employ GAT-sep as a backbone network of DPM-SNC. For all the baselines, we use the numbers reported by Platonov et al. [24].
Table 3 shows that our method again achieves competitive performance on heterophilic graphs. While existing heterophily-specific GNNs do not perform well on these datasets [24], our method shows improved performance stems from the extraordinary ability for considering label dependencies involving heterophily label dependencies.
6.2 Inductive setting
We further show that our DPM-SNC works well not only in transductive settings but also in inductive settings which involve inductive node classification and graph algorithmic reasoning. We provide the details of our implementation in Appendix C.2.
Inductive node classification. Following Qu et al. [9], we conduct experiments on both small-scale and large-scale graphs. We construct small-scale graphs from Pubmed, Cora, and Citeseer, and construct large-scale graphs from PPI [25]. The detailed data statistics are described in Appendix D.
We compare our DPM-SNC with the conventional GNN and three structured node classification methods: G3NN, CLGNN, and SPN [9]. As the backbone network of each method, we consider GCN and GAT. We evaluate node-level accuracy across all datasets and supplement it with additional metrics: graph-level accuracy for small-scale graphs and micro-F1 score for large-scale graphs. The graph-level accuracy measures the ratio of graphs with where all the predictions are correct. We report the performance measured using ten and five different random seeds for small-scale and large-scale graphs, respectively. The detailed experimental setup is described in Appendix E.3.
We report the results in Table 4. Here, DPM-SNC shows competitive results compared to all the baselines except for Pubmed. These results suggest that the DPM-SNC also solves inductive node classification effectively, thanks to their capability for learning joint dependencies.
Algorithmic reasoning. We also evaluate our DPM-SNC to predict the outcomes of graph algorithms, e.g., computing the shortest path between two nodes. Solving such tasks using GNNs has gained much attention since it builds connections between deep learning and classical computer science algorithms. Here, we show that the capability of DPM-SNC to make a structured prediction even brings benefits to solving the reasoning tasks by a deep understanding between algorithmic elements.
We evaluate the performance of our DPM-SNC on three graph algorithmic reasoning benchmarks proposed by Du et al. [26]: edge copy, connected component, and shortest path. The detailed data statistics are described in Appendix D. Here, we evaluate performance on graphs with ten nodes. Furthermore, we also use graphs with 15 nodes to evaluate generalization capabilities. We report the performance using element-wise mean square error.
Table 5: Performance on graph algorithmic reasoning tasks. Bold numbers indicate the best performance. Same-MSE and Large-MSE indicate the performance on ten, and 15 nodes, respectively.
Method	Edge copy	Connected components	Shortest path
Same-MSE	Large-MSE	Same-MSE	Large-MSE	Same-MSE	Large-MSE
Feedforward	0.3016	0.3124	0.1796	0.3460	0.1233	1.4089
Recurrent [56]	0.3015	0.3113	0.1794	0.2766	0.1259	0.1083
Programmatic [57]	0.3053	0.4409	0.2338	3.1381	0.1375	0.1290
Iterative feedforward [58]	0.6163	0.6498	0.4908	1.2064	0.4588	0.7688
IREM [26]	0.0019	0.0019	0.1424	0.2171	0.0274	0.0464
DPM-SNC	0.0011	0.0038	0.0724	0.1884	0.0138	0.0286
Figure 5: Accuracy with varying Figure 6: Accuracy for changes Figure 7: Accuracy with varying GNN layers and diffusion steps. in diffusion steps. inference time.
We compare our method with five methods reported by Du et al. [26], including a feedforward neural network, recurrent neural network [56], Pondernet [57], iterative feedforward [58], and IREM [26]. For all the baselines, we use the numbers reported by Du et al. [26]. As these tasks are defined on edge-wise targets, we modify DPM-SNC to make edge-wise predictions. We describe the detailed experimental setup in Appendix E.4.
We report the results in Table 5. Our DPM-SNC outperforms the considered baselines for five out of the six tasks. These results suggest that the diffusion model can easily solve algorithmic reasoning tasks thanks to its superior ability to make structured predictions.
6.3 Ablation studies
Here, we conduct three ablation studies to empirically analyze our framework. All the results are measured over ten different random seeds.
Diffusion steps vs. number of GNN layers. We first verify that the performance gains in DPM-SNC mainly stem from the reverse diffusion process which learns the joint dependency between labels. To this end, we vary the number of diffusion steps along with the number of GNN layers. We report the corresponding results in Figure 5. One can observe that increasing the number of diffusion steps provides a non-trivial improvement in performance, which cannot be achieved by just increasing the number of GNN layers.
Accuracy over diffusion steps. We investigate whether the iteration in the reverse process progressively improves the quality of predictions. In Figure 6, we plot the changes in node-level accuracy in the reverse process as the number of iterations increases. The results confirm that the iterative inference process gradually increases accuracy, eventually reaching convergence.
Running time vs. performance. Here, we investigate whether the DPM-SNC can make a good trade-off between running time and performance. In Figure 7, we compare the change in accuracy of DPM-SNC with GCNII over the inference time on Citeseer by changing the number of layers and diffusion steps for DPM-SNC and GCNII, respectively. The backbone network of DPM-SNC is a two-layer GCN. One can observe that our DPM-SNC shows competitive performances compared to the GCNII at a similar time. Also, while increasing the inference times of the GCNII does not enhance performance, DPM-SNC shows further performance improvement.
In this paper, we propose diffusion probabilistic models for solving structured node classification (DPM-SNC). Extensive experiments on both transductive and inductive settings show that DPM-SNC outperforms existing structured node classification methods. An interesting avenue for future work is the study of characterizing the expressive power of GNNs to make structured predictions, i.e., the ability to learn complex dependencies between node-wise labels.
Limitations. Our DPM-SCN makes a trade-off between accuracy and inference time through the number of diffusion steps. Therefore, we believe accelerating our framework with faster diffusion-based models, e.g., denoising diffusion implicit models (DDIM) [59] is an interesting future work.
This work partly was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. IITP-2019-0-01906, Artificial Intelligence Graduate School Program(POSTECH)), the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. 2022R1C1C1013366), and Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(2022R1A6A1A03052954).
[1] Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The world wide web conference, pages 417-426, 2019. 1
[2] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine, 29(3):93-93, 2008. 1, 4
[3] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016. 1, 2, 3, 6, 7, 8
[4] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. NeurIPS, 30, 2017. 1, 3, 7, 8
[5] Karsten M Borgwardt and Hans-Peter Kriegel. Shortest-path kernels on graphs. In Fifth IEEE international conference on data mining (ICDM'05), pages 8-pp. IEEE, 2005. 1
[6] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855-864, 2016. 1
[7] Meng Qu, Yoshua Bengio, and Jian Tang. Gmnn: Graph markov neural networks. In International conference on machine learning, pages 5241-5250. PMLR, 2019. 1, 2, 3, 5, 6, 7, 22
[8] Jiaqi Ma, Weijing Tang, Ji Zhu, and Qiaozhu Mei. A flexible generative framework for graph-based semi-supervised learning. Advances in Neural Information Processing Systems, 32, 2019. 1, 6, 7, 8
[9] Meng Qu, Huiyu Cai, and Jian Tang. Neural structured prediction for inductive node classification. arXiv preprint arXiv:2204.07524, 2022. 1, 3, 8, 23
[10] Tengfei Ma, Cao Xiao, Junyuan Shang, and Jimeng Sun. CGNF: Conditional graph neural fields, 2019.
[11] Colin Graber and Alexander Schwing. Graph structured prediction energy networks. Advances in Neural Information Processing Systems, 32, 2019.
[12] Hongwei Wang and Jure Leskovec. Unifying graph convolutional neural networks and label propagation. arXiv preprint arXiv:2002.06755, 2020. 7
[13] Mengyue Hang, Jennifer Neville, and Bruno Ribeiro. A collective learning framework to boost gnn expressiveness for node classification. In International Conference on Machine Learning, pages 4040-4050. PMLR, 2021. 1, 2, 3, 6, 7, 8, 17
[54] Xiyuan Wang and Muhan Zhang. How powerful are spectral graph neural networks. In International Conference on Machine Learning, pages 23341-23362. PMLR, 2022. 7, 8
[55] Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arXiv:2009.03509, 2020. 7, 8
[56] Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems, 34:6695-6706, 2021. 9
[57] Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder. In 8th ICML Workshop on Automated Machine Learning (AutoML). 9
[58] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015. 9
[59] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations. 10
[60] Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. Advances in Neural Information Processing Systems, 34:9061-9073, 2021. 17
[61] Ryan Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Relational pooling for graph representations. In International Conference on Machine Learning, pages 4663-4673. PMLR, 2019. 18
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 22
[63] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265, 2019. 26
A Details of DPM on partially labeled graphs
A.1 Derivation of variational lower bound
In this section, we provide a detailed derivation of the variational lower bound in Equation (1).
$$
\begin{array}{l} \log p _ {\boldsymbol {\theta}} (\boldsymbol {y} _ {L} | G) = \log \mathbb {E} _ {p _ {\boldsymbol {\theta}} (\boldsymbol {y} _ {U}, \boldsymbol {y} ^ {(1: T)} | G)} \left[ p _ {\boldsymbol {\theta}} (\boldsymbol {y} _ {L} | G, \boldsymbol {y} _ {U}, \boldsymbol {y} ^ {(1: T)}) \right] \\ = \log \mathbb {E} _ {q (\boldsymbol {y} _ {U}, \boldsymbol {y} ^ {(1: T)} | \boldsymbol {y} _ {L})} \left[ \frac {p _ {\boldsymbol {\theta}} (\boldsymbol {y} , \boldsymbol {y} ^ {(1 : T)} | G)}{q (\boldsymbol {y} _ {U} , \boldsymbol {y} ^ {(1 : T)} | \boldsymbol {y} _ {L})} \right] \\ \geq \mathbb {E} _ {q (\boldsymbol {y} _ {U}, \boldsymbol {y} ^ {(1: T)} | \boldsymbol {y} _ {L})} \left[ \log p _ {\boldsymbol {\theta}} (\boldsymbol {y}, \boldsymbol {y} ^ {(1: T)} | G) - q (\boldsymbol {y} _ {U}, \boldsymbol {y} ^ {(1: T)} | \boldsymbol {y} _ {L}) \right] \\ = \mathbb {E} _ {q (\boldsymbol {y} _ {U}, \boldsymbol {y} ^ {(1: T)} | \boldsymbol {y} _ {L})} \left[ \log p _ {\boldsymbol {\theta}} (\boldsymbol {y}, \boldsymbol {y} ^ {(1: T)} | G) - q (\boldsymbol {y} ^ {(1: T)} | \boldsymbol {y}) - \log q (\boldsymbol {y} _ {U} | \boldsymbol {y} _ {L}) \right] \\ = \mathbb {E} _ {q (\boldsymbol {y} _ {U} | \boldsymbol {y} _ {L})} \left[ \mathbb {E} _ {q (\boldsymbol {y} ^ {(1: T)} | \boldsymbol {y})} [ \log p _ {\boldsymbol {\theta}} (\boldsymbol {y}, \boldsymbol {y} ^ {(1: T)} | G) - q (\boldsymbol {y} ^ {(1: T)} | \boldsymbol {y}) ] - \log q (\boldsymbol {y} _ {U} | \boldsymbol {y} _ {L}) \right]. \\ \end{array}
$$
A.2 Parameterization
Here, we provide more detailed parameterization for DPM-SNC. Following the Gaussian diffusion, we parameterize  $p(\pmb{y}^T)$  and  $p_{\pmb{\theta}}(\pmb{y}^{(t - 1)}|G,\pmb{y}^{(t)})$  as  $\mathcal{N}(\pmb{y}^{(T)};\mathbf{0},\mathbf{I})$  and  $\mathcal{N}(\pmb{y}^{(t - 1)};\pmb{\mu}_{\pmb{\theta}}(\pmb{y}^{(t)},G,t),\sigma_t^2)$ , respectively. Here, we set  $\sigma_t^2$  to  $\beta_{t}$ . We also define  $\mu_{\pmb{\theta}}(\pmb{y}^{(t)},G,t)$  as follows:
$$
\mu_ {\boldsymbol {\theta}} \left(\boldsymbol {y} ^ {(t)}, G, t\right) = \frac {1}{\sqrt {\alpha_ {t}}} \left(\boldsymbol {y} ^ {(t)} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} \left(\boldsymbol {y} ^ {(t)}, G, t\right)\right), \tag {2}
$$
where  $\alpha_{t} = 1 - \beta_{t}$  and  $\bar{\alpha}_{t} = \prod_{i = 1}^{t}\alpha_{i}$ . We implement the residual function  $\epsilon_{\theta}(\pmb{y}^{(t)},G,t)$  through a GNN. The implementation details are described in Appendix C.
A.3 Detailed training objective
We describe the detailed training objective of  $\mathcal{L}_{\mathrm{VLB}}$  for optimization. We first rewrite  $\mathcal{L}_{\mathrm{VLB}}$  as follows:  $\mathcal{L}_{\mathrm{VLB}}$