Title: Diffusion Probabilistic Models for Structured Node Classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion model

graph neural network

structured prediction

node classification
Abstract: This paper studies structured node classification on graphs, where the predictions should consider dependencies between the node labels. In particular, we focus on solving the problem for partially labeled graphs where it is essential to incorporate the information in the known label for predicting the unknown labels. To address this issue, we propose a novel framework leveraging the diffusion probabilistic model for structured node classification (DPM-SNC). At the heart of our framework is the extraordinary capability of DPM-SNC to (a) learn a joint distribution over the labels with an expressive reverse diffusion process and (b) make predictions conditioned on the known labels utilizing manifold-constrained sampling. Since the DPMs lack training algorithms for partially labeled data, we design a novel training algorithm to apply DPMs, maximizing a new variational lower bound. We also theoretically analyze how DPMs benefit node classification by enhancing the expressive power of GNNs based on proposing AGG-WL, which is strictly more powerful than the classic 1-WL test. We extensively verify the superiority of our DPM-SNC in diverse scenarios, which include not only the transductive setting on partially labeled graphs but also the inductive setting and unlabeled graphs.

Diffusion Probabilistic Models for Structured Node Classification

Abstract

This paper studies structured node classification on graphs, where the predictions should consider dependencies between the node labels. In particular, we focus on solving the problem for partially labeled graphs where it is essential to incorporate the information in the known label for predicting the unknown labels. To address this issue, we propose a novel framework leveraging the diffusion probabilistic model for structured node classification (DPM-SNC). At the heart of our framework is the extraordinary capability of DPM-SNC to (a) learn a joint distribution over the labels with an expressive reverse diffusion process and (b) make predictions conditioned on the known labels utilizing manifold-constrained sampling. Since the DPMs lack training algorithms for partially labeled data, we design a novel training algorithm to apply DPMs, maximizing a new variational lower bound. We also theoretically analyze how DPMs benefit node classification by enhancing the expressive power of GNNs based on proposing AGG-WL, which is strictly more powerful than the classic 1-WL test. We extensively verify the superiority of our DPM-SNC in diverse scenarios, which include not only the transductive setting on partially labeled graphs but also the inductive setting and unlabeled graphs.

1 Introduction

In this paper, we address the node classification problem, which is a fundamental problem in machine learning on graphs with various applications, such as social networks [1] and citation networks [2]. One representative example is a transductive problem to classify documents from a partially labeled citation graph. Recently, graph neural networks (GNNs) [3, 4] have shown great success in this problem over their predecessors [5, 6]. Their success stems from the high capacity of non-linear neural architectures combined with message passing.

However, the conventional GNNs are incapable of structured node classification : predicting node labels while considering the node-wise label dependencies [7-9]. That is, given a graph G with vertices V and node labels { y i : i ∈ V} , a GNN with parameter θ outputs an unstructured prediction, i.e., p θ ( y i , y j | G ) = p θ ( y i | G ) p θ ( y j | G ) for i, j ∈ V . Especially, this limitation becomes problematic in a transductive setting where the prediction can be improved by incorporating the known labels, e.g., output p θ ( y i | G,y j ) for known y j . In Figures 1(a) and 1(b), we elaborate on this issue with an example where the conventional GNN fails to make the correct prediction.

To resolve this issue, recent studies have investigated combining GNNs with classical structured prediction algorithms, i.e., schemes that consider dependencies between the node labels [8-13]. They combine GNNs with conditional random fields [14], label propagation [15], or iterative classification algorithm [2]. Despite the promising outcomes demonstrated by these studies, their approach relies on the classical algorithms to express joint dependencies between node labels and may lack sufficient expressive power or consistency in incorporating known labels.

(a) Input

(b) GCN [3]

(d) CLGNN [13]

(e) DPM-SNC (ours)

Contribution. We propose a novel framework for structured node classification called DPM-SNC. Our key idea is to leverage the diffusion probabilistic model (DPM) [16], motivated by two significant advantages of DPM for structured node classification: (a) it can effectively learn a joint distribution over both the unknown and the known labels, and (b) it can easily incorporate conditions during inference via posterior sampling. Figure 1 highlights that DPM significantly outperforms previous methods when the prediction problem requires a complete understanding of label dependencies.

However, DPM cannot be directly applied to the transductive scenario, where the model needs to maximize its log-likelihood for partially labeled graphs. We propose a novel training algorithm to address this challenge. Our method maximizes a new variational lower bound of the marginal likelihood of the graph over the unlabeled nodes, involving the alternative optimization of DPM and a variational distribution. In this process, we estimate the variational distribution from the current DPM using a similar approach to fixed-point iteration [17]. After training, DPM can predict the remaining node labels by constraining its predictions with the given labeled nodes, utilizing the manifold-constrained sampling technique [18].

In addition, we provide a theoretical analysis that supports the benefits of DPM-SNC for node classification by enhancing the expressive power of GNNs. To this end, we derive an analog of our DPM-SNC based on the Weisfeiler-Lehman (WL) [19] test, evaluating the ability to distinguish non-isomorphic graphs. Specifically, we prove that DPM-SNC is as powerful as our newly derived aggregated Weisfeiler-Lehman (AGG-WL) test, which is strictly more powerful than the 1-WL test, a known analog to standard GNN architectures [20] such as GCN [3] and GAT [21].

We demonstrate the effectiveness of DPM-SNC on various datasets, covering both transductive and inductive settings. In the transductive setting, we conduct experiments on seven benchmarks: Pubmed, Cora, and Citeseer [22]; Photo and Computer [23]; and Empire and Ratings [24]. Additionally, we introduce a synthetic benchmark to evaluate the ability to capture both short and long-range dependencies. DPM-SNC outperforms the baselines, including both homophilic and heterophilic graphs. In the inductive setting, we conduct experiments on four benchmarks: Pubmed, Cora, Citeseer, and PPI [25]. Furthermore, we evaluate DPM-SNC on the algorithmic reasoning benchmark [26], which requires a higher-level understanding of node relations. DPM-SNC also outperforms previous algorithms for these tasks.

To summarize, our contribution can be listed as follows:

We explore DPM as an effective solution for structured node classification due to its inherent capability in (a) learning the joint node-wise label dependency in data and (b) making predictions conditioned on partially known data (Section 3).

We propose a novel method for training a DPM on partially labeled graphs that leverages the probabilistic formulation of DPM to maximize the variational lower bound (Section 4).

We provide a theoretical analysis of how DPM enhances the expressive power of graph neural networks, strictly improving its capability over the 1-WL test (Section 5).

Our experiments demonstrate the superiority over the baselines in both transductive and inductive settings. We additionally consider heterophilic graphs and algorithmic reasoning benchmarks to extensively evaluate the capability of DPMs in understanding the label dependencies (Section 6).

2 Related Work

Graph neural networks (GNNs). Graph neural networks (GNNs) are neural networks specifically designed to handle graph-structured data. Over the past years, GNNs have shown promising outcomes in various tasks related to graph representation learning [3, 4, 27, 28]. In the case of node classification, GNNsaggregate neighboring node information and generate node representations for label predictions. Their expressive power has been analyzed through the lens of the Weisfeiler-Lehman (WL) test [19], a classical algorithm to distinguish non-isomorphic graphs. For example, Xu et al. [20] proved how the expressive power of conventional GNNs is bounded by the 1-dimensional WL (1-WL) test.

Structured node classification with GNNs. Apart from investigating the power of GNNs to discriminate isomorphic graphs, several studies focused on improving the ability of GNNs to learn the joint dependencies [7, 9, 13, 26, 29-31]. In particular, researchers have considered GNN-based structured node classification [7, 9, 13]. First, Qu et al., [7] parameterize the potential functions of conditional random fields using a GNN and train it using pseudo-likelihood maximization [32]. Next, Qu et al., [9] proposed a training scheme with a GNN-based proxy for the conditional random field. Hang et al., [13] proposed an iterative classification algorithm-based learning framework of GNNs that updates each node-wise prediction by aggregating previous predictions on neighboring nodes.

Diffusion probabilistic models (DPMs). Inspired by non-equilibrium thermodynamics, DPMs are latent variable models (LVMs) that learn to reverse a diffusion process and construct data from a noise distribution. Recent studies demonstrated great success of DPMs in various domains, e.g., generating images [33], text [34], and molecules [35]. An important advantage of diffusion models is their ability to incorporate additional constraints. This allows DPM to solve the posterior inference problems, e.g., conditioning on the partially observed data to recover the original data [18, 36, 37].

3 Diffusion Probabilistic Models for Structured Node Classification

In this section, we introduce the problem of structured node classification and discuss the limitations of graph neural networks (GNNs) in addressing this problem (Section 3.1). We then explain how the diffusion probabilistic models (DPMs) offer a promising solution for this problem (Section 3.2).

3.1 Structured node classification

We address structured node classification, which involves predicting node labels while considering their joint dependencies. Specifically, we focus on predicting labels in partially labeled graphs, where some true labels are known in advance. 1 Our goal is to devise an algorithm that considers the dependencies with known labels to enhance predictions for unknown labels.

To be specific, our problem involves a graph G = ( V , E , x ) consisting of nodes V , edges E , and node attributes x = { x i : i ∈ V} . We also denote the node labels by y = { y i : i ∈ V} . We let V L and V U denote the set of labeled and unlabeled nodes, while y L and y U denote the corresponding labels for each set, e.g., y L = { y i : i ∈ V L } . Our objective is to predict the unknown labels y U by training on the partially labeled graph, aiming to infer the true conditional distribution p ( y U | G, y L ) .

diffusion and projection

reverse process

step 1

step 50

step 100

node-wise

predictions

known

labels

We point out that the conventional GNNs [3, 4, 27, 28] are suboptimal for estimating p ( y U | G, y L ) since their predictions on the node labels are independently conditioned on the input, i.e., their predictions are factorized by p θ ( y U | G, y L ) = ∏ i ∈V U p θ ( y i | G ) . Namely, conventional GNNs cannot incorporate the information of the known labels y L into a prediction of the unknown labels y U .

(𝐺, 𝒚)

(𝐺, 𝒚

𝐋

)

posterior

unlabeled node

buffer

𝓑

inserts

sampling

𝒚

𝑼

lower bound

(𝐺, 𝒚

(")

)

maximize

(𝐺, 𝒚)

buffer

𝓑

diffusion

samples

𝒚

𝑼

(b) Update the DPM parameter to maximize L VLB.

Algorithm 1 DPM-SNC

1: Train a mean-field GNN p ϕ ( y | G ) .

2: Initialize the buffer B by p ϕ ( y U | G ) .

3: repeat

4: for i = 1 , . . . , N 1 do

5: Get y U ∼ p θ ( y U | G, y L ) using manifold-constrained sampling.

6: Update B ← B ∪ { y U } .

7: If |B| > K , remove oldest one in B .

8: end for

9: for i = 1 , . . . , N 2 do

10: Sample y U ∼ B .

11: Update θ to maximize L VLB with G , and y = y L ∪ y U .

12: end for

13: until converged

Structured prediction algorithms [2, 14] overcome this issue by solving two tasks: (a) learning a joint distribution p θ ( y U , y L | G ) to maximize the likelihood of labeled data p θ ( y L | G ) and (b) inferring from the distribution p θ ( y U | G, y L ) conditioned on known labels.

3.2 Diffusion probabilistic models for structured node classification

In this work, we consider diffusion probabilistic models for structured node classification (DPMSNC) motivated by how their strengths align with the essential tasks for solving structured node classification, outlined in Section 3.1. In particular, DPMs are equipped with (a) high expressivity in learning a joint distribution over data and (b) the ability to easily infer from a posterior distribution conditioned on partially observed data.

To this end, we formally describe DPMs that generate node-wise labels y associated with a graph G . At a high level, our DPM consists of two parts: the forward process, which injects noise into the labels, and the reverse process, which denoises to predict the true labels y [16]. Given the number of diffusion steps T , the forward process constructs a sequence of noisy labels y (1: T ) = [ y (1) , . . . , y ( T ) ] using a fixed distribution q ( y (1: T ) | y (0) ) starting from the true label y (0) = y . 2 Next, given an initial noises y ( T ) sampled from p ( y ( T ) ) , the reverse process p θ ( y (0: T -1) | y ( T ) , G ) is trained to recover the forward process conditioned on the graph G . We defer the detailed parameterization to Section 4. To be specific, the forward and the reverse process are factorized as follows:

where the training of the reverse process is trivial on a fully-labeled graph ( y , G ) by maximizing the variational lower bound of the marginal log-likelihood log p θ ( y | G ) = log ∑ y (1: T ) p θ ( y (0: T ) | G ) with respect to the noisy labels y (1: T ) sampled from the forward process [16].

By leveraging shared latent variables y ( t ) across multiple steps in the reverse process, the reverse process effectively considers the dependencies in the output. Next, DPMs can easily infer from a posterior distribution conditioned on partially observed data. Specifically, the incremental updating of y ( t ) for t = 1 , . . . , T allows the DPM to incorporate the known label y L , e.g., applying iterative projection with manifold-based correction [18]. This enables inference from the distribution p θ ( y U | G, y L ) conditioned on the known labels y L . See Figure 2 for an illustration of DPM for structured node classification with known labels.

4 Training Diffusion Probabilistic Models on Partially Labeled Graphs

Despite the potential of DPMs for structured node classification, they lack the algorithms to learn from partially labeled graphs, i.e., maximize the likelihood of known labels. To resolve this issue, we introduce a novel training algorithm for DPM-SNC, based on maximizing a variational lower bound for the log-likelihood of known labels.

Variational lower bound. At a high-level, our algorithm trains the DPM to maximize the loglikelihood of training data log p θ ( y L | G ) , which is defined as follows:

However, this likelihood is intractable due to the exponentially large number of possible combinations for the unknown labels y U and the noisy sequence of labels y (1: T ) . To address this issue, we train the DPM based on a new variational lower bound L ≥ L VLB, expressed as follows:

Here, q ( · ) is a variational distribution factorized by q ( y U , y (1: T ) | y L ) = q ( y (1: T ) | y ) q ( y U | y L ) , where y = y U ∪ y L . We provide detailed derivation in Appendix A.1.

Parameterization. We define the reverse process p θ ( y ( t -1) | y ( t ) , G ) with a Gaussian distribution, where the mean value parameterized by a graph neural network (GNN) and the variance set to be a hyper-parameter. Following the prior work [16], we also employ a Gaussian diffusion to parameterize the forward process q ( y (1: T ) | y ) as follows:

where I is an identity matrix, β 1 , . . . , β T are fixed variance schedules. We set the variance schedule to promote q ( y ( T ) | y (0) ) ≈ N ( y ( T ) ; 0 , I ) by setting β t < β t +1 for t = 0 , . . . , T -1 and β T = 1 . Finally, we describe the variational distribution q ( y U | y L ) as an empirical distribution over a fixed-size buffer B containing multiple estimates of the unknown labels y U . This buffer is updated throughout the training. We provide more details on our parameterization in Appendix A.2.

Training algorithm. To maximize the variational lower bound, we alternatively update the GNNbased reverse process p θ ( y , y (1: T ) | G ) and the buffer-based variational distribution q ( y U | y L ) . In particular, we update the parameters θ of the reverse process p θ ( y , y (1: T ) | G ) to maximize the Monte Carlo approximation of L LVB by applying ancestral sampling to the variational distribution q ( y (1: T ) | y ) q ( y U | y L ) , i.e., sampling y U from the buffer B and applying the diffusion process to y . This is the same as the original denoising-based training of DPM [16] with estimated unknown labels, where the detailed training objective is described in Appendix A.3.

Next, we update the variational distribution q ( y U | y L ) by inserting samples from the distribution p θ ( y U | G, y L ) into the buffer B . 3 This update is derived from the condition q ( y U | y L ) = p θ ( y U | G, y L ) being necessary to maximize L VLB, similar to the derivation of fixed-point iteration for optimization [17]. We describe the overall optimization procedure in Figure 3 and Algorithm 1. 4

Finally, we emphasize that our training algorithm is specialized for DPMs. Previous studies introducing variational lower bounds for structured node classifications [7, 38] face intractability in maximizing log p θ ( y | G ) or sampling from p θ ( y U | G, y L ) . They require defining the pseudolikelihood for the former [32], or parameterizing variational distribution for the latter. However, in our case, the formal simply requires maximizing the lower bound L VLB, and the latter is easily solved by manifold-constrained sampling [18].

5 Theoretical Analysis

In this section, we present a secondary motivation for using DPMs in node classification, distinct from the one given in Section 3 and Section 4. Specifically, we demonstrate that DPMs provably enhance the expressive power of conventional GNNs for solving the graph isomorphism test, implying improved expressive power for node classification problems as well.

To this end, we assess the expressive power of our DPM-SNC by its analog for discriminating isomorphic graphs, which we call the aggregated Weisfeiler-Lehman (AGG-WL) test. Then, we show that AGG-WL is strictly more powerful than the 1-dimensional WL (1-WL) test [19], which is an analog of GNNs [20]. We formalize our results in the following theorem.

Theorem 1. Let 1-WL-GNN be a GNN as powerful as the 1-WL test. Then, DPM-SNC using a 1-WL-GNN is strictly more powerful than the 1-WL-GNN in distinguishing non-isomorphic graphs.

We provide the formal proof in Appendix B. At a high level, both the 1-WL test and the AGG-WL test assign colors to nodes and iteratively refine them based on the colors of neighboring nodes, enabling the comparison of graph structures. The key difference is that AGG-WL applies perturbations into the initial graph to create multiple augmented graphs, and aggregates the results of the 1-WL algorithm on each augmented graph. The perturbations and aggregation can be done by node-wise concatenations with binary random features and hashing a multiset of the refined graph colors, respectively. Hence, any graph pair distinguishable by the 1-WL test is also distinguishable by our AGG-WL test, and there exist pairs of graphs indistinguishable by the WL test but distinguishable by our variant.

We remark that our analysis can be easily extended to applying other latent variable models, e.g., variational auto-encoders [39] or normalizing flows [40], for node classification. In this analysis, sampling a latent variable corresponds to fixing a perturbation for color assignments at initialization, and aggregating over the perturbed graphs corresponds to computing the prediction over the true labels with marginalization over the latent variables. Our theory is based on that of Bevilaqua et al. [41] which was originally developed for analyzing a particular GNN architecture; our key contribution lies in extending this analysis to the latent variable models.

6 Experiments

6.1 Transductive setting

We first evaluate the performance of our algorithm in the transductive setting. Specifically, we conduct an evaluation of DPM-SNC on synthetic data, as well as real-world node classification problems. We consider different types of label dependencies by considering both homophilic and heterophilic graphs. We provide the details of our implementation in Appendix C.1.

Synthetic data. In this experiment, we create a 2 × n non-attributed cyclic grid. Each node is assigned either a binary label, with neighboring nodes having different labels. We consider two scenarios: one where the known labels are randomly scattered throughout the graph, and another where they are clustered in a local region. Examples of both scenarios are illustrated in Figures 4(a) and 4(b). These two scenarios verify the capability for capturing both short and long-range dependencies between node labels. The data statistics are described in Appendix D.

We compare DPM-SNC with conventional GNN that ignores the label dependencies and other structured node classification methods: GMNN [7], G 3 NN [8], and CLGNN [13]. We describe the detailed experimental setup in Appendix E.1. We report the performance using five different random seeds.

(a) Scattered

(b) Localized

The results, presented in Table 1, demonstrate that our method significantly improves accuracy compared to the baselines when labeled nodes are scattered. This highlights the superiority of DPMSNC in considering label dependencies. Furthermore, our method also excels in the localized labeled nodes scenario, while the other baselines fail. These results can be attributed to the capabilities of DPMs for capturing long-range dependencies through iterative reverse diffusion steps.

Homophilic graph. In this experiment, we consider five datasets: Pubmed, Cora, and Citeseer [22]; Photo and Computer [23]. For all datasets, 20 nodes per class are used for training, and the remaining nodes are used for validation and testing. Detailed data statistics are in Appendix D.

We compare DPM-SNC with conventional GNN, and structured prediction baselines. We compare with label propagation-based methods: LP [42], C&S [43], BPN [44], and PTA [45]. We also consider G 3 NN, GMNN, and CLGNN. We describe the detailed experimental setup in Appendix E.2. As the backbone network, we consider GCN and GAT [21]. We further evaluate our algorithm with the recently developed GCNII [46]. For all the datasets, we evaluate the node-level accuracy. We also report the subgraph-level accuracy, which measures the ratio of nodes with all neighboring nodes being correctly classified. The performance is measured using ten different random seeds.

The results are presented in Table 2. Our method outperforms the structured prediction-specialized baselines in both node-label and subgraph-level accuracy. These results highlight the superiority of DPM-SNC in solving real-world node classification problems. Furthermore, even when we combine with GCNII [46], DPM-SNC consistently improves performance regardless of the backbone network.

Heterophilic graph. To validate whether our framework can also consider heterophily dependencies, we consider recently proposed heterophilic graph datasets: Empire and Ratings [24], where most heterophily-specific GNNs fail to solve. In Table 3, we compare our method with six GNNs:

GCN, SAGE [4], GAT, GAT-sep [24], GT [55], and GT-sep [24]. We also compare with eight heterophily-specific GNNs: H 2 GCN [47], CPGNN [48], GPR-GNN [49], FSGNN [50], GloGNN [51], FAGCN [52], GBK-GNN [53], and JacobiConv [54]. We employ GAT-sep as a backbone network of DPM-SNC. For all the baselines, we use the numbers reported by Platonov et al. [24].

Table 3 shows that our method again achieves competitive performance on heterophilic graphs. While existing heterophily-specific GNNs do not perform well on these datasets [24], our method shows improved performance stems from the extraordinary ability for considering label dependencies involving heterophily label dependencies.

6.2 Inductive setting

We further show that our DPM-SNC works well not only in transductive settings but also in inductive settings which involve inductive node classification and graph algorithmic reasoning. We provide the details of our implementation in Appendix C.2.

Inductive node classification. Following Qu et al. [9], we conduct experiments on both small-scale and large-scale graphs. We construct small-scale graphs from Pubmed, Cora, and Citeseer, and construct large-scale graphs from PPI [25]. The detailed data statistics are described in Appendix D.

We compare our DPM-SNC with the conventional GNN and three structured node classification methods: G3NN, CLGNN, and SPN [9]. As the backbone network of each method, we consider GCN and GAT. We evaluate node-level accuracy across all datasets and supplement it with additional metrics: graph-level accuracy for small-scale graphs and micro-F1 score for large-scale graphs. The graph-level accuracy measures the ratio of graphs with where all the predictions are correct. We report the performance measured using ten and five different random seeds for small-scale and large-scale graphs, respectively. The detailed experimental setup is described in Appendix E.3.

We report the results in Table 4. Here, DPM-SNC shows competitive results compared to all the baselines except for Pubmed. These results suggest that the DPM-SNC also solves inductive node classification effectively, thanks to their capability for learning joint dependencies.

Algorithmic reasoning. We also evaluate our DPM-SNC to predict the outcomes of graph algorithms, e.g., computing the shortest path between two nodes. Solving such tasks using GNNs has gained much attention since it builds connections between deep learning and classical computer science algorithms. Here, we show that the capability of DPM-SNC to make a structured prediction even brings benefits to solving the reasoning tasks by a deep understanding between algorithmic elements.

We evaluate the performance of our DPM-SNC on three graph algorithmic reasoning benchmarks proposed by Du et al. [26]: edge copy, connected component, and shortest path. The detailed data statistics are described in Appendix D. Here, we evaluate performance on graphs with ten nodes. Furthermore, we also use graphs with 15 nodes to evaluate generalization capabilities. We report the performance using element-wise mean square error.

5

10

20

40

80

Diffusion steps

5

4

3

2

1

Number of layers

65.7

67.1

68.4

67.3

68.0

69.6

70.4

71.1

70.7

71.0

70.9

71.3

71.3

71.5

71.9

72.6

73.3

74.0

74.3

74.4

71.4

72.1

72.4

72.4

72.9

0

20

40

60

80

Step

65

70

75

80

85

Pubmed

Citeseer

0.02

0.04

0.06

0.08

Time (s)

71

72

73

74

75

DPM-SNC

Figure 6: Accuracy for changes in diffusion steps. Figure 7: Accuracy with varying inference time.

We compare our method with five methods reported by Du et al. [26], including a feedforward neural network, recurrent neural network [56], Pondernet [57], iterative feedforward [58], and IREM [26]. For all the baselines, we use the numbers reported by Du et al. [26]. As these tasks are defined on edge-wise targets, we modify DPM-SNC to make edge-wise predictions. We describe the detailed experimental setup in Appendix E.4.

We report the results in Table 5. Our DPM-SNC outperforms the considered baselines for five out of the six tasks. These results suggest that the diffusion model can easily solve algorithmic reasoning tasks thanks to its superior ability to make structured predictions.

6.3 Ablation studies

Here, we conduct three ablation studies to empirically analyze our framework. All the results are measured over ten different random seeds.

Diffusion steps vs. number of GNN layers. We first verify that the performance gains in DPM-SNC mainly stem from the reverse diffusion process which learns the joint dependency between labels. To this end, we vary the number of diffusion steps along with the number of GNN layers. We report the corresponding results in Figure 5. One can observe that increasing the number of diffusion steps provides a non-trivial improvement in performance, which cannot be achieved by just increasing the number of GNN layers.

Accuracy over diffusion steps. We investigate whether the iteration in the reverse process progressively improves the quality of predictions. In Figure 6, we plot the changes in node-level accuracy in the reverse process as the number of iterations increases. The results confirm that the iterative inference process gradually increases accuracy, eventually reaching convergence.

Running time vs. performance. Here, we investigate whether the DPM-SNC can make a good trade-off between running time and performance. In Figure 7, we compare the change in accuracy of DPM-SNC with GCNII over the inference time on Citeseer by changing the number of layers and diffusion steps for DPM-SNC and GCNII, respectively. The backbone network of DPM-SNC is a two-layer GCN. One can observe that our DPM-SNC shows competitive performances compared to the GCNII at a similar time. Also, while increasing the inference times of the GCNII does not enhance performance, DPM-SNC shows further performance improvement.

7 Conclusion and Discussion

In this paper, we propose diffusion probabilistic models for solving structured node classification (DPM-SNC). Extensive experiments on both transductive and inductive settings show that DPM-SNC outperforms existing structured node classification methods. An interesting avenue for future work is the study of characterizing the expressive power of GNNs to make structured predictions, i.e., the ability to learn complex dependencies between node-wise labels.

Limitations. Our DPM-SCN makes a trade-off between accuracy and inference time through the number of diffusion steps. Therefore, we believe accelerating our framework with faster diffusionbased models, e.g., denoising diffusion implicit models (DDIM) [59] is an interesting future work.

References

Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The world wide web conference , pages 417-426, 2019. 1

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine , 29(3):93-93, 2008. 1, 4

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016. 1, 2, 3, 6, 7, 8

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. NeurIPS , 30, 2017. 1, 3, 7, 8

Karsten M Borgwardt and Hans-Peter Kriegel. Shortest-path kernels on graphs. In Fifth IEEE international conference on data mining (ICDM'05) , pages 8-pp. IEEE, 2005. 1

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining , pages 855-864, 2016. 1

Meng Qu, Yoshua Bengio, and Jian Tang. Gmnn: Graph markov neural networks. In International conference on machine learning , pages 5241-5250. PMLR, 2019. 1, 2, 3, 5, 6, 7, 22

Jiaqi Ma, Weijing Tang, Ji Zhu, and Qiaozhu Mei. A flexible generative framework for graphbased semi-supervised learning. Advances in Neural Information Processing Systems , 32, 2019. 1, 6, 7, 8

Meng Qu, Huiyu Cai, and Jian Tang. Neural structured prediction for inductive node classification. arXiv preprint arXiv:2204.07524 , 2022. 1, 3, 8, 23

Tengfei Ma, Cao Xiao, Junyuan Shang, and Jimeng Sun. CGNF: Conditional graph neural fields, 2019.

Colin Graber and Alexander Schwing. Graph structured prediction energy networks. Advances in Neural Information Processing Systems , 32, 2019.

Hongwei Wang and Jure Leskovec. Unifying graph convolutional neural networks and label propagation. arXiv preprint arXiv:2002.06755 , 2020. 7

Mengyue Hang, Jennifer Neville, and Bruno Ribeiro. A collective learning framework to boost gnn expressiveness for node classification. In International Conference on Machine Learning , pages 4040-4050. PMLR, 2021. 1, 2, 3, 6, 7, 8, 17

John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001. 1, 4

Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. 2002. 1

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 2020. 2, 4, 5, 14

BE Rhoades. Some fixed point iteration procedures. International Journal of Mathematics and Mathematical Sciences , 14(1):1-16, 1991. 2, 5

Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206.00941 , 2022. 2, 3, 4, 5, 15

Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. nti, Series , 2(9):12-16, 1968. 2, 3, 6, 16

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018. 2, 3, 6, 21

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018. 2, 7, 8

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016. 2, 7, 23

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 , 2018. 2, 7, 23

Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of gnns under heterophily: Are we really making progress? In The Eleventh International Conference on Learning Representations . 2, 7, 8, 23, 25

Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics , 33(14):i190-i198, 2017. 2, 8, 23

Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Learning iterative reasoning through energy minimization. In International Conference on Machine Learning , pages 55705582. PMLR, 2022. 2, 3, 8, 9, 22, 24, 26

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017. 3

Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. In International Conference on Machine Learning , pages 1407-1418. PMLR, 2021. 3

David Belanger and Andrew McCallum. Structured prediction energy networks. In International Conference on Machine Learning , pages 983-992. PMLR, 2016. 3

Colin Graber, Ofer Meshi, and Alexander Schwing. Deep structured prediction with nonlinear output transformations. Advances in Neural Information Processing Systems , 31, 2018.

Victor Garcia Satorras, Zeynep Akata, and Max Welling. Combining generative and discriminative models for hybrid inference. Advances in Neural Information Processing Systems , 32, 2019. 3

Julian Besag. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society: Series D (The Statistician) , 24(3):179-195, 1975. 3, 5

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021. 3

Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-LM improves controllable text generation. arXiv preprint arXiv:2205.14217 , 2022. 3

Emiel Hoogeboom, Vıctor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning , pages 8867-8887. PMLR, 2022. 3

Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687 , 2022. 3

Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Representations , 2023. 3

Joseph J Pfeiffer III, Jennifer Neville, and Paul N Bennett. Overcoming relational learning biases to accurately predict preferences in large scale networks. In Proceedings of the 24th International Conference on World Wide Web , pages 853-863, 2015. 5

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. 6

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning , pages 1530-1538. PMLR, 2015. 6

Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. arXiv preprint arXiv:2110.02910 , 2021. 6, 17

Fei Wang and Changshui Zhang. Label propagation through linear neighborhoods. In Proceedings of the 23rd international conference on Machine learning , pages 985-992, 2006. 7

Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin Benson. Combining label propagation and simple models out-performs graph neural networks. In International Conference on Learning Representations , 2020. 7

Jaemin Yoo, Hyunsik Jeon, and U Kang. Belief propagation network for hard inductive semi-supervised learning. In IJCAI , pages 4178-4184, 2019. 7

Hande Dong, Jiawei Chen, Fuli Feng, Xiangnan He, Shuxian Bi, Zhaolin Ding, and Peng Cui. On the equivalence of decoupled graph convolution network and label propagation. In Proceedings of the Web Conference 2021 , pages 3651-3662, 2021. 7

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning , pages 1725-1735. PMLR, 2020. 7

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems , 33:7793-7804, 2020. 7, 8

Jiong Zhu, Ryan A Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K Ahmed, and Danai Koutra. Graph neural networks with heterophily. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11168-11176, 2021. 7, 8

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations , 2021. 7, 8

Sunil Kumar Maurya, Xin Liu, and Tsuyoshi Murata. Improving graph neural networks with simple architecture design. arXiv preprint arXiv:2105.07634 , 2021. 7, 8

Xiang Li, Renyu Zhu, Yao Cheng, Caihua Shan, Siqiang Luo, Dongsheng Li, and Weining Qian. Finding global homophily in graph neural networks when meeting heterophily. arXiv preprint arXiv:2205.07308 , 2022. 7, 8

Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolutional networks. In AAAI . AAAI Press, 2021. 7, 8

Lun Du, Xiaozhou Shi, Qiang Fu, Xiaojun Ma, Hengyu Liu, Shi Han, and Dongmei Zhang. Gbk-gnn: Gated bi-kernel graph neural networks for modeling both homophily and heterophily, 2022. 7, 8

Xiyuan Wang and Muhan Zhang. How powerful are spectral graph neural networks. In International Conference on Machine Learning , pages 23341-23362. PMLR, 2022. 7, 8

Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arXiv:2009.03509 , 2020. 7, 8

Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems , 34:6695-6706, 2021. 9

Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder. In 8th ICML Workshop on Automated Machine Learning (AutoML) . 9

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015. 9

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations . 10

Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. Advances in Neural Information Processing Systems , 34:9061-9073, 2021. 17

Ryan Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Relational pooling for graph representations. In International Conference on Machine Learning , pages 4663-4673. PMLR, 2019. 18

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 22

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265 , 2019. 26

A Details of DPM on partially labeled graphs

A.1 Derivation of variational lower bound

In this section, we provide a detailed derivation of the variational lower bound in Equation (1).

A.2 Parameterization

Here, we provide more detailed parameterization for DPM-SNC. Following the Gaussian diffusion, we parameterize p ( y T ) and p θ ( y ( t -1) | G, y ( t ) ) as N ( y ( T ) ; 0 , I ) and N ( y ( t -1) ; µ θ ( y ( t ) , G, t ) , σ 2 t ) , respectively. Here, we set σ 2 t to β t . We also define µ θ ( y ( t ) , G, t ) as follows:

where α t = 1 -β t and ¯ α t = ∏ t i =1 α i . We implement the residual function ϵ θ ( y ( t ) , G, t ) through a GNN. The implementation details are described in Appendix C.

A.3 Detailed training objective

We describe the detailed training objective of L VLB for optimization. We first rewrite L VLB as follows: L VLB

where L ( t ) DPM is a training objective for a t step, and C is a constant with respect to the parameters θ . Following Ho et al. [16], we simplify L ( t ) DPM with a residual function ϵ θ ( y ( t ) , G, t ) in Equation (2).

where C is a constant with respect to the parameters θ . An additional suggestion from Ho et al. [16] is to set all weights of the mean squared error to one instead of β 2 t / 2 σ 2 t α t (1 -¯ α t ) , and we follow this suggestion in this paper.

A.4 Manifold-constrained sampling

Algorithm 2 Manifold-constrained sampling

1: Input: Graph G , node attributes x , labels y L , and temperature of randomness τ 2 .

3:

for

t

=

T

-

1

, . . . ,

0

do

4: Get z ∼ N ( z ; 0 , τ 2 I )

8:

9:

Set

Get

y

(

t

U

z

(

t

U

¯

y

(

z

2

11:

Set

∼ N

y

(

t

)

12: end for

13: return y (0) U

To sample y U from p θ ( y U | G, y L ) , we use a manifold-constrained sampling proposed by Chung et al. [18]. Here, the update rule of the reverse process for t = 0 , . . . , T -1 is defined as follows:

where z is sampled from N ( z ; 0 , τ 2 I ) . The Equation (3) is a temporal reverse diffusion step before applying the manifold-constrained samplings. The Equation (4) applies the manifold-constrained gradient γ ∂ ∂ y ( t +1) ∥ y L -ˆ y ( t +1) L ∥ 2 2 . Here, ˆ y ( t +1) is the label estimate in t +1 steps defined as follows:

where γ is a hyper-parameter. We set γ to 1 / ∥ y L -ˆ y ( t +1) L ∥ 2 2 . The Equation (5) is a projection step. Additionally, we introduce a parameter τ to control the randomness; when we set τ to zero, the modified reverse step becomes deterministic. This allows us to control the randomness in obtaining samples. We describe the detailed sampling algorithm in Algorithm 2.

y

;

(

t

L

)

0

, τ

)

∪

y

I

)

(

t

)

U

)

▷ Manifold-constrained gradient

▷ Projection step

▷ Initial sampling

B WLtest and GNN's expressiveness

In this section, we provide the proof of Theorem 1 in detail.

B.1 Preliminaries

Algorithm 3 1-dimensional Weisfeiler-Lehman algorithm

We denote set as {} , and multiset as { {} } , which is a set allowing duplicate elements. We represent the cardinality of set or multiset S as |S| . A graph is denoted as G = ( V G , E G , x G ) , where V G stands for the set of nodes, E G for the set of edges, and x G = { x G v : v ∈ V G } for the node-wise attributes. We also use N G ( v ) := { u ∈ V G : { v, u } ∈ E G } to denote the set of neighbor nodes of node v in graph G . We abbreviate a set of integers using the notation [ m ] := { 0 , . . . , m } . We also assume the hash functions are all injective and denote them by hash( · ) .

Definition 1 (Vertex coloring) . Vertex coloring χ G ( v ) is an injective hash function that maps a vertex v in graph G to a color c from an abstract color set C . Next, with a slight abuse of notation, we let the graph color χ G indicate the multiset of node colors in graph G , i.e., χ G := { { χ G ( v ) : v ∈ V G } } .

Now we explain the 1-dimensional Weisfeiler-Lehman (1-WL) algorithm [19], a classical algorithm to distinguish non-isomorphic graphs. At a high level, the 1-WL test iteratively updates node colors based on their neighbors until a stable coloring is reached. To be specific, given a vertex v , the initial node color χ 0 G ( v ) is set using an injective hash function on the node attribute x v . At each iteration, each node color is refined based on the aggregation of neighbor node colors, e.g., { { χ t -1 G ( u ) : u ∈ N G ( v ) } } . At the final T -th iteration, the algorithm returns the graph color χ T G . We provide a detailed description in Algorithm 3.

The 1-WL test allows to compare a pair of graphs G,H using the refined graph colors χ T G , χ T H . If the two graph colors χ T G and χ T H are not equivalent, two graphs G,H are guaranteed to be nonisomorphic. Otherwise, the test is inconclusive, i.e., the two graphs G,H are possibly isomorphic. We provide an example run of the 1-WL test in Figure 8(a), where the test is inconclusive.

Related to the 1-WL algorithm, we first prove a characteristic of it that will be later used in our proof.

̸

Lemma 1. Consider running 1-WL on two graphs G = ( V G , E G , x G ) and H = ( V H , E H , x H ) . If the initial graph colors of two graphs are distinct, i.e., χ 0 G = χ 0 H , the respective outputs of the 1-WL are also distinct, i.e., χ T G = χ T H .

̸

̸

Proof. We first prove that χ t G = χ t H is satisfied when χ t -1 G = χ t -1 H . The rest of the proof is straightforward by induction on t . To be specific, the 1-WL updates χ t G and χ t H as follows:

̸

̸

Since hash( · ) is an injective function, χ t G and χ t H are distinct for χ t -1 G = χ t -1 H .

B.2 AGG-WL test

Next, we describe the newly proposed AGG-WL test, which is an analog of our DPM-SNC. Similar to the 1-WL test, our AGG-WL test assigns node colors and iteratively refines them based on the

neighbor node colors. However, AGG-WL generates augmented graphs at initialization, creating multiple 'views' on the graph with diverse initial vertex coloring. Then it applies 1-WL on each of the augmented graphs to obtain the augmented graph colors. Finally, AGG-WL aggregates the augmented graphs colors.

The complete algorithm is described in Algorithm 4. Given a graph G = ( V G , E G , x G ) and the set of possible node-wise augmentations Z G , the algorithm generates augmented graph G m . In particular, the augmented graph G m is defined as follows:

where z m is an augmentation method in Z G . The algorithm creates an augmented graph G m by node-wise concatenation of z m to its node attributes x G , where ˜ x G m denotes the node-wise attribute augmented by z m . The symbol ·∥· denotes the concatenation of two elements. Furthermore, z 0 is a unique token, where G 0 has the same information as G . Also, with a slight abuse of notation, we let the graph color returned by the AGG-WL χ AGG G indicate the multiset of augmented graph colors, i.e., χ AGG G := { { χ T G m : m ∈ [ |Z G | ] } } . Here, |Z G | denotes the cardinality of set Z G .

Algorithm 4 Aggregation Weisfeiler-Lehman algorithm

) :

Our contribution is establishing the connection between the AGG-WL and the DPM-SNC: aggregation of refined graph colors for augmented graphs and marginalization over latent variables. More detail between DPM-SNC and the AGG-WL test is described in Appendix B.3.2.

We note that our algorithm bears some similarities with several previous studies. The DS-WL and DSS-WL test [41] defined a modified WL-test based on the set of subgraphs (instead of augmented graphs) with a modified edge set E ′ . Hang et al. [13] uses a collective algorithm to consider pseudolabels as additional inputs to boost GNN expressiveness. However, they rely on the assumption that one can find an 'optimal' pseudo-label to discriminate a pair of graphs, which may be hard to realize in practice. Zhange et al. [60] proposed labeling tricks that learns to capture dependence between nodes also by adding additional features, but mainly focus on link prediction in inductive settings.

B.3 Proof of Theorem 1

Let us start by restating Theorem 1.

Theorem 1. Let 1-WL-GNN be a GNN as powerful as the 1-WL test. Then, DPM-SNC using a 1-WL-GNN is strictly more powerful than the 1-WL-GNN in distinguishing non-isomorphic graphs.

Proof. We divide the proof into two parts, Appendices B.3.1 and B.3.2. In Appendix B.3.1, we show that the AGG-WL test is strictly more powerful than the 1-WL test. In Appendix B.3.2, we show that DPM-SNC using a 1-WL-GNN is as powerful as the AGG-WL test. From these two proofs, one can conclude that DPM-SNC using a 1-WL-GNN is strictly more powerful than 1-WL-GNNs in distinguishing non-isomorphic graphs.

B.3.1 Expressiveness of AGG-WL test

In this subsection, we prove that the AGG-WL test is strictly more powerful that the 1-WL test. The high-level idea is showing that (i) the AGG-WL test is at least as powerful as the 1-WL test, i.e., any two graphs distinguishable by the 1-WL test is also distinguishable by the AGG-WL test, and (ii) there exist two non-isomorphic graphs that are indistinguishable by the 1-WL test, but distinguishable by our AGG-WL test.

Lemma 2. AGG-WL is at least as powerful as WL in distinguishing non-isomorphic graphs, i.e., any two non-isomorphic graphs G,H distinguishable by WL are also distinguishable by AGG-WL.

̸

Proof. We first note that both 1-WL and AGG-WL can discriminate two non-isomorphic graphs with distinct sizes in a straightforward way. Therefore, we focus on non-trivial cases for two graphs G = ( V G , E G , x G ) , H = ( V H , E H , x H ) where the number of nodes and edges are same, i.e., |V G | = |V H | , |E G | = |E H | . The cardinality of the augmentation sets are also the same, |Z G | = |Z H | . We prove if these two graphs G,H are distinguishable by the 1-WL, two graphs G,H are also distinguishable by the AGG-WL, i.e., if χ T G = χ T H , then χ AGG G = χ AGG H .

̸

̸

̸

First, since χ T G = χ T H is satisfied, χ T G 0 = χ T H 0 is trivial. Additionally, the initial graphs colors of graph augmented by the unique token and any other augmentations are distinct, i.e., χ 0 G 0 = χ 0 H 1 , . . . , χ 0 H |Z H | . This implies χ T G 0 ̸∈ χ AGG H = { χ T H 0 , . . . , χ T H |Z H | } according to Lemma 1 which shows distinct initial graph colors produce the distinct outputs of 1-WL. Since χ T G 0 ∈ χ AGG G , one can see that χ AGG G = χ AGG H .

̸

Next, we show that there exist two graphs G,H indistinguishable by the 1-WL test, but distinguishable by the AGG-WL test. We first show a simple example from Figure 8(a), then for a family of circular skip link (CSL) graphs [61].

Lemma 3. Two graphs A,B are indistinguishable by the 1-WL, but distinguishable by the AGG-WL.

Proof. In Figure 8(a), two graphs A,B with 6 nodes are given. Here, the graph color computed by the 1-WL are equivalent, i.e., χ T G = χ T H (two light blue nodes and four gray nodes). Therefore, two graphs A,B are indistinguishable by the 1-WL.

Next, we prove that two graphs A,B are distinguishable by the AGG-WL. Since Lemma 1 shows 1-WL always computes a distinct graph color from distinct initial graph colors, we only need to show that there exists augmented graphs A m , B m with the same initial color has distinct graph colors computed by the 1-WL. Then, for two augmented graph sets { A m : m ∈ [ |Z V A | ] } and { B m : m ∈ [ |Z V B | ] } , we can show that the graph colors computed by AGG-WL are always distinct.

We show a simple case, where augmented graphs with non-zero augmentation on one node results distinct graph colors computed by the AGG-WL. We denote 'single binary node feature augmentation', z i as the case where node i is non-zero augmented and other nodes are augmented with zero, e.g., z 3 = { 0 , 0 , 1 , 0 , 0 , 0 } . In Figure 8(b) the non-zero augmented node i is colored in blue, and others are colored transparent. After applying the 1-WL algorithm on each graph, we obtain the graph color χ T A i and χ T B i for i ∈ { 1 , . . . , 6 } . It is clear that { { χ T A i : i ∈ { 1 , . . . , 6 }} } ̸ = { { χ T B i : i ∈ { 1 , . . . , 6 }} } , thereby two graphs A,B are distinguishable by AGG-WL.

Circular skip link graphs. CSL graph is denoted as CSL ( n, r ) , where n is the number of nodes, .i.e., V = { 0 , . . . , n -1 } and r is the skip connection length. For n and r , r < n -1 must hold. There exists 2 n edges, between node i and ( i +1) mod n forming a cycle, and between nodes i and ( i + r ) mod n forming a skip link for i ∈ { 0 , . . . , n -1 } .

Lemma 4. For n ≥ 8 , r ∈ [3 , n/ 2 -1] , two graphs CSL ( n, 2) and CSL ( n, r ) are indistinguishable by the 1-WL, but distinguishable by the AGG-WL.

Proof. We first give a brief proof of why CSL ( n, 2) and CSL ( n, r ) are indistinguishable by the 1-WL, then prove they are distinguishable by AGG-WL. We consider a non-trivial case where the node attributes are all same, i.e., χ 0 G ( v ) = χ 0 G ( u ) for v, u ∈ V G .

Let the initial color be c 0 for all the nodes for graphs CSL ( n, 2) and CSL ( n, r ) , χ 0 G ( v ) = c 0 ∀ v ∈ V G , G ∈ { CSL ( n, 2) , CSL ( n, r ) } . Then, for all nodes v ∈ V G , the color refinement process can

̸

(a) 1-WL

(b) AGG-WL

1

?

?

A

A1

A2

A3

A4

A5

A6

8

XA

XA1

XA2

XA5

x

XA4

6

B

B1

B2

B3

B4

B5

B6

XB

XB1

XB2

XB4

XB5

XB6

than be written as χ 1 G ( v ) = hash( c 0 , { { c 0 , c 0 , c 0 , c 0 } } ) . All nodes have identical colors c 1 for both graphs CSL ( n, 2) , CSL ( n, r ) , refining the initial graph color. Therefore CSL ( n, 2) and CSL ( n, r ) are indistinguishable by the 1-WL.

Now we consider the AGG-WL. Again, we only need to show that there exists augmented graphs A i , B j with the same initial color but having distinct graph colors refined by the 1-WL. Given symmetry, there is only one case of an augmented graph that adds a non-zero augmentation to one node. We use v i to denote the i -th node in graph and denote the augmented node as v 0 . We denote each augmented graph as CSL ( n, 2) 1 , CSL ( n, r ) 1 , and let the initial color χ 0 G ( v 0 ) = c 1 , and ∀ i ∈ { 1 , . . . , n -1 } , ∀ G ∈ { CSL ( n, 2) 1 , CSL ( n, r ) 1 } χ 0 G ( v i ) = c 0 .

Iteration 1. We focus on the four nodes connected to v 0 . Since r ∈ [3 , n/ 2 -1] , two node v n -r , v r are distinct. The color refinement can be written as following:

For v ∈ { v 1 , v 2 , v n -1 , v n -2 } , χ 1 CSL ( n, 2) 1 ( v ) = hash( c 0 , { { c 1 , c 0 , c 0 , c 0 } } ) = c 2 .

For v / ∈ { v 1 , v 2 , v n -1 , v n -2 } , χ 1 CSL ( n, 2) 1 ( v ) = hash( c 0 , { { c 0 , c 0 , c 0 , c 0 } } ) = c 3 .

For v ∈ { v 1 , v r , v n -1 , v n -r } , χ 1 CSL ( n,r ) 1 ( v ) = hash( c 0 , { { c 1 , c 0 , c 0 , c 0 } } ) = c 2 .

For v / ∈ { v 1 , v r , v n -1 , v n -r } , χ 1 CSL ( n,r ) 1 ( v ) = hash( c 0 , { { c 0 , c 0 , c 0 , c 0 } } ) = c 3 .

CSL(8,2)

XCSL(8,2)

T

CSL(8,3)

XCSL(8,3)

T

CSL(8, 2)1

XCSL(8,2)1

1

XCSL(8,2)1

2

CSL(8, 3)1

XCSL(8,3)1

1

XCSL(8,3)1

2

̸

Figure 9: Example on two non-attribute CSL (8 , 2) , CSL (8 , 3) graph. (a) 1-WL returns same graph color for two graphs, i.e., χ T CSL (8 , 2) = χ T CSL (8 , 3) . (b) Considering symmetry, there is only one type of augmented graph that with non-zero augmentation on one node, we denoted the augmented graph as CSL (8 , 2) 1 and CSL (8 , 3) 1 . In the process of 1-WL assigning colors to augmented graph, they result different graph color from the second iteration, and one can conclude χ T CSL (8 , 2) 1 = χ T CSL (8 , 3) 1 . Therefore CSL (8 , 2) , CSL (8 , 3) are distinguishable by the AGG-WL.

Since 1-WL showed χ 1 CSL ( n, 2) = χ 1 CSL ( n,r ) and augmented graphs showed χ 1 CSL ( n, 2) 1 = χ 1 CSL ( n,r ) 1 , two graphs are indistinguishable.

Iteration 2. Again, we focus on four nodes connected to v 0 . Here, we describe color refinement for nodes v 1 , v n -1 , v 2 , v n -2 , since the following is enough to prove two graph colors are distinct.

For v ∈ { v 1 , v n -1 } , χ 2 CSL ( n, 2) 1 ( v ) = hash( c 2 , { { c 1 , c 2 , c 2 , c 3 } } ) = c 4 .

For v ∈ { v 2 , v n -2 } , χ 2 CSL ( n, 2) 1 ( u ) = hash( c 2 , { { c 1 , c 2 , c 3 , c 3 } } ) = c 5 .

For v ∈ { v 1 , v n -1 } , χ 2 CSL ( n,r ) 1 ( v ) = hash( c 2 , { { c 1 , c 3 , c 3 , c 3 } } ) = c 6 .

̸

Since c 6 / ∈ χ 2 CSL ( n, 2) 1 , two graphs are distinguishable, i.e., χ t CSL ( n, 2) 1 = χ t CSL ( n,r ) 1 .

̸

Then χ T CSL ( n, 2) 1 = χ T CSL ( n,r ) 1 is satisfied as proved in Lemma 1, thereby two graphs CSL ( n, 2) and CSL ( n, r ) are distinguishable by the AGG-WL.

In Figure 9, we provide an example with CSL (8 , 2) and CSL (8 , 3) , indistinguishable by the 1-WL but distinguishable the AGG-WL.

B.3.2 Correspondence between DPM-SNC and AGG-WL

Now, we explain the connection between the DPM-SNC and the AGG-WL test. At a high-level, we show how DPM-SNC simulates the color refinement process of the AGG-WL. Our main idea stems from the marginalization in DPM-SNC, which can define the probability of the graph color χ with the marginalization over augmented graphs, i.e., p θ ( χ | G ) = ∫ p θ ( χ | G, z ) p ( z | G ) d z . 5 Specifically, DPM-SNCconsiders sample space of graph colors, where each member χ is obtained from p θ ( χ | G, z ) with an augmented graph G m represented by ( G, z m ) . By construction, DPM-SNC considers graph colors refinements for multiple augmented graphs, similar to how AGG-WL works.

Lemma 5. DPM-SNC using a 1-WL-GNN is as powerful as the AGG-WL test in distinguishing non-isomorphic graphs.

Proof. We show that if two graphs G,H are distinguishable by AGG-WL test, it is also distinguishable by the DPM-SNC. Assume that the two graphs G,H are distinguishable by the AGG-WL. Then, the following condition is satisfied.

̸

̸

where the AGG-WL produces distinct sets of refined graph colors for G,H . Next, we discuss how the DPM-SNC can distinguish G,H . To this end, we first assume a GNN which is as powerful as the 1-WL test [20], denoted as 1-WL-GNN. Specifically, we denote χ GNN G as a graph color refined by the 1-WL-GNN, where χ T G m = χ T H n implies χ GNN G m = χ GNN H n for any augmented graph pair G m , H n . Then, under the χ AGG G = χ AGG H , the following condition is also satisfied.

̸

̸

Here, we assume that DPM-SNC utilizes this 1-WL-GNN to output a graph color conditioned on an augmented graph, i.e., p θ ( χ | G, z ) . Then, we can connect p θ ( χ | G, z m ) with χ GNN G m as follows:

where I [ a = b ] is an indicator function whose value is 1 if a = b and 0 otherwise. Next, we also assume that p ( z | G ) is a uniform distribution over Z G . Then one can show that:

̸

Then it follows that p θ ( χ | G ) = p θ ( χ | H ) since { { χ GNN G m : m ∈ [ |Z G | ] } } = { { χ GNN H m : m ∈ [ |Z H | ] } } . Therefore, G,H are distinguishable by DPM-SNC.

̸

Our proof is valid when considering the multiple outputs. However, our practical implementation of DPM-SNC does not use multiple outputs at inference time since the DPM only considers the multiple random variables in the training objective. 6 To this end, in Appendix F, we also investigate the inference scheme which aggregates multiple outputs to make final predictions. 7

̸

C Implementation

In this section, we provide more details on how we implement the DPM-SNC for experiments.

C.1 Transductive settings

Here, we provide the detailed implementation of DPM-SNC for transductive node classification.

Model architecture. We parameterize the residual function ϵ θ ( y ( t ) , G, t ) of reverse diffusion step using a L -layer message-passing GNN as follows:

where g ( h ( L ) ) is a multi-layer perceptron that estimates the residual using the final node representation. AGGREGATE ( · ) and COMBINE ( · ) functions are identical to the backbone GNN, and ·∥· indicates the concatenation. Here, h 0 i is x i ∥ y ( t ) i . The f ( · ) is a sinusoidal positional embedding function [62]. We fix the dimension of sinusoidal positional embedding to 128.

Buffer construction. Following the temperature annealing approach of Qu et al. [7] in sampling pseudo-labels for optimization, we also control the temperature of randomness in obtaining y U from p θ ( y U | G, y L ) for buffer construction. To be specific, we use the variance multiplied by the temperature τ ∈ [0 , 1] , instead of the original variance in the reverse diffusion step, e.g., setting τ to zero makes the deterministic sampling.

Inference. To make final predictions y U from p θ ( y U | G, y L ) for evaluation, we eliminate the randomness in inference time, i.e., set temperature τ to zero. 8 If the target is one-hot relaxation of discrete labels, we also discretize the final prediction by choosing a dimension with maximum value.

C.2 Inductive setting

Here, we provide the details of DPM-SNC for inductive node classification and graph algorithmic reasoning. For the inductive node classification, we use the same model architecture as in the transductive setting and use the deterministic inference strategy. For the graph algorithmic, we modify DPM-SNC to perform edge-wise prediction.

Graph algorithmic reasoning. Since the targets of the graph algorithmic reasoning task are defined on the edge-level, we apply a diffusion process to the edge labels. We then recover the edge-level noisy labels through the reverse process.

The denoising model architecture in the reverse process has a similar architecture to the model architecture of IREM [26]. Specifically, the noisy edge target y ( t ) is updated as follows. First, the noisy edge labels y ( t ) and edge features are concatenated and passed through to the GNN layer, which aggregates them to obtain the node representation. Next, we apply element-wise addition of the time embedding vector to the node representation. Then, we concatenate a pair of node representations and noisy targets for the given edges and then apply a two-layer MLP to update edge labels.

In contrast to the node classification, we maintain randomness at inference time, i.e., we use the stochastic reverse process for obtaining edge labels. This approach is consistent with the IREM, which also includes randomness at inference time.

D Data statistics

D.1 Synthetic data

We generate 1000 × 2 non-attributed cyclic grid and 100 × 2 non-attributed cyclic grid for scattered and localized training nodes scenarios, respectively. Then, we split 30%, 30%, and 40% of the entire nodes into training, validation, and test nodes.

Scattered training nodes : We randomly sample nodes in the graph to split them into training, validation, and test nodes. 9

Localized training nodes : We select the nodes in the region within the 30 × 2 grid as training nodes. Then, we randomly sample the remaining nodes in the graph to split them into validation and test nodes.

For illustrative purposes, we also describe both scenarios in Figure 10 with smaller graphs.

(a) Scattered-training

(b) Scattered-test/validation

(c) Localized-training

(d)

Localized-

test/validation

D.2 Transductive node classification datasets

Here, we consider a graph with partially labeled nodes. We describe the data statistics in Table 6.

D.3 Inductive node classification datasets

Here, we consider datasets consists of a set of graphs. We describe the data statistics in Table 7.

D.4 Graph algorithmic reasoning datasets

Following Du et al. [26], we generate training graphs in each training step. Here, the training graphs are composed of graphs of varying sizes, ranging from two to ten nodes. The node features are initialized to zero, and the labels are defined on the edges, e.g., the shortest distance between two nodes. Then, we evaluate performance on graphs with ten nodes. Furthermore, we also use graphs with 15 nodes to evaluate generalization capabilities.

E Experiments setup

In this section, we describe the detailed experimental setup. For all experiments, we use a single GPU of NVIDIA GeForce RTX 3090. The hyper-parameters for each experiment are described in the following subsections.

E.1 Synthetic dataset

In this experiment, we implement each method with a one-layer GCN with 16 hidden dimensions. We search the learning rate within { 1e -3 , 5e -3 , 1e -2 } for all methods. Other hyper-parameters of each method follow their default settings. For DPM-SNC, we fix the diffusion step to 100 . We also set the size of the buffer to 50 and insert five samples into the buffer for every 30 training step. We use a pre-trained mean-field GNN until the buffer is updated 10 times.

E.2 Transductive node classification

Homophilc graph. We describe the hyper-parameter search ranges in Table 8. Additionally, we apply dropout with p = 0 . 5 except for LP. Other hyper-parameters of each method follow their default settings. For DPM-SNC, we fix the diffusion step to 80 . We also set the size of the buffer to 50 and insert five samples into the buffer for every 100 training step. We use a pre-trained mean-field GNN until the buffer is updated 20 times.

Heterophilic graph. Here, we describe the hyperparameter settings for DPM-SNC as we use the numbers reported by Plantov et al. [24] for baselines. We describe the hyper-parameter search ranges in Table 9. Additionally, we apply dropout with p = 0 . 5 , and we fix the diffusion step to 80 . We also set the size of the buffer to 50 and insert five samples into the buffer for every 100 training step. We use a pre-trained meanfield GNN until the buffer is updated 100 times.

E.3 Inductive node classification

We describe the hyper-parameter search ranges in Table 10. For the small-scale graph datasets, i.e., Pubmed, Cora, and Citeseer, we apply dropout with p = 0 . 5 . For the huge-scale graph datasets, i.e., PPI, we include the linear skip connection between each GNN layer. Other hyper-parameters of each method follow their default settings. For DPM-SNC, we fix the diffusion step to 80 .

E.4 Algorithmic reasoning

Here, we describe the hyper-parameter settings for DPM-SNC as we use the numbers reported by Du et al. [26] for baselines. We search the learning rate and weight decay within { 1e -4 , 3e -4 , 1e -3 } and { 0 , 1e -5 , 1e -4 } , respectively. The hyper-parameters of the model are the same as the model implementation of IREM, using a three-layer GINEConv [63] with a 128 hidden dimension. We fix the diffusion step to 80 .

F Additional experiments

Ablations on various inference schemes. We also study various inference strategies for our DPM-SNC. We first investigate how temperature control affects label inference in real-world node classification tasks. In Figure 11(a), we plot the changes in accuracy for various temperatures τ . One can see that reducing the randomness of DPM-SNC gives a better prediction in real-world node classification tasks.

We also consider sampling various numbers of predictions for node-wise aggregation to improve performance. In Figure 11(b), even the number of samples is increased to 2 10 , the deterministic and the node-wise aggregation inference schemes show similar accuracy, and there are only minor performance improvements on Citeseer. In practice, we use deterministic inference in the node classification tasks since the node-wise aggregation requires a relatively long time.

0.00

0.25

0.50

0.75

1.00

Temperature

50

60

70

80

Pubmed

Citeseer

(a)

2 0

2 2

2 4

2 6

2 8

2 10

Number of samples

50

60

70

80

Pubmed

Citeseer

(b)

Peer reviews:
comments: SUMMARY:
This paper focuses on structured node classification on graphs by integrating diffusion probabilistic models to generate label distributions for unknown nodes. The authors design a novel training algorithm to apply DPMs, and a new variational lower bound. Theoretical analysis proves the proposed model is strictly more powerful than the classic 1-WL test. Experimental results also show its superiority over baseline methods on both inductive and transductive settings.

STRENGTHS:
- The paper is well written and structured.
- Application of diffusion problistic models in node classification is novel.
- The theoredical analysis of 1-WL test is convincing.

WEAKNESSES:
- The use of diffusion problistic model is not well motivated? The reason for using diffusion process for partially labeled graphs is not explained.
- It lacks a analysis of computational complexity of DPM-SNC, since it requires multiple steps of sampling.

QUESTIONS:
- Please explain the reason of introducing diffusion problistic models. How can diffusion process improves GNNs on partially labeled graphs?
- What is the computational complexity of DPM-SNC?

LIMITATIONS:
Please refer to the weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the issue of structured node classification on graphs that consider dependencies between node labels. While GNNs have shown great success in machine learning on graphs, they predict node labels independently.\nThe authors propose a novel framework, DPM-SNC (Diffusion Probabilistic Models for Structured Node Classification), in which they use DPMs to learn a joint label distribution over both known and unknown labels while making predictions based on known labels using manifold-constrained sampling. Additionally, they present a novel training algorithm for DPMs. Theoretical analysis shows that it enhances the expressive power of GNNs. Experiments across multiple settings demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS:\n**Originality**\n\nThis paper ...\n- combined GNNs and DPMs for structured node classification.\n- proposed using manifold-constraint sampling for the transductive setting.\n- gave a novel expressivity metric, AGG-WL and proved that DPM-SNC is more powerful than its backbone 1-WL-GNN.\n\n**Quality**\n\nIn general, the math is solid and the claims are intuitive and/or empirically validated. The experiments are extensive.\n\n**Clarity**\n\nThe paper is well-written and easy to follow.\n\n**Significance**\n\nThe paper makes interesting and inspiring theoretical connections. Realistically, I am not so sure that this technique would be adopted in a production setting considering its cost and complexity.\n\nWEAKNESSES:\nNo major flaws. See questions.\n\nQUESTIONS:\n1. What are the training and inference costs of the proposed method? How do they compare to the baselines?\n2. Regarding model training, do you use the same objective as DDPM to optimize the GNN parameters? Based on my understanding of your paper and code (correct me if I am wrong), the $q(y_U|y_L)$ term in the objective is not used to update $\\theta$. Instead, it is optimized by updating the buffer $\\mathcal{B}$. Maybe the \"Training algorithm\" paragraph could be revised so that it better reflects the proposed method?\n3. Among all kinds of generative models, why do you choose DPMs?\n\nLIMITATIONS:\nI do not see any significant, unreported negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes a diffusion probability model learning method for structured node classification, named DPM-SNC. The contributions of this paper are as follows. First, this paper leverages the high capacity of DPMs in learning joint label dependency and predicting posterior distribution conditioned on partially known data by devising DPM-SNC. Second, this paper introduces a new variational lower bound to train a DPM on partially labelled graphs. Moreover, this paper proposes AGG-WL test to theoretically analyse the expressive power of DPM-SNC using 1-WL-GNNs, which is more powerful than 1-WL-GNNs. Finally, detailed experiments on both transductive and inductive settings demonstrate DPM-SNC’s impressive performance.

STRENGTHS:
There are strengths of proposed method:
1.	Applying DPM to learning partially labeled node label, to the best of our knowledge, is novel.

2.	The experiments are detailed analyzed with both synthetic and real world data, which points out the strength of the method in different scenarios. The algorithmic reasoning shows the influence of different algorithmic elements in DPM, with great performance compared with baseline.

WEAKNESSES:
However, there are several drawbacks:
1.	The idea to apply DPM to learn the joint distribution over the labels is novel. However, this paper has little novelty to DPM itself, which utilizes manifold-constrained sampling from previous works.
2.	The analysis for expressive power follows the color assignments description in 1-WL test. The details of AGG-WL, the perturbations, and the aggregation need more explanation.

QUESTIONS:
1.	Add more explanation to AGG-WL.

LIMITATIONS:
Authors have adequately addressed the limitations in this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the use of diffusion probabilistic models (DPM) to model node-wise label dependencies for structured node classification. The proposed DPM-SNC generalizes diffusion probabilistic models to work on partially labeled graphs in the transduction settings. The proposed model is proved to be more expressive than the classic 1-WL test. Experimental results demonstrate the effectiveness of the proposed model.\n\nSTRENGTHS:\n1. This paper leverages the merits of diffusion probabilistic models (DPM) for structured node classification. A new training algorithm is proposed, which generalises DPM to work on partially labeled graphs.\n2. The theoretical analysis proves the improved expressive power of the proposed model.\n3. The design of experiments is thorough and the reported results demonstrate the effectiveness of the proposed model on various graph-based tasks.\n\nWEAKNESSES:\n1. The idea of using diffusion models is borrowed from the field of computer vision, where the diffusion and reverse process is more intuitive. However, why structured node classification would benefit from the diffusion and reverse process? Why the reverse process can be defined as a Gaussian distribution on graphs? \n\n2. It is unclear how the proposed model is applied to inductive node classification and what benefits the diffusion model can offer in inductive settings. \n\n3. As observed from Table 4, the performance of the proposed method is comparable or with limited improvements. More elaborations on these points are necessary. Also, the datasets used for inductive settings (except for PPI) are mostly used for transductive settings. Experiments on other relevant datasets should be added for inductive settings.\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nNo potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new framework called the Diffusion Probabilistic Model for Structured Node Classification (DPM-SNC), which leverages diffusion probabilistic models for structured node classification on graphs. The framework aims to account for dependencies between node labels. The two key aspects of DPM-SNC are: (a) learning a joint distribution over the labels using an expressive reverse diffusion process and (b) making predictions based on known labels using manifold-constrained sampling. The authors test DPM-SNC in diverse scenarios, including partially labeled graphs in a transductive setting, as well as unlabeled graphs and in an inductive setting.\n\nSTRENGTHS:\n1. The paper effectively applies probabilistic diffusion, one of the hottest techniques in the community, to the structured node classification problem.\n2. DPM-SNC can be used in combination with any graph neural network (GNN) and to enhance performance on node classification tasks. This improvement is not solely empirical, as the authors provide theoretical analysis tying it to enhanced theoretical expressive power.\n3. The authors conduct a comprehensive experimental evaluation, showing DPM-SNC's superior performance in both transductive and inductive node classification tasks.\n\nWEAKNESSES:\n1. Although it is understandable that certain technical details are relegated to the appendix due to space limitations, the main paper could benefit from being more self-contained. This could be achieved by shortening the introduction or reducing some figures. Without referring to the appendix, especially for readers without expertise in diffusion models, it's difficult to grasp exactly how DPM-SNC makes predictions.\n2. The paper lacks a discussion or comparison with similar works that combine deep neural networks and label propagation for structured node classification [1, 2]. These works could be used as additional baselines or in discussions related to the proposed method.\n3. It appears that a limitation of the proposed method is that the diffusion process operates on (noisy versions of) labels of nodes, not their representation vectors. As a result, while DPM-SNC seems to work well for node classification problems, it's unclear whether the trained model could be effectively used as a general node embedding model for solving edge- or graph-level problems, or multi-label classification problems.\n\n[1] Huang et al., Combining Label Propagation and Simple Models Outperforms Graph Neural Networks, ICLR 2021\n[2] Yoo et al., Belief Propagation Network for Hard Inductive Semi-Supervised Learning, IJCAI 2019\n\nQUESTIONS:\n1. Why did the authors exclusively use GCNII for the proposed method and not for the competitor models?\n\nLIMITATIONS:\nThe authors have acknowledged the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}