Title: Directional diffusion models for graph representation learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion models

graph representation learning

unsupervised learning
Abstract: Diffusion models have achieved remarkable success in diverse domains such as image synthesis, super-resolution, and 3D molecule generation. Surprisingly, the application of diffusion models in graph learning has garnered little attention. In this paper, we aim to bridge this gap by exploring the use of diffusion models for unsupervised graph representation learning. Our investigation commences with the identification of anisotropic structures within graphs and the recognition of a crucial limitation in the vanilla forward diffusion process when dealing with these anisotropic structures. The original forward diffusion process continually adds  isotropic Gaussian noise to the data, which may excessively dilute anisotropic signals, leading to rapid signal-to-noise conversion. This rapid conversion poses challenges for training denoising neural networks and obstructs the acquisition of semantically meaningful representations during the reverse process. To overcome this challenge, we introduce a novel class of models termed {\it directional diffusion models}.  These models adopt data-dependent, anisotropic, and directional noises in the forward diffusion process. In order to assess the effectiveness of our proposed models, we conduct extensive experiments on 12 publicly available datasets, with a particular focus on two distinct graph representation learning tasks. The experimental results unequivocally establish the superiority of our models over state-of-the-art baselines, underscoring their effectiveness in capturing meaningful graph representations. Our research not only sheds light on the intricacies of the forward process in diffusion models but also underscores the vast potential of these models in addressing a wide spectrum of graph-related tasks. Our code is available at \url{https://github.com/statsle/DDM}.

Directional diffusion models for graph representation learning

Abstract

Diffusion models have achieved remarkable success in diverse domains such as image synthesis, super-resolution, and 3D molecule generation. Surprisingly, the application of diffusion models in graph learning has garnered little attention. In this paper, we aim to bridge this gap by exploring the use of diffusion models for unsupervised graph representation learning. Our investigation commences with the identification of anisotropic structures within graphs and the recognition of a crucial limitation in the vanilla forward diffusion process when dealing with these anisotropic structures. The original forward diffusion process continually adds isotropic Gaussian noise to the data, which may excessively dilute anisotropic signals, leading to rapid signal-to-noise conversion. This rapid conversion poses challenges for training denoising neural networks and obstructs the acquisition of semantically meaningful representations during the reverse process. To overcome this challenge, we introduce a novel class of models termed directional diffusion models . These models adopt data-dependent, anisotropic, and directional noises in the forward diffusion process. In order to assess the effectiveness of our proposed models, we conduct extensive experiments on 12 publicly available datasets, with a particular focus on two distinct graph representation learning tasks. The experimental results unequivocally establish the superiority of our models over state-of-the-art baselines, underscoring their effectiveness in capturing meaningful graph representations. Our research not only sheds light on the intricacies of the forward process in diffusion models but also underscores the vast potential of these models in addressing a wide spectrum of graph-related tasks. Our code is available at https://github.com/statsle/DDM .

1 Introduction

Unsupervised representation learning through diffusion models has emerged as a prominent area of research in computer vision. Several methods that leverage diffusion models, such as those proposed by Zhang et al. (2022); Preechakul et al. (2022); Abstreiter et al. (2021); Baranchuk et al. (2021), have been put forth for representation learning. Notably, Baranchuk et al. (2021) showed that the intermediate actiations obtained from denoising networks contain valuable semantic information. Their findings emphasize the effectiveness of diffusion models in learning meaningful

Yuling Yang

SUFE

Qiang Sun ∗

y

0.05

0.00

0.05

x

0.06

0.04

0.02

0.00

0.02

0.04

0.06

0.08

label

airplane

automobile

(a) CIFAR-10

0.4

0.2

0.0

0.2

0.4

x

0.6

0.4

0.2

0.0

0.2

0.4

0.6

y

label

0

1

(b) Amazon-Photo

0.4

0.2

0.0

0.2

0.4

x

0.6

0.4

0.2

0.0

0.2

0.4

0.6

y

label

0

1

2

(c) IMDB-M

visual representations. More recently, Choi et al. (2022) showed that restoring data corrupted with certain noise levels offers a proper pretext task for the model to learn rich visual concepts.

Despite the increasing research on diffusion models in computer vision, there is still a noticeable shortage of studies exploring the use of diffusion models in graph learning. Previous works, such as those by Haefeli et al. (2022) and Jo et al. (2022), have primarily concentrated on employing diffusion models for generating discrete graph structures. However, the realm of graph representation learning, a fundamental and challenging aspect of graph learning, has yet to harness the potential of diffusion models. To effectively adapt and incorporate diffusion models into graph representation learning, and to facilitate progress in this domain, it is imperative to identify and understand the obstacles hindering the application of diffusion models.

To gain insights into the limitations of the vanilla diffusion models which are commonly adopted for image generation tasks (Ho et al., 2020), we first investigate the underlying structural disparities between images and graphs. In particular, we employ singular value decomposition (SVD) on both image and graph data, and visualize the projected data in a 2-dimensional plane, as shown in Figure 1. This figure illustrates that the projected data from Amazon-Photo and IMDB-M exhibit strong anisotropic structures along only a few directions, while the projected images from CIFAR-10 form a relatively more isotropic distribution within a circular shape centered around the origin. This observation strongly implies that graph data may feature distinctive anisotropic and directional structures that are less prevalent in image data. As we will subsequently demonstrate, the vanilla diffusion models with isotropic forward diffusion processes result in a rapid decrease in the signal-tonoise ratios (SNRs), which in turn diminishes their effectiveness in learning anisotropic structures. Therefore, it is imperative to develop new approaches that can effectively account for these anisotropic structures.

This paper presents directional diffusion models as a solution for better learning anisotropic structures. Our approach involves incorporating data-dependent and directional noise into the forward diffusion process, effectively mitigating the challenge of rapid signal-to-noise ratio deterioration. The intermediate activations obtained from the denoising network excel at capturing valuable semantic and topological information crucial for downstream tasks. Consequently, the proposed directional diffusion models offer a promising approach for generative graph representation learning. Numerically, we perform experiments on 12 benchmark datasets covering both node and graph classification tasks. The results consistently highlight the superior performance of our models when compared to state-of-the-art contrastive learning and generative approaches (Hou et al., 2022). Notably, for graph classification problems, our directional diffusion models even surpass supervised baseline methods, underscoring the considerable potential of diffusion models in the field of graph representation learning.

Our main contributions are summarized as follows.

We contribute to the exploration of anisotropic structures in graph data, being among the pioneers in the literature. We demonstrate that the vanilla forward diffusion process with isotropic white noise leads to a rapid decline in signal-to-noise ratios for graph learning problems. This issue hampers the ability of denoising networks to extract fine-grained feature representations across a wide range of SNRs.

We propose novel directional diffusion models specifically designed for graph data, incorporating data-dependent and directional noise in the forward diffusion process. Our proposed models effectively address the issue of the rapid decline of SNRs, enabling better graph representation learning.

Numerically, our proposed directional diffusion models outperform state-of-the-art selfsupervised methods and even supervised methods on 12 benchmark datasets. Additionally, we provide comprehensive ablation studies to gain a deeper understanding of the mechanisms underlying directional diffusion models.

2 Related work

Graph representation learning Graph representation learning aims to embed nodes or entire graphs into a low-dimensional vector space, where the structural and relational properties can be used for downstream tasks. Two prevalent paradigms for graph representation learning are contrastive learning and generative self-supervised learning. Contrastive learning approaches such as DGI (Velickovic et al., 2019), Infograph (Sun et al., 2019), GraphCL (You et al., 2020), GRACE (Zhu et al., 2020), and GCC (Qiu et al., 2020), have achieved promising results in some particular graph learning tasks. These methods leverage local-global mutual information maximization for node and graph representation learning. GraphCL learns node embeddings that are invariant to graph-level transformations, while GRACE and GCC use subgraph sampling and graph perturbation to create augmented pairs. Generative self-supervised learning aims to recover masked components of the input data through approaches such as GraphMAE (Hou et al., 2022), a masked graph autoencoder that focuses on feature reconstruction by utilizing a masking strategy and scaled cosine error. This method outperforms self-supervised learning baselines, and revitalizes the concept of generative self-supervised learning on graphs. GPT-GNN (Hu et al., 2020b) is a recent approach that introduces a self-supervised attributed graph generation task to pre-train a GNN so that it can capture the structural and semantic properties of the graph.

Denoising diffusion probabilistic models Denoising diffusion probabilistic models (Ho et al., 2020; Song et al., 2020), or simply diffusion models, are a class of probabilistic generative models that turn noise to a data sample and thus are mainly used for generation tasks (Dhariwal and Nichol, 2021; Rombach et al., 2022).Recently, diffusion models have been used as a representation learning toolbox in computer vision (Preechakul et al., 2022; Abstreiter et al., 2021; Baranchuk et al., 2021). For instance, Preechakul et al. (2022) proposed Diff-AE, a method that concurrently trains an encoder to discover high-level semantics and a conditional diffusion model that uses these representations as input conditions. Abstreiter et al. (2021) augmented the denoising score matching framework to enable representation learning without any supervised signal. Recently, diffusion models have also found applications in handling graph data. Haefeli et al. (2022) showed that diffusion models for graphs benefited from discrete state spaces. Jo et al. (2022) proposed the graph diffusion using stochastic differential equations. To the best of our knowledge, there have been no works for diffusion-model-based graph representation learning.

3 The effect of anisotropic structures

As discussed in the introduction section, significant structural differences exist between graphs and natural images. In addition to the analysis of the Amazon-Photo and IMDB-M datasets, we conducted similar examinations on all other graph benchmark datasets, and the detailed results can be found in the appendix. Moreover, it is important to highlight that the anisotropic structures, referred to as categorical directional dependence, are frequently encountered in natural language data (Gao et al., 2019; Li et al., 2020). It is intriguing to observe that diffusion models have not yet made significant strides in the realm of natural language processing. This further underscores the significance of

0

200

400

600

800

1000

step

0.0

0.2

0.4

0.6

0.8

SNR

white noise

directional noise

(a) Amazon-Photo

0

200

400

600

800

1000

step

0.0

0.5

1.0

1.5

SNR

white noise

directional noise

(b) IMDB-M

investigating and tackling the challenges posed by anisotropic structures within the framework of diffusion models.

This section delves deeper into the examination of how anisotropic structures in graphs impact the effectiveness of vanilla diffusion models. In the vanilla forward diffusion process, isotropic Gaussian noise is sequentially added to the raw data point x 0 ∼ q ( x 0 ) until it transforms into isotropic white noise N (0 , I ) 2 . This approach is reasonable when the data follow isotropic distributions, as it gradually transforms the data into noise, generating a sequence of noisy samples with a wide range of SNRs. However, in cases where the data exhibits anisotropy, the addition of isotropic noise can quickly contaminate the data structure, causing the SNRs to rapidly approach zero. Consequently, denoising networks struggle to extract fine-grained feature representations.

To explore the impact of introducing isotropic noise on the learning of anisotropic graphs, we conduct an experiment to measure the SNRs for both node and graph classification tasks at each forward step. We observe how these SNRs change throughout the forward diffusion process. Initially, we pre-train a graph neural network (GNN) denoted as E to serve as a feature extractor, projecting the graph data into a linearly separable space. Subsequently, we optimize the weight vector w ∈ R d in this linear space using Fisher's linear discriminant analysis. The weight vector w is then employed to compute the SNR at each forward diffusion step, where SNR = w T S B w/w T S W w . In this equation, S B represents the scatter between class variability, and S W represents the scatter within-class variability. This SNR quantifies the discriminative power of the learned representations at different stages of the diffusion process.

We conducted this experiment on all graph benchmark datasets to assess the impact of isotropic noise on learning anisotropic graphs. Here, we present the results for IMDB-M and Amazon-Photo in Figure 2a, while the additional results can be found in the appendix. In Figure 2a, we observe that for anisotropic graph data and isotropic noise, the SNR rapidly decreases to 0 at around 50 steps for Amazon-Photo and 400 steps for IMDB-M. Furthermore, the SNR remains close to 0 thereafter, indicating that the incremental isotropic white noise quickly obscures the underlying anisotropic structures or signals. Consequently, the denoising networks are unable to learn meaningful and discriminative feature representations that can be effectively utilized for downstream classification tasks. In contrast, when utilizing our directional diffusion models, which incorporate a data-dependent and directional forward diffusion process (to be introduced later), the SNR declines at a slower pace. This slower decline enables the extraction of fine-grained feature representations with varying SNRs, preserving the essential information of the anisotropic structures.

Overall, these studies underscore the significance of considering anisotropic data structures when designing forward diffusion processes and the corresponding diffusion models, especially in the context of graph data.

Forward Process with Directional Noise

𝑡 = 0

𝑡 = 100

𝑡 = 500

𝑡 = 800

𝑡 = 1000

𝑡 = 0

𝑡 = 100

𝑡 = 500

𝑡 = 800

𝑡 = 1000

Forward Process with White Noise

4 Directional diffusion models

In this section, we start by introducing the requisite notation. Subsequently, we propose the directional diffusion models (DDMs), with a specific focus on their application to graph representation learning. Furthermore, we discuss how to extract feature representations from DDMs, a pivotal aspect for downstream tasks.

Notation Wedenote a graph by G = ( V , A , X ) , where V is the node set, N = | V | is the node number, A ∈ R N × N is the adjacency matrix (binary or weighted), and X = ( x 1 , x 2 , · · · , x N ) T ∈ R N × d is the node feature matrix. Our goal is to learn a network, denoted as f : R N × d × R N × N → R N × d h , to encode graph features into representations H = ( h 1 , h 2 , · · · , h N ) T ∈ R N × d h , where h i ∈ R d h is the representation for node i . Mathematically, we have H = f ( X , A ) . These representations can subsequently be used for downstream tasks, such as graph and node classification.

Directional diffusion models In the preceding section, our investigation uncovered a pivotal factor contributing to the underwhelming performance of vanilla diffusion models in graph learning: the swift deterioration of signal-to-noise ratios. To tackle this challenge, we introduce the directional noise in the forward diffusion process, which involves transforming the isotropic Gaussian noise into an anisotropic noise by incorporating two additional constraints. These two constraints play a vital role in enhancing the efficacy of vanilla diffusion models.

Let G t = ( A , X t ) be the noisy graph at the t -th forward diffusion step, where X t = { x t, 1 , x t, 2 , ..., x t,N } represents the learned features at the t -th step. Specifically, the node feature x t,i ∈ R d of node i at time t is obtained as follows:

where x 0 ,i is the raw feature vector of node i , µ ∈ R d and σ ∈ R d consist of the means and standard deviations of the d features across N nodes, respectively. The symbol ⊙ denotes the Hadamard product. During the mini-batch training, µ and σ are calculated using graphs within the batch. The parameter ¯ α t := ∏ t i =0 (1 -β i ) ∈ (0 , 1) represents the variance schedule (Ho et al., 2020) and is parameterized by a decreasing sequence { β 1: T ∈ (0 , 1) } .

In contrast to the conventional forward diffusion process, our directional diffusion models integrate two additional constraints, as denoted in equations (2) and (3). The second constraint, (3), transforms the data-independent Gaussian noise into an anisotropic and batch-dependent noise. In this constraint, each coordinate of the noise vector shares the same empirical mean and empirical standard deviation as the corresponding coordinate in the data within the same batch. This constraint confines the diffusion process to the local neighborhood of the batch, preventing excessive deviation from the batch and preserving local coherence.

The first constraint, (2), aligns the noise ϵ ′ with the feature x 0 ,i to ensure that they share the same coordinate signs. This guarantees that adding noise does not result in noisy features pointing in the opposite direction of x 0 ,i . By preserving the directionality of the original feature, this constraint plays a crucial role in maintaining the inherent data structure during the forward diffusion process. These two constraints work in tandem to ensure that the forward diffusion process respects the underlying data structure and mitigates the rapid degradation of signals. Consequently, the SNR decays slowly, enabling our directional diffusion models to effectively extract meaningful feature representations across various steps. This, in turn, enhances the utility of these representations in downstream tasks by providing reliability and informativeness.

To illustrate the impact of directional noise, we refer to the experiments conducted in Section 3. Our newly proposed "directional noise" ensures a smoother decline of the SNRs throughout the diffusion process, confirming our initial intuition. To further visualize the differences between using directional noise and isotropic noise in the forward diffusion process, we conducted simulations on two ellipses and sequentially added noise, as depicted in Figure 3. The figure clearly illustrates the distinct behaviors exhibited by the two types of noise. With directional noise, the samples maintain a clear decision boundary, indicating the preservation of discriminative structures during the diffusion process. Conversely, samples with white noise quickly blend into pure noise, leading to the loss of discriminative information. This visual comparison unequivocally underscores the superiority of directional noise in preserving the structural information of the data during the forward diffusion process.

Model architecture We follow the same training strategy as in the vanilla diffusion models, where we train a denoising network f θ in the reverse diffusion process. Since the posterior of the forward process with directional noise cannot be expressed in a closed form, we borrow the idea from Bansal et al. (2022); Li et al. (2022) and let the denoising model f θ directly predict X 0 . The loss function L is defined as the expected value of the Euclidean distance between the predicted feature representation f θ ( X t , A , t ) and the original feature representation X 0 :

This loss function ensures that the model predicts X 0 at every step.

To parameterize the denoising network f θ , we employ a symmetrical architecture inspired by the successful UNet architecture in computer vision (Dhariwal and Nichol, 2021). Figure 4 illustrates our DDM framework, comprising four GNN layers and one multilayer perception (MLP). The first two GNN layers function as the encoder , responsible for denoising the target node by aggregating neighboring information. The final two GNN layers serve as the decoder , mapping the denoised node features to a latent code and smoothing the latent code among neighboring nodes. To address the potential issue of over-smoothing and account for long-distance dependencies in the graph, we introduce skip-connections between the encoder and decoder. The algorithm, along with the mini-batch training procedure, is presented in the appendix.

Learning representations For a given graph G = ( A , X ) , the learned node-level representations are obtained from the activations of the denoising network f θ at user-selected time steps. It is important to note that we only utilize the activations from the decoder of f θ since they incorporate the encoder activations through skip connections. As depicted in Figure 4, at each time step k , we introduce k steps of directional noise following (1) and employ the denoising network f θ to denoise and encode the noisy data X k . The decoder of f θ maps the denoised node features to a latent code while smoothing the latent code among neighboring nodes. We extract the activations from the decoder of f θ and concatenate them to obtain H k = { h k, 1 , h k, 2 , · · · , h k,N } ∈ R N × d h . The complete pipeline is presented in the appendix.

directional noise

𝜖

!

"

⋮

Forward Process

𝑡 = 0,1, ⋯ , 𝑇

time embedding

+

MLP

Representation

Denoising Process

+

Embedding step

Skip connection

Concatenate

MLP

5 Experiments

This section provides an evaluation of the directional diffusion models from two perspectives. First, we compare our models with existing state-of-the-art methods on various graph learning tasks, including node and graph classification tasks. This allows us to assess the effectiveness of our approach for graph representation learning problems. Second, we conduct several studies to gain a better understanding of the effect of our directional noise and evaluate the necessity of our design choices.

In all experiments, we follow a two-step process. First, we pre-train a DDM on the dataset in an unsupervised manner. Then, we extract feature representations from diffusion steps 50 , 100 , 200 using the pre-trained model. Although this approach is inspired by the experimental results and insights from Section 3, it is deliberately not fine-tuned for each dataset. Ideally, fine-tuning with carefully selected steps for each dataset could further improve the performance.

5.1 Graph classification

To demonstrate the effectiveness of our method, we conduct comparisons with state-of-the-art (SOTA) unsupervised learning methods, which include GCC (Qiu et al., 2020), Infograph (Sun et al., 2019), GraphCL (You et al., 2020), JOAO (You et al., 2021), MVGRL (Hassani and Khasahmadi, 2020), and GraphMAE (Hou et al., 2022). We also compare our approach with supervised learning methods, specifically GIN (Xu et al., 2018) and DiffPool (Ying et al., 2018). These experiments are conducted on seven widely-used datasets, namely MUTAG, IMDB-B, IMDB-M, PROTEINS, COLLAB, and REDDIT-B (Yanardag and Vishwanathan, 2015). Node degrees are used as initial node features for IMDB-B, IMDB-M, REDDIT-B, and COLLAB, while node labels are employed for MUTAG and PROTEINS, in accordance with prior literature (Hou et al., 2022). We extract graph-level representations at various steps, train linear SVMs using LIBSVM (Chang and Lin, 2011), and independently test them. The final predictions are obtained by majority vote, and we report the average accuracy and standard deviation, calculated after five runs. Additional details on hyper-parameters can be found in the appendix.

Table 1 presents the results, clearly indicating that our DDM achieves the most competitive, if not the best, performance across all benchmark datasets. Notably, our DDM outperforms even the supervised approaches in certain experiments, such as IMDB-B, COLLAB, and MUTAG. This remarkable performance can be attributed to two key factors. First, from a data perspective, these datasets have limited information in their node features, which can hinder the accuracy of supervised learning (Hou et al., 2022). By leveraging directional noise diffusion, our DDM effectively acts as a pseudo-infinite-step data augmentation technique, generating numerous samples while preserving the classification boundary. This augmentation significantly enhances the effectiveness of unsupervised learning. Second, from a model perspective, the DDM framework harnesses the power of directional

noise and ensures that the learned representations capture discriminative information by preventing the rapid decay of signal-to-noise ratios.

5.2 Node classification

To assess the quality of the node-level representations produced by our method, we conduct evaluations of DDM on six standard benchmark datasets: Cora, Citeseer, PubMed (Yang et al., 2016), Ogbn-arxiv (Hu et al., 2020a), Amazon-Computer (Zhang et al., 2021), and Amazon-Photo (Zhang et al., 2021). We follow the publicly available data-split schema and employ the evaluation protocol used in the literature. Graph-level representations are extracted at different diffusion steps, and an independent linear classifier is trained for each step. The final predictions are determined through majority voting, and we report the mean accuracy on the test nodes. Additional details regarding hyperparameters can be found in the appendix.

We compare DDM with state-of-the-art generative unsupervised models, specifically GPT-GNN (Hu et al., 2020b) and GraphMAE (Hou et al., 2022). Furthermore, we include the results of contrastive unsupervised models for comparison, which include DGI (Velickovic et al., 2019), MVGRL (Hassani and Khasahmadi, 2020), GRACE (Zhu et al., 2020), BGRL (Thakoor et al., 2021), InfoGCL (Xu et al., 2021), and CCA-SSG (Zhang et al., 2021). As shown in Table 2, DDM achieves competitive results across all benchmark datasets. This underscores the capacity of the generative diffusion method to learn meaningful node-level representations and highlights the effectiveness of DDM in node-level tasks. Notably, the node features used in node classification are text embeddings, showcasing the efficacy of our directional noise in continuous word vector spaces.

5.3 Understanding the directional noise

The aforementioned studies provide evidence that our approach either outperforms or is on par with existing state-of-the-art (SOTA) methods. To gain a deeper understanding, we conduct a comprehensive investigation into the impact of different types of noise. Additionally, we analyze the effect of directional noise by removing the two constraints specified in equations (3) and (2).

We examine the representations extracted from models trained with directional noise and isotropic white noise at each reverse step. The results, as depicted in Figure 5, reveal significant differences between the two approaches. With white noise, only the representations corresponding to the early steps of the reverse process contained useful discriminative information, while the representations for later steps are mostly uninformative. This is in stark contrast to the case of directional noise, where the learned representations consistently preserve sufficient information for downstream classification tasks.

In our additional experiments, we consistently observe that directional diffusion models outperform vanilla diffusion models across all datasets, particularly in node classification tasks. This superior performance can be attributed to the nature of node classification datasets, which often use word vectors as node features. These word vectors have higher feature dimensionality and greater anisotropy. The effectiveness of our directional approach is further supported by these findings, reinforcing its value in graph representation learning.

To examine the impact of different schedulers, we compare the performance of the vanilla diffusion model using isotropic white noise and DDM under the cosine and sigmoid scheduler in Chen (2023). Table 3 collects the results. Different schedulers indeed influence the model performance. However, the performance of the vanilla diffusion model with isotropic white noise heavily relies on the hyperparameters of the scheduler. Yet, irrespective of the scheduler employed, our proposed data-dependent anisotropic noise consistently yields superior performance.

Lastly, we conduct an ablation study to examine the effects of the two constraints. Table 4 presents results, where "w/o R" indicates the removal of the constraint (2) and "w/o S&R" indicates the removal of both constraints. As shown in Table 4, the introduction of anisotropic Gaussian noise generated through (3) led to a significant improvement compared to isotropic Gaussian noise. Furthermore, the inclusion of constraint (2) provided an additional and indispensable improvement. This finding further confirms the importance of making the noise in the forward process data-dependent and anisotropic.

6 Conclusions

This paper unveils the presence of anisotropic structures in graphs, which pose challenges for vanilla diffusion models in graph representation learning. To address this limitation, we introduce directional diffusion models, a novel class of diffusion models that leverage data-dependent and anisotropic noise to better handle anisotropic structures. Through experiments conducted on 12 benchmark datasets, we demonstrate the effectiveness of our proposed method.

There are several promising avenues for future research. One direction is to develop methods that can automatically determine the optimal set of diffusion steps for each dataset, further enhancing the performance of our directional diffusion models. This could involve techniques such as adaptive selection of diffusion steps. Additionally, exploring the application of our method to computer vision and natural language processing tasks holds great potential for advancing these domains. By adapting and extending our directional diffusion models to these areas, we may leverage their inherent strengths to improve representations and enable effective learning tasks such as image recognition, object detection, sentiment analysis, and language understanding.

0

200

400

600

800

1000

Step

0.3

0.4

0.5

0.6

0.7

0.741

0.725

0.735

0.724

0.719

0.695

0.296

0.315

0.329

0.321

0.321

0.292

Method

Directional Noise

White Noise

(a) Citeseer

0

200

400

600

800

1000

Step

0.65

0.70

0.75

0.80

0.85

0.90

0.8936

0.8988

0.8556

0.8725

0.8827

0.8456

0.8187

0.6851

0.6491

0.6646

0.6544

0.6804

Method

Directional Noise

White Noise

(b) MUTAG

0

200

400

600

800

1000

Step

0.4

0.5

0.6

0.7

0.8

0.7089

0.807

0.809

0.808

0.798

0.799

0.711

0.595

0.53

0.457

0.413

0.414

Method

Directional Noise

White Noise

(c) PubMed

0

200

400

600

800

1000

Step

0.35

0.40

0.45

0.50

0.506

0.51

0.4993

0.4913

0.4787

0.476

0.5073

0.508

0.472

0.3693

0.358

0.354

Method

Directional Noise

White Noise

(d) IMDB-M

References

Abstreiter, K., Mittal, S., Bauer, S., Schölkopf, B., and Mehrjou, A. (2021). Diffusion-based representation learning. arXiv preprint arXiv:2105.14257.

Bansal, A., Borgnia, E., Chu, H.-M., Li, J. S., Kazemi, H., Huang, F., Goldblum, M., Geiping, J., and Goldstein, T. (2022). Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392.

Baranchuk, D., Rubachev, I., Voynov, A., Khrulkov, V., and Babenko, A. (2021). Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126.

Chang, C.-C. and Lin, C.-J. (2011). Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):1-27.

Chen, T. (2023). On the importance of noise scheduling for diffusion models. arXiv preprint arXiv:2301.10972.

Choi, J., Lee, J., Shin, C., Kim, S., Kim, H., and Yoon, S. (2022). Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472-11481.

Dhariwal, P. and Nichol, A. (2021). Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794.

Gao, J., He, D., Tan, X., Qin, T., Wang, L., and Liu, T.-Y. (2019). Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009.

Haefeli, K. K., Martinkus, K., Perraudin, N., and Wattenhofer, R. (2022). Diffusion models for graphs benefit from discrete state spaces. arXiv preprint arXiv:2210.01549.

Hassani, K. and Khasahmadi, A. H. (2020). Contrastive multi-view representation learning on graphs. In International conference on machine learning, pages 4116-4126. PMLR.

Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851.

Hou, Z., Liu, X., Dong, Y., Wang, C., Tang, J., et al. (2022). Graphmae: Self-supervised masked graph autoencoders. arXiv preprint arXiv:2205.10803.

Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, J. (2020a). Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118-22133.

Hu, Z., Dong, Y., Wang, K., Chang, K.-W., and Sun, Y. (2020b). Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857-1867.

Jo, J., Lee, S., and Hwang, S. J. (2022). Score-based generative modeling of graphs via the system of stochastic differential equations. arXiv preprint arXiv:2202.02514.

Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. (2020). On the sentence embeddings from pre-trained language models. arXiv preprint arXiv:2011.05864.

Li, X. L., Thickstun, J., Gulrajani, I., Liang, P., and Hashimoto, T. B. (2022). Diffusion-lm improves controllable text generation. arXiv preprint arXiv:2205.14217.

Preechakul, K., Chatthee, N., Wizadwongsa, S., and Suwajanakorn, S. (2022). Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619-10629.

Qiu, J., Chen, Q., Dong, Y., Zhang, J., Yang, H., Ding, M., Wang, K., and Tang, J. (2020). Gcc: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1150-1160.

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695.

Song, J., Meng, C., and Ermon, S. (2020). Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.

Sun, F.-Y., Hoffmann, J., Verma, V., and Tang, J. (2019). Infograph: Unsupervised and semisupervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000.

Thakoor, S., Tallec, C., Azar, M. G., Azabou, M., Dyer, E. L., Munos, R., Veliˇ ckovi´ c, P., and Valko, M. (2021). Large-scale representation learning on graphs via bootstrapping. arXiv preprint arXiv:2102.06514.

Velickovic, P., Fedus, W., Hamilton, W. L., Liò, P., Bengio, Y., and Hjelm, R. D. (2019). Deep graph infomax. ICLR (Poster), 2(3):4.

Xu, D., Cheng, W., Luo, D., Chen, H., and Zhang, X. (2021). Infogcl: Information-aware graph contrastive learning. Advances in Neural Information Processing Systems, 34:30414-30425.

Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2018). How powerful are graph neural networks? arXiv preprint arXiv:1810.00826.

Yanardag, P. and Vishwanathan, S. (2015). Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1365-1374.

Yang, Z., Cohen, W., and Salakhudinov, R. (2016). Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pages 40-48. PMLR.

Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., and Leskovec, J. (2018). Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31.

You, Y., Chen, T., Shen, Y., and Wang, Z. (2021). Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121-12132. PMLR.

You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., and Shen, Y. (2020). Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812-5823.

Zhang, H., Wu, Q., Yan, J., Wipf, D., and Yu, P. S. (2021). From canonical correlation analysis to self-supervised graph neural networks. Advances in Neural Information Processing Systems, 34:76-89.

Zhang, Z., Zhao, Z., and Lin, Z. (2022). Unsupervised representation learning from pre-trained diffusion probabilistic models. arXiv preprint arXiv:2212.12990.

Zhu, Y., Xu, Y., Yu, F., Liu, Q., Wu, S., and Wang, L. (2020). Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131.

Peer reviews:
comments: SUMMARY:
The authors present directional diffusion model (DDM) to learn graph / node representation. Compared to vanilla diffusion based representation learning techniques, DDM adds a batch based mean and variance, as well as preserving direction of diffusion. The authors then demonstrate that their model perform better than other models on representating learning by experimenting on various dataset and compare with other methods.

STRENGTHS:
- It is a nice finding by the authors that compared to diffusion model for generative modeling, diffusion model for representation learning does not require sampling from final distribution, thus we don't need to know the limiting behavior of the diffusion process.
- The experiments supports the authors' claim on the performance of the model.

WEAKNESSES:
- The authors should include a brief introduction on how diffusion models work on representation learning

QUESTIONS:
- A lot of citations by the paper, including some baselines like Infograph and GraphMAE, come from arxiv. Are these methods reliable?
- Directional diffusion preserves direction, so in recovery process, won't the representation fail to learn the direction of the entries? If it doesn't learn the directions, does directions don't matter?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-28

comments: SUMMARY:
This paper offers a good study on anisotropic and directional structure learning of the diffusion model. The authors first conduct a statistical analysis emphasizing the importance of anisotropic structure in graph dataset and demonstrate the disadvantage of the vanilla diffusion model through signal-to-noise ratio. Later on, they propose their work, which is a new pipeline considering preserving the characteristic of data in the diffusion process. Its idea is simple: adding directional diffusion on data, preserving more information than vanilla diffusion. Its model is simple, leveraging 4 layers GNNs and one layer MLP, it achieves improvement in several benchmarks. The idea is elegant.

STRENGTHS:
Good idea! The work not only offers inspiration to the graph learning community, but it also contributes to the diffusion model. The two constraints added in the work are elegant and proper.

WEAKNESSES:
The writing is not so well, especially the formula part. There are too many mistakes there. Their performances aren't satisfying, but I believe the underlying problem doesn't come from the pipeline, it comes from the simple model. There aren't convergence proofs for their learning architecture. Noticed that the diffusion model, which stems from the quasi-static process, possesses a good property of converging. I hope to see proof of the effectiveness of the constraints adding to the original process.

QUESTIONS:
1. Do you have proof of the effectiveness of your idea?
2. Can you offer more experimental results?

LIMITATIONS:
1. They haven't provided limitations for their work. 
2. Though the idea is inspirational, the writing is not good, and there are too many writing mistake.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method named Directional Diffusion Model (DDM) for unsupervised representation learning, targeting applications in graph and node classification. The model's performance is evaluated on various benchmark datasets and compared to both unsupervised and supervised models. The results demonstrate that DDM can effectively learn meaningful graph- or node-level representations. Furthermore, the paper offers some exploration of how different types of noise impact the learning process.\n\nSTRENGTHS:\n1. The authors present empirical evaluations of DDM on multiple benchmark datasets, which validate the effectiveness of the proposed method to some extent.\n\n2. The paper includes an investigation into the impact of different types of noise, which adds depth to the analysis and understanding of the proposed method.\n\nWEAKNESSES:\n1. The SVD visualizations could be more insightful. It's observed that the 2D projections computed from graph datasets appear biased and are predominantly on the right plane. However, the methodology behind these visualizations remains unclear. Are they based on singular values? Further explanation would be beneficial.\n\n2.  The Signal-to-Noise Ratio (SNR) plots in the supplementary material show comparable results under white noise and directional noise. This implies that the DDM may not offer significant advantages on these datasets. The authors should provide some clarification.\n\n3. The paper is primarily empirical and lacks theoretical foundations, which might limit its generalizability.\n\n4.  The algorithm snippet doesn't seem to provide sufficient implementation details, which might impede attempts to reproduce the study.\n\nQUESTIONS:\n1. Please clarify the issues raised in the weaknesses section.\n\n2. How does DDM compare with widely-known baselines in terms of performance, and are there any specific scenarios where DDM particularly excels or falls short?\n\n3. Regarding the means and deviations at line 170: are these calculated relative to the node features at time 0, or do they consider other time steps?\n\n\nIn its current form, I believe the paper does not meet the rigorous quality and contribution standards expected at a top-tier conference like NeurIPS.\n\nLIMITATIONS:\nThe authors did not explicitly state any limitations of their study. However, the identified weaknesses could be viewed as potential limitations. These include the lack of theoretical foundations and potential issues in reproducibility due to insufficient implementation details. Additionally, the paper could benefit from a clear discussion of the limitations of DDM in handling specific scenarios or types of data.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This study proposed adding directional noise on learning anisotropic graphs using diffusion models. The study adds new perspectives in exploring the anisotropic structures in graph data. The numerical results are promising to support the authors' ideas.

STRENGTHS:
I find this is an interesting paper - the rational is convincing, and the authors performed extensive experiments to support the idea. The discussions on the noise and ablation studies provide further insights into understanding the usefulness of adding directional noises. The paper provides a valuable perspective in understanding diffusion models for graph learning.

WEAKNESSES:
The authors mainly prove the utility of the proposed approach through experiments; there is a lack of theorectical proof.

QUESTIONS:
The authors have performed extensive experiments, but it would add more evidence if the authors can add some theoretical proof. 

For figure 3, does each dot represent a graph? Are they from simulated datasets?

LIMITATIONS:
The authors have discussed several future directions that can be used to improve the current model.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a class of diffusion models to improve the accuracy of graph representation learning. The model incorporates both data-dependent and anisotropic noise in the forward noising process, by scaling its magnitude and direction based on each coordinate of the data. This structured noise maintains the signal present in the data over longer time windows than using standard isotropic white noise during the forward noising process. The authors show that this model improves upon state-of-the-art methods on a large collection of benchmark datasets. Moreover, they perform an ablation study to understand the effect of the two proposed modification to the noise process.\n\nSTRENGTHS:\n- The authors visually explain and empirically demonstrate the effect of using non-isotropic noise in the forward diffusion process to improve classification tasks. These noise processes provide clear intuition why this is preferred to white noise for these tasks.\n- The authors propose a novel strategy for extracting graph representations based on time-dependent denoiser that combines graph neural networks and UNet architectures.\n- The application of diffusion models to graph representation learning is novel and the new model is shown to yield superior results to existing algorithms.\n\nWEAKNESSES:\nIt would be great for the authors to comment and compare with other non-isotropic noise processes that have been proposed for diffusion models for sampling and generative modeling. Some examples outside of the graph representation learning context are: \n* Score-based Denoising Diffusion with Non-Isotropic Gaussian Noise Models, Vikram Voleti, Christopher Pal, Adam Oberman, 2022\n* Blurring Diffusion Models, Emiel Hoogeboom, Tim Salimans, 2023\n\nIn addition to the changes in the noise process, the authors propose a specific architecture for the denoiser, and selection of representative features from their hidden layers. This architecture could be relevant on its own to extract representations of the dataset, without the denoising time components. Do any of the compared methods investigate how this denoiser architecture, without the diffusion model, would perform for representation learning. This might be helpful as an additional ablation study to see the effect of the architecture. \n\nThe metrics used to evaluate their experiments can be described in more detail. The values for the results in Tables 1,2,3 were not clearly mentioned in the caption or the main text. Mathematical equations may also be helpful to precisely describe the accuracy measurements in Figure 5. \n\nThe authors comment that the word vectors often exhibit greater anisotropy, which yields superior performance in node classification tasks. It would be great if the authors could quantify this to validate this claim that the performance improves with more anisotropy.\n\nQUESTIONS:\nAn important choice for defining the representations are the user selected time-steps where the outputs are extracted from the denoising network. Do the authors have a guideline or recommendation (e.g., based on the empirical studies) for how to choose these time steps. This seems particularly relevant given that the SNR does not degrade monotonically for some datasets (e.g., IMDB in Figure 2). \n\nEven if the SNR in standard diffusion models decay quickly with increasing time, how do the results presented here compare to a baseline diffusion model with isotropic noise where the time-steps are chosen \"optimally\". For example, what if the time-steps are chosen all near the initial time, $t = 0$, when there is still signal contained in the noisy data?\n\nLIMITATIONS:\nThe authors show improved results on almost all datasets. It would be great if the authors could comment on what leads to the similar performance on the Ogbn-arxiv dataset. Is it because the data is more isotropic in this case? When do the authors expect the algorithm to under-perform for the evaluated tasks?\n\nIn addition to the questions above, it would be good to address these minor comments:\n- Define the acronym for their proposed framework, DDM\n- Include standard errors for the results in Table 3\n- Explain why $f_\\theta$ depends on $A$ in equation (4). It is sufficient to highlight that this is the structure of a GNN, which may be helpful for some readers.\n- Clarify in Figure 4 that the directional noise is only added to $X_0$ and not $A$.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the gap in unsupervised graph representation learning by exploring the use of diffusion models. They propose directional diffusion models that incorporate data-dependent, anisotropic, and directional noises in the forward diffusion process to better handle anisotropic structures in graphs. Experiments on publicly available datasets showcase the superiority of their models over state-of-the-art baselines, demonstrating their effectiveness in capturing meaningful graph representations. Overall, the paper presents a compelling approach that contributes to the advancement of unsupervised graph representation learning.\n\nSTRENGTHS:\n1. **Motivation**\n\n    The introduction is well-motivated, providing a thorough explanation of the challenge and task at hand. The authors go beyond textual descriptions and use simple visualizations on both real and synthetic data to demonstrate their points effectively. This approach enhances the clarity and understanding of the presented research, making it accessible to a wider audience.\n&nbsp;\n\n2. **Method**\n\n    The authors present a straightforward yet effective solution for incorporating directional noise into node embeddings. This approach effectively addresses the challenge posed by anisotropic structures in various domains, including graphs and potentially text data. Their proposed method demonstrates promising results in handling directional noise and enhancing the quality of node and graph embeddings.\n&nbsp;\n\n3. **Architecture**\n\n    I appreciate the authors intention to adapt the well-known and effective U-Net architecture from the image domain to the graph domain. The incorporation of skip connections in the U-Net is particularly relevant for denoising tasks. This thoughtful adaptation enhances the model's ability to handle graph-related denoising effectively.\n&nbsp;\n\n4. **Experiments**\n\n    The authors conduct a comprehensive comparison with numerous baselines across multiple datasets. Furthermore, their evaluation settings, which involves 10-fold cross-validation with standard deviation after five runs, are robust and reliable. This rigorous evaluation methodology ensures the validity and statistical significance of their results.\n\nWEAKNESSES:\n1. **missing releted work**\n\n    There are existing works in the intersection of graphs and diffusions are missing, contradicting the authors statement \"To the best of our knowledge, there have been no works for diffusion-model-based graph representation learning.\". Some for example:\n    - Niu, Chenhao, et al. \"Permutation invariant graph generation via score-based generative modeling.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.\n    - Xu, Minkai, et al. \"Geodiff: A geometric diffusion model for molecular conformation generation.\" International Conference on Learning Representations, 2022.\n    - Vignac, Clement, et al. \"Digress: Discrete denoising diffusion for graph generation.\" International Conference on Learning Representations, 2023.\n\n    Furthermore, some simple techniques can be applied to create a smoother SNR curves over the different diffusion steps. For instance:\n    - Chen, Ting. \"On the importance of noise scheduling for diffusion models.\" arXiv preprint arXiv:2301.10972 (2023).\n\n    The same problem was presented in it over the image domain (on high-resolution images), and the solution was to use different noise schedulers. Why not simply try this trick?\n&nbsp;\n\n2. **missing details**\n\n   There are missing details in the paper that makes it hard to fully understand and reproduce the papers results. For example:\n    - \"µ and σ are calculated using graphs within the batch.” -- what is done during inference? it is EMA over what was been seen in training time? does it handle only batch inference?\n    - They paper did not state how exactly the entire graph-level representation is obtained, it is sum/mean of all node representations?\n    - Algorithm 2 in appendix, line 6: what is exactly $A_t$? should it be just $A$?\n\nQUESTIONS:\n1. \"µ and σ are calculated using graphs within the batch.” -- what is done during inference? why not just pre-calculate those on the entire train split?\n2. \"allowing the latent code to contain meaningful compressed knowledge while preventing the decoder from degrading into an identical mapping” -- why will that happen that requires a dedicate MLP? what is this different from the simple U-Net used in the image domain?\n3. Algorithm 2 in appendix, line 6, what is $A_t$?\n4. In some cases, the SNR goes even higher for larger diffusion steps (like in the Computer dataset for instance), why is that?\n5. How is the entire graph-level representation obtained?\n6. Referring to figure 2, shouldn't the SNR be 0 in the last diffusion steps? It doesn't reach zero as the directional noise keeps them separated, is that a wanted outcome? If so, why?\n7. Why not apply just a different noise scheduler? instead of linear, coisne for instance. Similar observation was shown on high-resolution images and different schedulers where proposed [1]. I know it is still isotropic noise, so at least show results as a baseline/ablation.\n\n\n___\n\n[1] Chen, Ting. \"On the importance of noise scheduling for diffusion models.\" arXiv preprint arXiv:2301.10972 (2023).\n\nLIMITATIONS:\nThe authors address the limitations of their work, and support their claims with experiments and ablations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}