Directional diffusion models for graph representation learning
Authors: Run Yang
Yuling Yang
Fan Zhou
Qiang Sun
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion models
graph representation learning
unsupervised learning

Abstract: Diffusion models have achieved remarkable success in diverse domains such as image synthesis, super-resolution, and 3D molecule generation. Surprisingly, the application of diffusion models in graph learning has garnered little attention. In this paper, we aim to bridge this gap by exploring the use of diffusion models for unsupervised graph representation learning. Our investigation commences with the identification of anisotropic structures within graphs and the recognition of a crucial limitation in the vanilla forward diffusion process when dealing with these anisotropic structures. The original forward diffusion process continually adds  isotropic Gaussian noise to the data, which may excessively dilute anisotropic signals, leading to rapid signal-to-noise conversion. This rapid conversion poses challenges for training denoising neural networks and obstructs the acquisition of semantically meaningful representations during the reverse process. To overcome this challenge, we introduce a novel class of models termed {\it directional diffusion models}.  These models adopt data-dependent, anisotropic, and directional noises in the forward diffusion process. In order to assess the effectiveness of our proposed models, we conduct extensive experiments on 12 publicly available datasets, with a particular focus on two distinct graph representation learning tasks. The experimental results unequivocally establish the superiority of our models over state-of-the-art baselines, underscoring their effectiveness in capturing meaningful graph representations. Our research not only sheds light on the intricacies of the forward process in diffusion models but also underscores the vast potential of these models in addressing a wide spectrum of graph-related tasks. Our code is available at \url{https://github.com/statsle/DDM}.

Directional diffusion models for graph representation learning
Run Yang SUFE and Baidu
Yuling Yang SUFE sibyllayang@163. sufe. edu. en
1uckyyangrun0163. sufe. edu. cn
Fan Zhou SUFE
Qiang Sun"
University of Toronto qiang. www.stooontootooooomm ca
zhoufan@maii wwwfe.euu.cc
edu.
Abstract
Diffusion models have achieved remarkable success in diverse domains such as image synthesis, super-resolution, and 3D molecule generation. Surprisingly, the application of diffusion models in graph learning has garnered little attention. this paper, we aim to bridge this gap by exploring the use of diffusion models for unsupervised graph representation learning. Our investigation commences with the identification of anisotropic structures within graphs and the recognition of a crucial limitation the vanilla forward diffusion process when dealing with these anisotropic structures. The original forward diffusion process continually adds isotropic Gaussian noise to the data, which may excessively dilute anisotropic signals, leading rapid eignal-to-noise conversion. This rapid conversion poses challenges for training denoising neural networks and obstructs the acquisition semantically meaningful representations during the reverse process. To overcome this challenge, we introduce a novel class of models termed directional diffisiin models. These models adopt data-dependent, anisotropic, and directional noises in the forward diffusion process. In order to assess the effectiveness of our pro- posed models, we conduct extensive experiments with a particular focus on two distinct graph representation learning tasks. The experimental results unequivocally establish the superiority of models over state-of-the art baselines, underscoring their effectiveness in capturing meaningful graph representations. Our research not only sheds light on the intricacies of the forward process in diffusion models but also underscores the vast potential of these models in addressing wide spectrum graph-related tasks. Our code available at htttt::/// :/s //g thub. com/ statsle/DDM
12 publicly available datasets,
Introduction
Unsupervised representation learning through diffusion models has emerged prominent area of research computer vision. Several methods that leverage diffusion models, such as those proposed by IZhang et al.] (202)); Preechakul et al. (2022); Abstreiter et al. (202T); Baranchuk ct al. (2021) have been put forth for representation learning. Notably, Baranchuk et al. (2021)
showed that the intermediate actiations obtained from denoising networks contain valuable semantic information. Their findings emphasize the effectiveness diffusion models in learning meaningful
"Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

0.08
0.6
label
label
label
0.06
airplane
0.4
04
uutomobile
1
2
02
0.0-
0.0
0.02
-0.2
0.04
-0.4
-0.4
0.06
-0.05
0.00
0.05
-0.4
0.0
0.4
-0.4
-0.2
0.0
0.4
x
x
x
(a) CIFAR-10
(b) Amazon-
(c) IMDB-M
Figure I: 2D visualization of the data using SVD decomposition. (a) Visualization of the node features in CIFAR-1O, with different colors indicating different labels. (b) Visualization of the two classes in Amazon-Phoue Photo.. (c) Visualization of the graph features in IMDB-M.
visual representations. More recently, Choi et al. (2022) showed that restoring data corrupted with certain noise levels offers proper pretext task for the model to learn rich visual concepts.
Despite the increasing research on diffusion models in computer vision, there is still noticeable shortage of studies exploring the use of diffusion models in graph learning. Previous works, such as those by HHefell et al.] (2022) and Do et al. (2022). have primarily concentrated on employing diffusion models for generating discrete graph structures. However, the realm of graph representation learning, a fundamental and challenging aspect of graph learning, has yet harness the potential of diffusion models. To effectively adapt and incorporate diffusion models into graph representation learning, and to facilitate progress in this domain, imperative to identify and understand the obstacles hindering the application of diffusion models.
To gain insights into the limitations of the vanilla diffusion models which are commonly adopted for image generation tasks (Ho et al. 2020), we first investigate the underlying structural disparities between images and graphs. In particular, we employ singular value decomposition (SVD) on both image and graph data, and visualize the projected data in a a 2-dimensional plane, shown in Figure I This figure illustrates that the projected data from Amazon Photo and IMDB-M exhibit strong anisotropic structures along only a few directions, while the projected images from CIFAR-10 form a relatively more isotropic distribution within a circular shape centered around the origin. This observation strongly implies that graph data may feature distinctive anisotropic and directional structures that are less prevalent in image data. As we will subsequently demonstrate, the vanilla diffusion models with isotropic forward diffusion processes result in rapid decrease in the signal-to- noise ratios (SNRs), which in turn diminishes their effectiveness learning anisotropic structures. Therefore, it is imperative to develop approaches that effectively account for these anisotropic structures.
This paper presents directional diffusion models solution for better learning anisotropic struc- tures. Our approach involves incorporating data-dependent and directional noise into the forward diffusion process, effectively mitigating the challenge of rapid signal-to-noise ratio deterioration. The intermediate activations obtained from the denoising network excel at capturing valuable semantic and topological information erucial for downstream tasks. Consequently, the proposed directional diffusion models offer a promising approach for generative graph representation learning. Numeri- cally, we perform experiments 12 benchmark datasets covering both node and graph classification tasks. The results consistently highlight the superior performance of our models when compared to state-of-the teeffftth-aam- art contrastive learning and generative approaches (Hou et al. 2022). Notably, for graph classification problems, our directional diffusion models even surpass supervised baseline methods, underscoring the considerable potential of diffusion models in the field of graph representation learning-
Our main contributions summarized as follows.

---

1. We contribute to the exploration of anisotropic structures in graph data, being among the pioneers in the literature. We demonstrate that the vanilla forward diffusion process with isotropic white noise leads to rapid decline in signal-to-noise ratios for graph learning problems. This issue hampers the ability of denoising networks to extract fine-grained feature representations across wide range of SNRs.
2. We propose novel directional diffusion models specifically designed for graph data, incorpo- rating data-dependent and directional noise in the forward diffusion process. Our proposed models effectively address the issue of the rapid decline of SNRs, enabling better graph representation learning.
3. Numerically, our proposed directional diffusion models outperform state-of-the-art self- supervised methods and even supervised methods on 12 benchmark datasets. Additionally, we provide comprehensive ablation studies gain a deeper understanding of the mechanisms underlying directional diffusion models.
2
Related work
Graph representation learning Graph representation learning aims to embed nodes or entire graphs into a dow-dimensional vector space, where the structural and relational properties can be used for downstream tasks. Two prevalent paradigms for graph representation learning are contrastive learning and generative self-supervised learning. Contrastive learning approaches such as DGI (Velickovic et al. 2019), Infograph (Sun et al., 2019), GraphCL (You et al., 2020), GRACE (Zhu al. 2020p, and GCC (Qiu al., (020), have achieved promising results in some particular graph learning tasks. These methods leverage local-global mutual information maximization for node and graph representation learning. GraphCL learns node embeddings that are invariant to graph-level transformations, while GRACE and GCC use subgraph sampling and graph perturbation to create augmented pairs. Generative self-supervised learning aims to recover masked components of the input data through approaches such GraphMAE (Hou al. 2022). masked graph autoencoder that focuses on feature reconstruction by utilizing a masking strategy and scaled cosine error. This method outperforms self-supervised learning baselines, and revitalizes the concept of generative self-supervised learning on graphs. GPT-GNN (Hu et al.] (2220b) is a recent approach that introduces a self- self--upprvised attributed graph generation task to pre-train GNN so that it can capture the structural and semantic properties of the graph.
Denoising diffusion probabilistic models Denoising diffusion probabilistic models (Ho et al. 2020 Song et al. 2020), or simply diffusion models, are class of probabilistic generative models that turn noise to data sample and thus are mainly used for generation tasks (Dhariwal and Nichol 202T; Rombach et al.) Recently. diffusion models have been used represennation learning toolbox in computer vision (Preechakul et al. 2022 Abstreiter et al.) 2021 Baranchuk et al. 22221) For instance, Prrechakul et al. (2022) proposed Diff-AE, a method that concurrently trains an encoder to discover high-level semantics and a conditional diffusion model that uses these representations
as input conditions. [Abstreiter et al. (202T) augmented the denoising score matching framework to enable representation learning without any supervised signal. Recently, diffusion models have also found applications in handling graph data. Haefeli et al. (2022) showed that diffusion models for graphs benefited from discrete state spaces. Jo et al. (2022) proposed the graph diffusion using stochastic differential equations. To the best of our knowledge, there have been no works for maesiion-modl-beee graph representation learning.
3 The effect of anisotropic structures
As discussed in the introduction section, significant structural differences exist between graphs and natural images. In addition to the analysis of the Amazon Photo and IMDB-M datasets, we conducted similar examinations on all other graph benchmark datasets, and the detailed results can be found in the appendix. Moreover, is important to highlight that the anisotropic structures, referred to as categorical directional dependence, are frequently encountered in natural language data (Gao et al. 2019 Li et al. 2020). intriguing to observe that diffusion models have not yet made significant strides in the realm of natural language processing. This further underscores the significance of
3

---

0.8
white noise directional noise
1.5-
white noise
directional noise
0.6
SNR
SNR
1.0
0.4
0.5
0.2
0.0
0.0
200
400
600 step
800
1000
200
400
600
800
1000
step
(a) Amazon-Photo
(b) IMDB-M
Figure 2: The signal-to-noise ratio curve along different diffusion steps.
investigating and tackling the challenges posed by anisotropic structures within the framework of diffusion models.
This section delves deeper into the examination of how anisotropic structures in graphs impact the effectiveness of vanilla diffusion models. In the vanilla forward diffusion process, isotropic Gaussian noise is sequentially added to the raw data point zi ~ ((zo) until it transforms into isotropic white noise N (0, P This approach reasonable when the data follow isotropic distributions, as gradually transforms the data into noise, generating sequence of noisy samples with wide range of SNRs. However, in cases where the data exhibits anisotropy, the addition of isotropie noise can quickly contaminate the data structure, causing the SNRs rapidly approach zero. Consequently, denoising networks struggle to extract fine-gre aine-grinedd feature representations.
To explore the impact of introducing isotropic noise on the learning of anisotropic graphs, we conduct an experiment measure the SNRs for both node and graph classification tasks at each forward step. We observe how these SNRs change throughout the forward diffusion process. Initially, we pre-train a graph neural network (GNN) denoted E to serve as a feature extractor, projecting the graph data into a linearly separable space. Subsequently, we optimize the weight vector R in this linear space using Fisher's linear discriminant analysis. The weight vector w then employed to compute the SNR at each forward diffusion step, where SNR = w Snw w/w""/"/// Sww. In this equation, Sn represents the scatter between class variability, and Sw represents the scatter within-clase variability, This SNR quantifies the discriminative power of the learned representations at different stages of the diffusion process.
We conducted this experiment on all graph benchmark datasets to assess the impact of isotropic noise on learning anisotropic graphs. Here, we present the results for IMDB-M and Amazon-Phott in Figure 2a while the additional results can be found in the appendix. In Figure 2a we observe that for anisotropic graph data and isotropic noise, the SNR rapidly decreases to at around 50 steps for Amazon-Phott and 400 steps for IMDB-M. Furthermore, the SNR remains close to 0 thereafter, indicating that the incremental isotropic white noise quickly obseures the underlying anisotropic structures signals. Consequently, the denoising networks unable to learn meaningful and discriminative feature representations that can be effectively utilized for downstream classification tasks. In contrast, when utilizing our directional diffusion models, which incorporate data data-dependent and directional forward diffusion process (to be introduced later), the SNR declines at a slower pace. This slower decline enables the extraction of fine-graine grained feature representations with varying SNRs, preserving the essential information of the anisotropic structures.
Overall, these studies underscore the significance of considering anisotropic data structures when designing forward diffusion processes and the corresponding diffusion models, especially in the context of graph data.
BBold symbols used for matrices, but not for vectors.

---

Forward Process with Directional Noise
0
100
500
800
1000
Forward Process with White Noise
100
t 500
800
t 1000
Figure 3: Directional noise white noise. We sample directional and white noise'. and subsequently add the noise in sequential manner. The upper panel displays the samples with directional noise at the diffusion steps 0, 100, 500, 800, 1000, while the lower panel exhibits the samples with white noise at these identical diffusion steps. The two distinct colors indicate two different classes.
Directional diffusion models
In this section, start by introducing the requisite notation. Subsequently, propose the directional diffusion models (DDMs), with a specific focus on their application to graph representation learning. Furthermore, discuss how extract feature representations from DDMs, pivotal aspect for downstream tasks.
Notation We denote graph by G (V, X). where V is the node set, N V is the node num- ber, A E R. N is the adjacency matrix (binary weighted), and X ( 22, _n E R is the node feature matrix. Our goal is to learn a network, denoted as f R x NxN + R Nxds encode graph features into representations
(h1 f2.... hx) T Nxd,, where h2 E Rd is the representation for node i. Mathematically, we have H = f(X, A). These representations can subsequently be used for downstream tasks, such as graph and node classification.
Directional diffusion models In the preceding section, our investigation uncovered a pivotal factor contributing the underwhelming performance of vanilla diffusion models in graph learning: the swift deterioration of signal-to-noise ratios. To tackle this challenge, we introduce the directional noise in the forward diffusion process, which involves transforming the isotropic Gaussian noise into an anisotropic noise by incorporating two additional constraints. These two constraints play a vital role enhancing the efficacy of vanilla diffusion models.
Let G, = (A, be the noisy graph at the t-th forward diffusion step, where X, (R0,11 11,2, ... F4,ND represents the learned features at the t-th step. Specifically, the node feature Ft, E Rd of node at time t is obtained follows:
It, = Vārzo + VI-are,
(1)
é' sgu()n.)) & [E),
p where (0,1),
where wi, the raw feature vector node i, μ E Rdd and a E Rd consist the means and standard deviations of the features across N nodes, respectively. The symbol @ denotes the Hadamard product. During the mini- batch training, and a are calculated using graphs within the batch. The
parameter ag := o B,) e (0, represents the variance schedule (Ho et al. 2020) and parameterized by decreasing sequence {ßI:7 1))
'The generation parameters provided in the appendix.
5

---

In contrast to the conventional forward diffusion process, our directional diffusion models integrate two additional constraints, as denoted in equations (2) and (3). The second constraint, (3), transforms the data cate-innependent Gaussian noise into an anisotropic and batch-dependent noise. In this constraint, each coordinate of the noise vector shares the same empirical mean and empirical standard deviation as the corresponding coordinate in the data within the same batch. This constraint confines the diffusion process to the local neighborhood of the batch, preventing excessive deviation from the
batch and preserving local coherence.
The first constraint, 2 aligns the noise c' with the feature zo, to ensure that they share the same coordinate signs. This guarantees that adding noise does not result in noisy features pointing in the opposite direction of zo. By preserving the directionality of the original feature, this constraint plays a crucial role in maintaining the inherent data structure during the forward diffusion process. These two constraints work tandem to ensure that the forward diffusion process respects the underlying data structure and mitigates the rapid degradation of signals. Consequently, the SNR decays slowly, enabling our directional diffusion models to effectively extract meaningful feature representations across various steps. This, in turn, enhances the utility of these representations in downstream tasks by providing reliability and informativeness.
To illustrate the impact of directional noise, we refer to the experiments conducted in Section 3 Our newly proposed "directional noise" ensures a smoother decline of the SNRs throughout the diffusion process, confirming our initial intuition. To further visualize the differences between using directional noise and isotropic noise in the forward diffusion process, we conducted simulations on two ellipses and sequentially added noise, as depicted in Figure 3 The figure clearly illustrates the distinct behaviors exhibited by the two types of noise. With directional noise, the samples maintain a clear decision boundary, indicating the preservation of discriminative structures during the diffusion process, Conversely, samples with white noise quickly blend into pure noise, leading to the loss of discriminative information. This visual comparison unequivocally underscores the superiority of directional noise preserving the structural information of the data during the forward diffusion process.
Model architecture We follow the same training strategy as in the vanilla diffusion models, where we train denoising network fe in the reverse diffusion process. Since the posterior of the forward process with directional noise cannot be expressed in closed form, we borrow the idea from Bansal al. (2022); Li et al. (2022) and let the denoising model fo directly predict Xg. The loss function L is defined as the expected value of the Euclidean distance between the predicted feature representation fo A, t) and the original feature representation X,::
L = Exo, (x,, A, Xol12.
This loss function ensures that the model predicts Xg at every step.
To parameterize the denoising network fo. we employ symmetrical architecture inspired by the successful UNet architecture in computer vision (Dhariwal and Nichol 2021).
our DDM framework, comprising four GNN layers and one multilayer perception (MLP). The first two GNN layers function as the encoder, responsible for denoising the target node by aggregating neighboring information. The final two GNN layers serve as the decoder, mapping the denoised node features to a latent code and smoothing the latent code among neighboring nodes. To address the potential issue of over-smoothing and account for long-distance dependencies in the graph, we introduce skip-connections between the encoder and decoder. The algorithm, along with the mini-batch training procedure, is presented in the appendix.
Learning representations For a given graph G = (A, the learned node-level representations are obtained from the activations of the denoising network fo at user-selected time steps. It is important to note that we only utilize the activations from the decoder of fo since they incorporate the encoder activations through skip connections. As depicted in Figure 4 each time step k. we introduce i steps of directional noise following (I and employ the denoising network fo to denoise and encode the noisy data X,- The decoder of Jo maps the denoised node features to a latent code while smoothing the latent code among neighboring nodes. We extract the activations from the decoder of fo and concatenate them to obtain H₃ {1u,, hu.2, ... ,,R,N) E R Xdd, The complete pipeline is presented in the appendix.

---

Say
>Gp
Forward Process
Denolsing
Skip connection
Enteediing step
Concalonalc:
Representation
Figure 4: The pipeline of model: (1) Adding directional noise the original graph Go- (2) Extracting the last two GNN's feature maps the denoising network as the representation of the graph.
Experiments
This section provides an evaluation of the directional diffusion models from two perspectives. First, we compare our models with existing state-of-the-art methods on various graph learning tasks, including node and graph classification tasks. This allows us to assess the effectiveness of our approach for graph representation learning problems. Second, we conduct several studies gain better understanding of the effect of our directional noise and evaluate the necessity of our design choices.
In all experiments, we follow a two-step process. First, we pre-train a DDM on the dataset in an unsupervised manner. Then, extract feature representations from diffusion steps 50, 100, 200 using the pre-trained model. Although this approach is inspired by the experimental results and insights from Section 3 it is deliberately not fine-tuned for each dataset. Ideally, fine-tuning with carefully selected steps for each dataset could further improve the performance.
5.1 Graph classification
To demonstrate the effectiveness of method, conduct comparisons with state-offthh--a (SOTA) unsupervised learning methods, which include GCC (Qiu et al. (2020), Infograph (Sun et al.
2019). GraphCL (You al., (020). JOAO (You et al., (021). MVGRL (Hassani and Khasahmadi
2020 and GraphMAE (Hou et al. 2022). D. We also compare our approach with supervised learning methods, specifically GIN (Xu al. 2018)) and DifiPoool (Ying al. 2018). These experiments are conducted on seven widely-used datasets, namely MUTAG, IMDB-B, IMDB-M, PROTEINS, Vishwanathan 2015). Node degrees are used initial node features for IMDB-E B. IMDB-M, REDDIT-B, and COLLAB, while node labels are employed for MUTAG and PROTEINS, in accordance with prior literature (Hou al., (022). We extract graph-level representations at various steps, train linear SVMs using LIBSVM (Chang and Lin 2011), and independently test them. The final predictions are obtained by majority vote, and we report the average accuracy and standard deviation, calculated after five runs. Additional details on hyper-parameters can found in the appendix.
COLLAB, and REDDIT-B (Yanardag
Table I presents the results, clearly indicating that our DDM achieves the most competitive, if not the best, performance across all benchmark datasets. Notably, our DDM outperforms even the supervised approaches in certain experiments, such as IMDB-B, COLLAB, and MUTAG. This remarkable performance can attributed two key factors. First, from data perspective, these datasets have limited information in their node features, which can hinder the accuracy of supervised learning (Hou et al., 2022). By leveraging directional noise diffusion, our DDM effectively acts as a Aneudo-infinite-Ste data augmentation technique, generating numerous samples while preserving the classification boundary. This augmentation significantly enhances the effectiveness unsupervised learning. Second, from model perspective, the DDM framework harnesses the power of directional
7

---

Table 1: Results unsupervised representation learning for graph classification.
Dataset
IMDB-B
IMDB-M
COLLAB
REDDIT-B
PROTEINS
MUTAG
GIN
75.1=5.1
52.342.8
80.2+1.9
92.4+2.5
76.2+2.8
89.425.6
DiffPool
72.6#3.9
78.9+2.3
92.1+2.6
75.1+2.3
85.0+10.3
Infograph
73.03=0.87
49.6920.53
70.65+1.13
82.50+1.42
74.44+0.31
89.01=1.13
GraphCL
71.14#0.44
48.5840.67
71.36*1.15
89.53+0.84
74.39+0.45
86.80#1.34
JOAO
70.21#3.08
49.20=0.77
69.5010.36
85.29+1.35
74.55+0.41
87.35+1.02
GCC
72
49.4
78.9
89.8
MVGRL
74.20=0.70
51.20=0.50
84.50+0.60
89.70=1.10
GraphMAR
75.52+0.66
51.63+0.52
80.3240.46
88.01
75.30+0.39
88. 18.1921.26
DDM
76.4020.22
52.5340.31
81.7240.31
$9.15+1.3
75.47 40.50
91.51 +1.45
Table 2: Results in unsupervised representation learning for node classification.
Dataset
Cora
Citeseer
PubMed
Ogbn-arxiv
Computer
Photo
GAT
83.0 0.7
72.5 0.7
79.0 0.3
72.10 0.13
86.93 + 0.29
92.56 0.35
DGI
82.3 # 0.6
71.8 0.7
76.8 0.6
70.34 + 0.16
83.95 + 0.47
91.61 #
MVGRL.
+ 83.5
83.5
0.4
73.3 0.5
80.1
87.52 +.1
91.74 0.07
BGRL
82.7 2.6
79.6 7.8
71.64 0.12
89.68 + 0.31
92.87 + 0.27
InfoGCL
83.5 0.3
0.2 0.4
CCA-SSG
0.4
81.0 0.3
71.24 + 0.20
88.74 + 0.28
93.14 + 0.14
GPT-GNN
80.1
1.6
7.83
GraphMAE
8.42
81.1 0.4
71.75 # 0.17
88.63 + 0.17
93.63 # 0.22
DDM
83.4
74.3
81.7 0.8
71.29 0.18
90.56 + 0.21
95.09 0.18
noise and ensures that the learned representations capture discriminative information by preventing the rapid decay of signal-to-noise ratios.
5.2 Node classification
To assess the quality of the node-level representations produced by our method, we conduct eval- uations of DDM six standard benchmark datasets: Cora, Citeseer, PubMed (Yang al. 2016).
Ogbn-arxiv arxiv (Hu et al., (020a). AmaazonCComuutt (Zhang et al. 2021). and Amazon-Phote (Zang et al. 2021). We follow the publicly available data-split schema and employ the evaluation protocol used in the literature. Graph-level representations are extracted at different diffusion steps, and an independent linear classifier is trained for each step. The final predictions determined through majority voting, and we report the mean accuracy on the test nodes. Additional details regarding hyperparameters can be found in the appendix.
We compare DDM with state-of-the-e art generative unsupervised models, specifically GPT-GNN Hu et al. 2020b) and GraphMAE (Hou et al. 2022 Furthermore, we include the results of contrastive unsupervised models for comparison, which include DGI (Velickovic et al. (2019) MVGRL (Hassani
and Khasahmadi 2020). GRACE (Zhu et al. 2020). BGRL (Thakoor et al. 2021). InfoGCL (Xu et al. 2021), and CCA-SSG (Zhang et al. 2. As shown in Table [ DDM achieves competitive results across all benchmark datasets. This underscores the capacity of the generative diffusion method to learn meaningful node-level representations and highlights the effectiveness of DDM in node-level tasks. Notably, the node features used in node classification are text embeddings, showcasing the efficacy of our directional noise in continuous word vector spaces.
5.3 Understanding the directional noise
The aforementioned studies provide evidence that our approach either outperforms or on par with existing state-of-the-arr (SOTA) methods. To gain deeper understanding, we conduct comprehensive investigation into the impact of different types of noise. Additionally, we analyze the
effect of directional noise by removing the two constraints specified in equations and (2)

---

We examine the representations extracted from models trained with directional noise and isotropic white noise at each reverse step. The results, as depicted in Figure [ reveal significant differences between the two approaches. With white noise, only the representations corresponding to the early steps of the reverse process contained useful discriminative information, while the representations for later steps are mostly uninformative. This is in stark contrast to the case of directional noise, where the learned representations consistently preserve sufficient information for downstream classification tasks.
In our additional experiments, we consistently observe that directional diffusion models outperform vanilla diffusion models across all datasets, particularly in node classification tasks. This superior performance can be attributed to the nature of node classification datasets, which often use word vectors as node features. These word vectors have higher feature dimensionality and greater anisotropy. The effectiveness of our directional approach is further supported by these findings, reinforcing its value in graph representation learning.
To examine the impact of different schedulers, we compare the performance of the vanilla diffusion model using isotropic white noise and DDM under the cosine and sigmoid scheduler in Chen (023). Table 3 collects the results. Different schedulers indeed influence the model performance. However, the performance of the vanilla diffusion model with isotropic white noise heavily relies on the hyperpa- rameters of the scheduler. Yet, irrespective of the scheduler employed, our proposed data-dependent anisotropic noise consistently yields superior per- formance.
Table 3: Results for different schedulers.
Nirisn schedile function cosine (am0) o=l.
Veisn
Cileser
PahMeed
MUTAA
0.715
0.824
cosine (em1), em) y
White Noive
0.371
0.453
0.692
sigmeid (s=0, ex3, n
DUM
0.710
0,806
0,777
sigmold 6-3,
White Noise
0.581
0.691
I Lastly, we conduct an ablation study to examine the effects of the two constraints. T presents results, where "w/o R" indicates the removal of Table 4: An ablation study the two constraints.
the constraint moval of both constraints. As shown in Table 4 the introduction of anisotropic Gaussian noise gen- erated through (3) led to significant improvement compared to isotropic Gaussian noise. Further- more, the inclusion of constraint (2) provided an additional and indispensable improvement. This finding further confirms the importance of making the noise in the forward process data-dependent and anisotropic.
2 and "w/o S&R" indicates the re-
Dataset
w/o S&R
w/o R
Citeseer
34.3740.5
60.77+0.2
74.3 = 0,3
PubMed
73.0740.7
77.60+0.4
81.7 0.8
IMDB-M
49.80=0.53
50.87+0.4 49
52.53+0.31
COLLAB
80.5020.36
81.72+0.31 81.0420.17
MUTAG
82.89=1.16
87.25
91.51
6 Conclusions
This paper unveils the presence of anisotropic structures in graphs, which pose challenges for vanilla diffusion models graph representation learning. To address this limitation, we introduce directional diffusion models, a novel class of diffusion models that leverage data-dependent and anisotropic noise to better handle anisotropic structures. Through experiments conducted on 12 benchmark datasets, we demonstrate the effectiveness of our proposed method.
There are several promising avenues for future research. One direction is develop methods that can automatically determine the optimal set of diffusion steps for each dataset, further enhancing the performance of our directional diffusion models. This could involve techniques such as adaptive selection of diffusion steps. Additionally, exploring the application of our method to computer vision and natural language processing tasks holds great potential for advancing these domains. By adapting and extending our directional diffusion models to these areas, we may leverage their inherent strengths to improve representations and enable effective learning tasks such as image recognition, object detection, sentiment analysis, and language understanding.

---

0.7
9.73%
0.715
0.695
0.90
A 87875
0.8927
Accuracy
A8877
0.80
Method Directional
0.5
Directional Noise
White Noise
0.75
White Noise
0.4
9.290
00.325
.0.221
-0.221
0.65
50.6852
0.0491
-0.6646
-0/6544
0.4804
200
400
800
1000
Step
Step
(a) Citeseer
(b) MUTAG
0. 728
7.790
9:7689
4772
0.5787
9.476
Accuracy
Mathod Directional Noibe White Nolse
Accurary
0.45
0.6
0.595
0.40
0.457
0.35
Method
9.3693
0.358
0.354
White Noise
200
400
600
800
1000
200
600
800
1000
Stop
Stop
(c) PubMed
IMDB-M
Figure 5: Comparing the accuracy in downstream tasks when using representations extracted from models trained with directional noise and white noise at every step of the reverse process.
Acknowledgements
FZ. is supported by National Natural Science Foundation of China (12001356), Shanghai Education Development Foundation and Shanghai Municipal Education Commission (Chenguang Program), Zhejiang Lab (Open Research Project NO. 2022RC0AB06), Shanghai Research Center for Data Science and Decision Technology. Fz is Supported by the Fundamental Research Funds for the Central Universities: "High-Quality Development of Digital Economy: An Investigation of Characteristics and Driving Strategies (Grant Numbers 20231 (0139). QS is supported in part by a NSERC Discovery Grant, New Frontiers in Research Fund, and Data Sciences Institute Catalyst Grant.
References
Abstreiter, K., Mittal, S., Bauer, S., Schölkopf, B., and Mehrjou, A. (2021). Diffusion-based representation learning. arXiv preprint arXiv:2105.14257.
Bansal, A., Borgnia, E., Chu, H.-M., J. S.,, Kazemi, H., Huang, F., Goldblum, M., Geiping, J., and Goldstein, T. (2022). Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.0925 093992.
Baranchuk, D., Rubachev, I., Voynov, A., Khrulkov, V., and Babenko, A. (2021). Label-efficient semantic segmentation with diffusion models. arXiv preprint 11X.v:21
Chang, C.-C. and Lin, C.-J. (2011). Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):1-27.
Chen, T. (2023). On the importance of noise scheduling for diffusion models. 10X1v:2301.10972.
arXiv preprint
Choi, J., Lee, J., Shin, C., Kim, S., Kim, H., and Yoon, S. (2022). Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1472-1148).
Dhariwal, P. and Nichol, A. (2021). Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8799
Gao, J., He, D., Tan, X., Qin, T., Wang, I.., and Liu, T.-Y. (2019). Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907. 1907.1007.9999
10

---

Haefeli, K. K., Martinkus, K., Perraudin, N., and Wattenhofer, R. (2022). Diffusion models for graphs benefit from discrete state spaces. arXiv preprint urXiv:2210.01549.
Hassani, K. and Khasahmadi, A. H. (2020). Contrastive multi-view representation learning on graphs. In International conference on machine learning, pages 41 4116-4126. PMLR.
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilisticc models. Advances in Neural Information Processing Systems, 33:6840-688 6851.
Hou, Z., Liu, X., Dong, Y., Wang, C., Tang, J., et al. (2022). Graphmae: Self-supervised masked graph autoencoders, arXiv preprint trXiv:2205.10800
Hu, W. Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, (2020a). Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 18-22133.
Hu, Z., Dong, Y., Wang, K., Chang, K.-W., and Sun, Y. (2020b). Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857-1867.
Jo, J., Lee, S., and Hwang. S. J. (2022). Score-based generative modeling of graphs via the system of stochastic differential equations. arXiv preprint arXiv:2202.025144
Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. (2020). On the sentence embeddings from pre-trained language models. arXiv preprint N10ii:2011100886.
Li, L., Thickstum, J., Gulrajani, I., Liang, P., and Hashimoto, T. B. (2022). Diffusion-Im improves controllable text generation. arXiv preprint arXiv:2205. 14217.
Preechakul, K., Chatthee, N., Wizadwongsa, S., and Suwajanakom, S. (2022). Diffusion autoencoders: Toward meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619-10629)
Qiu, J., Chen, Q.. Dong, Y., Zhang, J. Yang. H., Ding, M., Wang, K., and Tang, (2020), Gec: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1150-1 1160.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695.
Song, J., Meng, C., and Ermon, S. (2020). Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
Sun, F.-Y., Hoffmann, J., Verma, V., and Tang, J. (2019). Infograph: Unsupervised and semi- supervised graph-level representttn learning via mutual information maximization. arXiv preprint arXiv:1908.0100
Thakoor, S., Tallec, C., Azar, M. G., Azabou, M., Dyer, L., Munos, R., Veličković, P., and Valko, M. (2021). Large-scale representation learning graphs via bootstrapping. arXiv preprint arXiv:2102.06514.
Velickovic, P., Fedus, W., Hamilton, W. L., Liò, P., Bengio, Y., and Hjelm, R. D. (2019), Deep graph infomax. ICLR (Poster), 2(3):4.
Xu, D., Cheng, W., Luo, D., Chen, H., and Zhang, x. (2021). Infogel: Information-aware graph contrastive learning. Advances in Neural Information Processing Systems, 34:30414-30425.
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2018). How powerful are graph neural networks? arXiv preprint arXiv: 181.0008000
Yanardag, P. and Vishwanathan, S. (2015). Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1365-1374.
11

---

Yang, Z.. Cohen, W., and Salakhudinov, R. (2016). Revisiting semi-supervised learning with graph embeddings. International conference on machine learning, pages 40-48. PMLR.
Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., and Leskovec, J. (2018). Hierarchical graph representation learning with differentiable pooling, Advances in neural information processing systems, 31.
You, Y., Chen, T., Shen, Y., and Wang, Z. (2021). Graph contrastive learning automated. In International Conference Machine Learning, pages 12121-12132 PMLR.
You, Y., Chen, T., Sui, Y., Chen, T., Wang, and Shen, Y. (2020). Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:58 12-5823.
Zhang, H., Wu, Q., Yan, J., Wipf, D., and Yu, P. S. (2021). From canonical correlation analysis
to self-supervised graph neural networks. Advances in Neural Information Processing Systems, 34:76-89.
Zhang, Z., Zhao, Z., and Lin, Z. (2022). Unsupervised representation learning from pre-trained diffusion probabilistic models. arXiv preprint arXiv:2212.12990.
Zhu, Y. Xu, Y., Yu, F., Liu, Q. Wu, S., and Wang. L. (2020). Deep graph contrastive representation learning. arXiv preprint arXiv:2006.0413)
12


Reviews:
[{"review_id": "dJHqahNYeC", "reviewer": "Reviewer_emkS", "summary": "The authors address the gap in unsupervised graph representation learning by exploring the use of diffusion models. They propose directional diffusion models that incorporate data-dependent, anisotropic, and directional noises in the forward diffusion process to better handle anisotropic structures in graphs. Experiments on publicly available datasets showcase the superiority of their models over state-of-the-art baselines, demonstrating their effectiveness in capturing meaningful graph representations. Overall, the paper presents a compelling approach that contributes to the advancement of unsupervised graph representation learning.", "questions": "1. \"µ and σ are calculated using graphs within the batch.” -- what is done during inference? why not just pre-calculate those on the entire train split?\n2. \"allowing the latent code to contain meaningful compressed knowledge while preventing the decoder from degrading into an identical mapping” -- why will that happen that requires a dedicate MLP? what is this different from the simple U-Net used in the image domain?\n3. Algorithm 2 in appendix, line 6, what is $A_t$?\n4. In some cases, the SNR goes even higher for larger diffusion steps (like in the Computer dataset for instance), why is that?\n5. How is the entire graph-level representation obtained?\n6. Referring to figure 2, shouldn't the SNR be 0 in the last diffusion steps? It doesn't reach zero as the directional noise keeps them separated, is that a wanted outcome? If so, why?\n7. Why not apply just a different noise scheduler? instead of linear, coisne for instance. Similar observation was shown on high-resolution images and different schedulers where proposed [1]. I know it is still isotropic noise, so at least show results as a baseline/ablation.\n\n\n___\n\n[1] Chen, Ting. \"On the importance of noise scheduling for diffusion models.\" arXiv preprint arXiv:2301.10972 (2023).", "limitations": "The authors address the limitations of their work, and support their claims with experiments and ablations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Motivation**\n\n    The introduction is well-motivated, providing a thorough explanation of the challenge and task at hand. The authors go beyond textual descriptions and use simple visualizations on both real and synthetic data to demonstrate their points effectively. This approach enhances the clarity and understanding of the presented research, making it accessible to a wider audience.\n&nbsp;\n\n2. **Method**\n\n    The authors present a straightforward yet effective solution for incorporating directional noise into node embeddings. This approach effectively addresses the challenge posed by anisotropic structures in various domains, including graphs and potentially text data. Their proposed method demonstrates promising results in handling directional noise and enhancing the quality of node and graph embeddings.\n&nbsp;\n\n3. **Architecture**\n\n    I appreciate the authors intention to adapt the well-known and effective U-Net architecture from the image domain to the graph domain. The incorporation of skip connections in the U-Net is particularly relevant for denoising tasks. This thoughtful adaptation enhances the model's ability to handle graph-related denoising effectively.\n&nbsp;\n\n4. **Experiments**\n\n    The authors conduct a comprehensive comparison with numerous baselines across multiple datasets. Furthermore, their evaluation settings, which involves 10-fold cross-validation with standard deviation after five runs, are robust and reliable. This rigorous evaluation methodology ensures the validity and statistical significance of their results.", "weaknesses": "1. **missing releted work**\n\n    There are existing works in the intersection of graphs and diffusions are missing, contradicting the authors statement \"To the best of our knowledge, there have been no works for diffusion-model-based graph representation learning.\". Some for example:\n    - Niu, Chenhao, et al. \"Permutation invariant graph generation via score-based generative modeling.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.\n    - Xu, Minkai, et al. \"Geodiff: A geometric diffusion model for molecular conformation generation.\" International Conference on Learning Representations, 2022.\n    - Vignac, Clement, et al. \"Digress: Discrete denoising diffusion for graph generation.\" International Conference on Learning Representations, 2023.\n\n    Furthermore, some simple techniques can be applied to create a smoother SNR curves over the different diffusion steps. For instance:\n    - Chen, Ting. \"On the importance of noise scheduling for diffusion models.\" arXiv preprint arXiv:2301.10972 (2023).\n\n    The same problem was presented in it over the image domain (on high-resolution images), and the solution was to use different noise schedulers. Why not simply try this trick?\n&nbsp;\n\n2. **missing details**\n\n   There are missing details in the paper that makes it hard to fully understand and reproduce the papers results. For example:\n    - \"µ and σ are calculated using graphs within the batch.” -- what is done during inference? it is EMA over what was been seen in training time? does it handle only batch inference?\n    - They paper did not state how exactly the entire graph-level representation is obtained, it is sum/mean of all node representations?\n    - Algorithm 2 in appendix, line 6: what is exactly $A_t$? should it be just $A$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n> **missing releted work,** here are existing works in the intersection of graphs and diffusions are missing ....\n\nFirst, with full respect, the three existing works you mentioned do not conflict with our statements. These three works you pointed out are devoted to exploring graph structure generation based on diffusion models, whose core goal is to **generate graph–discrete graph structure**. However, instead of graph structure generation, our research focuses on unsupervised graph representation learning tasks, whose purpose is to **generate node-level or graph-level vector unsupervised learning**. We first pre-train a model without supervision and use it to generate node or graph representations for downstream classification tasks.\n\n> Furthermore, some simple techniques can be applied to create a smoother & Why not apply just a different noise scheduler?Why not apply just a different noise scheduler? ....(Question 7)\n> \nRegarding the impacts of different schedules, We adopt the same experimental setup as described in the article you referenced.  The experimental results are shown in the following table:\n\n| Noise schedule function | Noise type | Citeseer | PubMed | MUTAG |\n| --- | --- | --- | --- | --- |\n| cosine (s=0,e=1,τ = 1) | DDM | 0.715 | 0.824 | 0.867 |\n| cosine (s=0,e=1,τ = 1) | White Noise | 0.371 | 0.453 | 0.692 |\n| sigmoid (s=-3,e=3,τ = 1) | DDM | 0.735 | 0.803 | 0.889 |\n| sigmoid (s=-3,e=3,τ = 1) | White Noise | 0.672 | 0.606 | 0.689 |\n| sigmoid (s=0, e=3,τ = 1) | DDM | 0.710 | 0.806 | 0.877 |\n| sigmoid (s=0, e=3,τ = 1) | White Noise | 0.581 | 0.434 | 0.691 |\n\nDifferent schedulers do indeed influence the final effectiveness of the model. However, due to the data-independent nature of white noise, its ultimate performance heavily relies on the hyperparameters of the scheduler. Yet, irrespective of the scheduler employed, our proposed data-dependent anisotropic noise consistently yields superior outcomes. This confirms our impression of anisotropic structures in section 1.\n> µ and σ are calculated using graphs within the batch.” -- what is done during inference? ....\n> \nIn this paper, $\\mu$  and $\\sigma$ are also calculated in mini-batch during the model inference stage.  In node classification tasks, the term \"mini-batch\" refers to the nodes currently included in the computation, a requirement that can be easily fulfilled in node classification tasks. In graph classification tasks, \"mini-batch\" refers to the graphs currently included in the computation.\n> why not just pre-calculate those on the entire train split\n> \nDuring model inference, compared to pre-calculate or EMA, calculating the *μ* and *σ* in the mini-batch provides a more effective constraint on the diffusion process within the local neighborhood of the batch. This approach can more efficiently ensure a higher SNR.\nIn addition, it is worth noting that in order to prevent inference batch size from affecting, we adopted the same batch size in the GraphMAE [2] experiment to ensure the fairness of the benchmark.\n> why will that happen that requires a dedicate MLP? what is this different from the simple U-Net used in the image domain?\n> \nThe MLP in our network is composed of linear – relu – linear, which projects the feature map from the hidden dimension back to the data dimension. This is necessary from a computational point of view. By introducing such a nonlinear structure for dimension conversion, GAT can be restricted to learning in the hidden space. This design has proven effective in many classic self-supervised training methods (simclr, simclrv2, SimSiam, moco). Our statement is a possible explanation for the effectiveness of this design. If this structure, containing nonlinearity, is removed, the latter half of the decoder will have to learn the mapping from the hidden space to the original data distribution, and the former half of the decoder may degenerate into an identity map of the encoder part. We also provide more ablation experiments related to the network structure in the responses to reviewer 2.\n> The paper did not state how exactly the entire graph-level representation is obtained, it is sum/mean of all node representations?How is the entire graph-level representation obtained? (Question 5)\n> \nYes, graph-level representation is derived from pooling node-level representations, and we employ either max-pooling or mean-pooling. The specific approach remains consistent with that of the GraphMAE [2] baseline.\n> Algorithm 2 in appendix, line 6, what is $A_{t}$\n> \nThank you for your question and we apologize for our negligence. This is just a typo and it should be $A$. Our proposed DDM is for node features.\n> Referring to figure 2, shouldn't the SNR be 0 in the last diffusion steps?\n> \nThis phenomenon is consistent with our design. In section 4, our method transforms the data-independent Gaussian noise into an anisotropic and batch-dependent noise ϵˉ and ensures that noise ϵˉ into the same hyperplane of the feature. This makes $\\epsilon$ related to $x_{0}$ so that SNR may not reach 0 at the last diffusion steps .\n> In some cases, the SNR goes even higher for larger diffusion steps....\n> \nWe believe that this phenomenon arises from the inherent characteristics of certain graphs data. In fact, introducing slighter noise to node features on some graph data has been a well-established technique to enhance classification accuracy. This concept is elaborated [1], So our approach better leverages this particular characteristic of the data and renders it prominent enough to be observable in the SNR curve.\n\n[1] Sato, R., Yamada, M., & Kashima, H. (2021). Random features strengthen graph neural networks. In *Proceedings of the 2021 SIAM international conference on data mining (SDM)* (pp. 333-341). Society for Industrial and Applied Mathematics.\n[2] Hou, Zhenyu, et al. \"Graphmae: Self-supervised masked graph autoencoders.\" *Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*. 2022."}, {"review_id": "bGYIGoP7Xd", "reviewer": "Reviewer_BUxE", "summary": "This work proposes a class of diffusion models to improve the accuracy of graph representation learning. The model incorporates both data-dependent and anisotropic noise in the forward noising process, by scaling its magnitude and direction based on each coordinate of the data. This structured noise maintains the signal present in the data over longer time windows than using standard isotropic white noise during the forward noising process. The authors show that this model improves upon state-of-the-art methods on a large collection of benchmark datasets. Moreover, they perform an ablation study to understand the effect of the two proposed modification to the noise process.", "questions": "An important choice for defining the representations are the user selected time-steps where the outputs are extracted from the denoising network. Do the authors have a guideline or recommendation (e.g., based on the empirical studies) for how to choose these time steps. This seems particularly relevant given that the SNR does not degrade monotonically for some datasets (e.g., IMDB in Figure 2). \n\nEven if the SNR in standard diffusion models decay quickly with increasing time, how do the results presented here compare to a baseline diffusion model with isotropic noise where the time-steps are chosen \"optimally\". For example, what if the time-steps are chosen all near the initial time, $t = 0$, when there is still signal contained in the noisy data?", "limitations": "The authors show improved results on almost all datasets. It would be great if the authors could comment on what leads to the similar performance on the Ogbn-arxiv dataset. Is it because the data is more isotropic in this case? When do the authors expect the algorithm to under-perform for the evaluated tasks?\n\nIn addition to the questions above, it would be good to address these minor comments:\n- Define the acronym for their proposed framework, DDM\n- Include standard errors for the results in Table 3\n- Explain why $f_\\theta$ depends on $A$ in equation (4). It is sufficient to highlight that this is the structure of a GNN, which may be helpful for some readers.\n- Clarify in Figure 4 that the directional noise is only added to $X_0$ and not $A$.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors visually explain and empirically demonstrate the effect of using non-isotropic noise in the forward diffusion process to improve classification tasks. These noise processes provide clear intuition why this is preferred to white noise for these tasks.\n- The authors propose a novel strategy for extracting graph representations based on time-dependent denoiser that combines graph neural networks and UNet architectures.\n- The application of diffusion models to graph representation learning is novel and the new model is shown to yield superior results to existing algorithms.", "weaknesses": "It would be great for the authors to comment and compare with other non-isotropic noise processes that have been proposed for diffusion models for sampling and generative modeling. Some examples outside of the graph representation learning context are: \n* Score-based Denoising Diffusion with Non-Isotropic Gaussian Noise Models, Vikram Voleti, Christopher Pal, Adam Oberman, 2022\n* Blurring Diffusion Models, Emiel Hoogeboom, Tim Salimans, 2023\n\nIn addition to the changes in the noise process, the authors propose a specific architecture for the denoiser, and selection of representative features from their hidden layers. This architecture could be relevant on its own to extract representations of the dataset, without the denoising time components. Do any of the compared methods investigate how this denoiser architecture, without the diffusion model, would perform for representation learning. This might be helpful as an additional ablation study to see the effect of the architecture. \n\nThe metrics used to evaluate their experiments can be described in more detail. The values for the results in Tables 1,2,3 were not clearly mentioned in the caption or the main text. Mathematical equations may also be helpful to precisely describe the accuracy measurements in Figure 5. \n\nThe authors comment that the word vectors often exhibit greater anisotropy, which yields superior performance in node classification tasks. It would be great if the authors could quantify this to validate this claim that the performance improves with more anisotropy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> It would be great for the authors to comment and compare with other non-isotropic noise processes\n>\nWith full respect, The existing literature you mentioned differs fundamentally from our approach. As pointed out by reviewer 6, our paper proposes that the ultimate distribution of the diffusion process isn't essential; what's crucial is the rate of SNR attenuation. Thus, the noise we propose is data-dependent, with its distribution determined by the mini-batch data.(Section 4  eq2 & eq3) .The main objective of this design is to curb the speed of SNR reduction. In contrast, existing literature still relies on methods that are non-data-dependent, using fixed noise distributions.\n\n> This might be helpful as an additional ablation study to see the effect of the architecture.\n> \nWe sort out the network structures of the baselines that appeared in the comparison experiments below. Existing schemes, including contrastive learning, GrapMAE, and our proposed method use similar basic structures (gat) and amounts of parameters, which shows that our proposed unsupervised representation learning based on diffusion models, plays a dominant role in improving the representation. In addition, we conduct further ablation experiments for our special designs in the denoiser architecture, including symmetric skip-connection, and symmetric network structure. The results are as follows, which prove the correctness of choosing to transplant the U-net idea and the effectiveness of the detailed design.\n|  | GCN | MLP | Number layers of GCN |\n| --- | --- | --- | --- |\n| GraphMAE | ✓ | ✓ | 2-4 |\n| MVGRL | ✓ | ✓ | 4 |\n| Our(DDM) | ✓ | ✓ | 4 |\n\n|  | Citeseer | PubMed | MUTAG |\n| --- | --- | --- | --- |\n| wo-head | 73.1±0.2 | 80.2±0.2 | 87.8±1.4 |\n| wo-encoder | 73.4±0.1 | 81.4±0.3 | 88.9±1.3 |\n| wo-skip_connection | 73.5±0.2 | 81.3±0.5 | 86.7±1.1 |\n| Baseline | 74.3±0.3 | 81.7±0.8 | 91.51 ±1.4 |\n\n> The metrics used to evaluate their experiments can be described in more detail.\n> \nWe explained the evaluation metrics used in Sections 5.2 and 5.3. For node classification tasks, we report the classification accuracy of nodes in the validation set. And for graph classification tasks, we report the accuracy of the validation set. We will add this to the caption.\n\n> The authors comment that the word vectors often exhibit greater anisotropy, ...... It would be great if the authors could quantify this to validate the claim that the performance improves with more anisotropy. \nThe authors show improved results on almost all datasets. It would be great if the authors could comment on what leads to the similar performance on the Ogbn-arxiv dataset. (Question 1)\n> \nWe follow the method in [1] to measure the anisotropy of the data by measuring the ratio of the sum of the first two singular values to the sum of the first five singular values. The results are as follows:\n\n|  | Top@2/Top@5 |\n| --- | --- |\n| ogbn-arxiv | 0.460 |\n| citseer | 0.532 |\n| Pubmed | 0.533 |\n| Computer | 0.722 |\n| Cora | 0.419 |\n\nWe can observe that on datasets with low anisotropy (ogbn-arxiv, cora), our approach performs comparably to previous methods but falls short of supervised models. On datasets with strong anisotropy (Pubmed, Citeseer, Computer), our model has made significant strides, even surpassing some supervised training outcomes. This indicates that our approach can better learn the specific structure of such graph data.\n\n> An important choice for defining the representations are the user selected time-steps where the outputs are extracted from the denoising network. Do the authors have a guideline or recommendation (e.g., based on the empirical studies).....\n> \n\nThank you for your inquiry. We are pleased to offer our guidelines in this regard.\nGenerally speaking, the signal-to-noise ratio indicates important information in both the training and inference stages. From fig5 in the main text and the fig2 in the appendix, we can see that the trend of SNR is almost proportional to the accuracy of the test set. For example, when T-->1000, SNR of Citseer 1.5-->0.5, acc 0.741-->0.695. Additionally, considering that our final metric surpasses the performance of each individual timestep. we suggest that the selected multi time-step can cover different intervals of the SNR curve to get the best results.\n\n> Include standard errors for the results in Table 3\n> \nWe have updated the new table with the standard deviations below.\n\n\n| DataSet | w/o S&R    | w/o R      | Full       |\n|---------|------------|------------|------------|\n| Citseer | 34.37±0.5  | 60.77±0.2  | 74.3 ± 0.3 |\n| PubMed  | 73.07±0.7  | 77.60±0.4  | 81.7 ± 0.8 |\n| IMDB-M  | 49.80±0.53 | 50.87±0.49 | 52.53±0.31 |\n| COLLAB  | 80.50±0.36 | 81.04±0.17 | 81.72±0.31 |\n| MUTAG   | 82.89±1.16 | 87.25±1.12 | 91.51±1.45 |\n\n\n> Explain why $f_\\theta$ depends on $A$ in equation (4). It is sufficient to highlight that this is the structure of a GNN, which may be helpful for some readers.\n> \nIn this paper, we parameterize $f_\\theta$ as GCN. We will correct this unclear expression in the subsequent version.\n> Clarify in Figure 4 that the directional noise is only added to $A_{0}$ and not $A$.\n> \nThank you for your review, noise is indeed only added to x, and the specific form is in equation (1), (2), (3).\n\n[1] Gao J, He D, Tan X, et al. Representation degeneration problem in training natural language generation models[J]. arXiv preprint arXiv:1907.12009, 2019."}, {"review_id": "rkOaPIVU92", "reviewer": "Reviewer_jRHd", "summary": "This study proposed adding directional noise on learning anisotropic graphs using diffusion models. The study adds new perspectives in exploring the anisotropic structures in graph data. The numerical results are promising to support the authors' ideas.", "questions": "The authors have performed extensive experiments, but it would add more evidence if the authors can add some theoretical proof. \n\nFor figure 3, does each dot represent a graph? Are they from simulated datasets?\n", "limitations": "The authors have discussed several future directions that can be used to improve the current model.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I find this is an interesting paper - the rational is convincing, and the authors performed extensive experiments to support the idea. The discussions on the noise and ablation studies provide further insights into understanding the usefulness of adding directional noises. The paper provides a valuable perspective in understanding diffusion models for graph learning.", "weaknesses": "The authors mainly prove the utility of the proposed approach through experiments; there is a lack of theorectical proof. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> For figure 3, does each dot represent a graph? Are they from simulated datasets?\n> \n\nThanks for your question, Figure 3 shows a set of simulated data. Each sample point comes from an anisotropic (covariance is not equal to the identity matrix) normal distribution. As discussed in section 4, the simulation is designed to show that when the data distribution is anisotropy, our proposed directional can better preserve the structural information of the data during the forward process.\n\n> The authors have performed extensive experiments, but it would add more evidence if the authors can add some theoretical proof.\n> \n\nThanks for your advice! Indeed, in this paper, we didn’t provide some theoretical proof. However, we believe that the problem studied in this paper and our contributions are considerably interesting and extensible. Just as mentioned by Reviewer 5, for the graph learning community, our research introduces a novel paradigm for graph representation learning and provides comprehensive experimental evidence of the effectiveness of this paradigm. For the diffusion model community, our study also demonstrates impacts on specific data distributions with diffusion models and we propose a data-dependent forward process. And thus, as a pioneering work studying the impacts of data and focusing on data-dependent cases, it is difficult to formulate our method into existing theoretical frameworks. We are actively working on the theoretical proof and believe that it will be addressed in the future."}, {"review_id": "1hscHv5wWr", "reviewer": "Reviewer_tkhv", "summary": "This paper presents a method named Directional Diffusion Model (DDM) for unsupervised representation learning, targeting applications in graph and node classification. The model's performance is evaluated on various benchmark datasets and compared to both unsupervised and supervised models. The results demonstrate that DDM can effectively learn meaningful graph- or node-level representations. Furthermore, the paper offers some exploration of how different types of noise impact the learning process.", "questions": "1. Please clarify the issues raised in the weaknesses section.\n\n2. How does DDM compare with widely-known baselines in terms of performance, and are there any specific scenarios where DDM particularly excels or falls short?\n\n3. Regarding the means and deviations at line 170: are these calculated relative to the node features at time 0, or do they consider other time steps?\n\n\nIn its current form, I believe the paper does not meet the rigorous quality and contribution standards expected at a top-tier conference like NeurIPS.", "limitations": "The authors did not explicitly state any limitations of their study. However, the identified weaknesses could be viewed as potential limitations. These include the lack of theoretical foundations and potential issues in reproducibility due to insufficient implementation details. Additionally, the paper could benefit from a clear discussion of the limitations of DDM in handling specific scenarios or types of data.\n", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors present empirical evaluations of DDM on multiple benchmark datasets, which validate the effectiveness of the proposed method to some extent.\n\n2. The paper includes an investigation into the impact of different types of noise, which adds depth to the analysis and understanding of the proposed method.", "weaknesses": "1. The SVD visualizations could be more insightful. It's observed that the 2D projections computed from graph datasets appear biased and are predominantly on the right plane. However, the methodology behind these visualizations remains unclear. Are they based on singular values? Further explanation would be beneficial.\n\n2.  The Signal-to-Noise Ratio (SNR) plots in the supplementary material show comparable results under white noise and directional noise. This implies that the DDM may not offer significant advantages on these datasets. The authors should provide some clarification.\n\n3. The paper is primarily empirical and lacks theoretical foundations, which might limit its generalizability.\n\n4.  The algorithm snippet doesn't seem to provide sufficient implementation details, which might impede attempts to reproduce the study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The SVD visualizations could be more insightful. However, the methodology behind these visualizations remains unclear.\n> \nThanks for your suggestions,  Using the SVD decomposition to analyze the anisotropic is first proposed by [1] and has been widely used in NLP[1,2,3,4,5]. Specifically, we calculate the SVD decomposition of the graphs’ node feature matrix and then choose the first and second singular vectors as the x-axis and y-axis.  The methodology behind these visualizations, as discussed in section 1, line 44, is to gain an intuitive understanding of the differences in distribution between graph data and image data. Prompted by this, we further examine the disadvantage of the vanilla diffusion model through the signal-to-noise ratio under such data and propose our new pipeline considering preserving the characteristic of data in the diffusion process.\nFrom a singular value perspective, we can draw similar conclusions about anisotropy. This method was also introduced in [1].  We follow the method in [1] to measure the anisotropy of the data by measuring the ratio of the sum of the first two singular values to the sum of the first ten singular values. The results are as follows,\n\n> The Signal-to-Noise Ratio (SNR) plots in the supplementary material show comparable results under white noise and directional noise.....\n> \nWe follow the approach of [1] to measure the anisotropy of the data by measuring the ratio of the sum of the first two singular values to the sum of the first ten singular values. The results are as follows:\n\n|  | Top@2/Top@5 |\n| --- | --- |\n| ogbn-arxiv | 0.460 |\n| citseer | 0.532 |\n| Pubmed | 0.533 |\n| Computer | 0.722 |\n| Cora | 0.419 |\n\nWe can observe that on datasets with low anisotropy (Ogbn-arxiv, cora), our approach performs comparably to previous methods and falls short of supervised models. On datasets with strong anisotropy (Pubmed, Citeseer, Computer), our model has made significant strides, even surpassing some supervised training outcomes. As we discussed in section 5.1 & section .5.2  By utilizing the directional noise diffusion, our method acts as a pseudo-infinite-step data augmentation technique that generates numerous samples while preserving data structure.\n> The paper is primarily empirical and lacks theoretical foundations, which might limit its generalizability.\n> \nIndeed, in this paper, we didn’t provide some theoretical proof. But we believe that the problem studied in this paper and our contributions are considerably interesting and extensible. Just as mentioned by Reviewer 5, for the graph learning community, our research introduces a novel paradigm for graph representation learning and provides comprehensive experimental evidence of the effectiveness of this paradigm. For the diffusion model community, our study also demonstrates impacts on specific data distributions with diffusion models and we propose a data-dependent forward process. And thus, as a pioneering work studying the impacts of data and focusing on data-dependent cases, it is difficult to formulate our method into existing theoretical frameworks. We are actively working on the theoretical proof and believe that it will be addressed in the future.\n\n> The algorithm snippet doesn't seem to provide sufficient implementation details, which might impede attempts to reproduce the study.\n> \n\nWe apologize for not providing the definitions of $X_i$ and $A$ in the algorithm in Appendix: as we have defined in the paper, $X_i$ is the feature matrix of graph $\\mathcal{G}_i$， while $A$ is the adjacency matrix of $\\mathcal{G}$. Every time taking the gradient step, the parameters $\\theta$ are renewed.\n\n> How does DDM compare with widely-known baselines in terms of performance, and are there any specific scenarios where DDM particularly excels or falls short?\n> \n\nActually, we have conducted extensive experiments in section 5.1 and section 5.2 to compare our models with baselines, and the results are shown in Table 1 and Table 2: in graph classification tasks, we surpass the performance of all baselines on IMDB-B, IMDB-M, COLLAB, PROTEINS, and MUTAG datasets; in node classification tasks, we outperform those baselines on Citesser, PubMed, Amazon-Computer and Amazon-Photo datasets. The falling short and excelling performance are also discussed in section 5, which are attributed to the isotropy of data in short.\n\n> Regarding the means and deviations at line 170: are these calculated relative to the node features at time 0, or do they consider other time steps?\n> \n\nAs mentioned in line 171, $\\mu$ and $sigma$ are the means are deviations of the raw features vectors at time 0, and they can alleviate the impacts resulting from the rapid decline of SNR as discussed in section\n\n[1] Gao J, He D, Tan X, et al. Representation degeneration problem in training natural language generation models[J]. arXiv preprint arXiv:1907.12009, 2019.\n\n[2] Qiu R, Huang Z, Yin H, et al. Contrastive learning for representation degeneration problem in sequential recommendation[C]//Proceedings of the fifteenth ACM international conference on web search and data mining. 2022: 813-823.\n\n[3] Zou D, Wei W, Mao X L, et al. Multi-level cross-view contrastive learning for knowledge-aware recommender system[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022: 1358-1368.\n\n[4] Wang L, Huang J, Huang K, et al. Improving neural language generation with spectrum control[C]//International Conference on Learning Representations. 2019.\n\n[5] Yu S, Song J, Kim H, et al. Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022: 29-45."}, {"review_id": "9j7aSdrXke", "reviewer": "Reviewer_8dwQ", "summary": "This paper offers a good study on anisotropic and directional structure learning of the diffusion model. The authors first conduct a statistical analysis emphasizing the importance of anisotropic structure in graph dataset and demonstrate the disadvantage of the vanilla diffusion model through signal-to-noise ratio. Later on, they propose their work, which is a new pipeline considering preserving the characteristic of data in the diffusion process. Its idea is simple: adding directional diffusion on data, preserving more information than vanilla diffusion. Its model is simple, leveraging 4 layers GNNs and one layer MLP, it achieves improvement in several benchmarks. The idea is elegant.", "questions": "1. Do you have proof of the effectiveness of your idea?\n2. Can you offer more experimental results?", "limitations": "1. They haven't provided limitations for their work. \n2. Though the idea is inspirational, the writing is not good, and there are too many writing mistake.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Good idea! The work not only offers inspiration to the graph learning community, but it also contributes to the diffusion model. The two constraints added in the work are elegant and proper.", "weaknesses": "The writing is not so well, especially the formula part. There are too many mistakes there. Their performances aren't satisfying, but I believe the underlying problem doesn't come from the pipeline, it comes from the simple model. There aren't convergence proofs for their learning architecture. Noticed that the diffusion model, which stems from the quasi-static process, possesses a good property of converging. I hope to see proof of the effectiveness of the constraints adding to the original process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Do you have proof of the effectiveness of your idea?\n> \n\nIndeed, in this paper, we didn’t provide some theoretical proof. However, we believe that the problem studied in this paper and our contributions are considerably interesting and extensible. Just as mentioned by Reviewer 5, for the graph learning community, our research introduces a novel paradigm for graph representation learning and provides comprehensive experimental evidence of the effectiveness of this paradigm. For the diffusion model community, our study also demonstrates impacts on specific data distributions with diffusion models and we propose a data-dependent forward process. And thus, as a pioneering work studying the impacts of data and focusing on data-dependent cases, it is difficult to formulate our method into existing theoretical frameworks. We are actively working on the theoretical proof and believe that it will be addressed in the future.\n\n> Can you offer more experimental results?\n> \nExtensive results on baselines (2 tasks and 12 datasets) demonstrate that this method provides a novel solution for graph representation learning. Apart from contrast experiments, we added an ablation experiment on the model structure: \n|          | GCN | MLP | Number layers of GCN |\n|----------|-----|-----|----------------------|\n| GraphMAE | ✓   | ✓   | 2-4                  |\n| MVGRL    | ✓   | ✓   | 4                    |\n| Our(DDM) | ✓   | ✓   | 4                    |\n\n|  | Citeseer | PubMed | MUTAG |\n| --- | --- | --- | --- |\n| wo-head | 73.1±0.2 | 80.2±0.2 | 87.8±1.4 |\n| wo-encoder | 73.4±0.1 | 81.4±0.3 | 88.9±1.3 |\n| wo-skip_connection | 73.5±0.2 | 81.3±0.5 | 86.7±1.1 |\n| Baseline | 74.3±0.3 | 81.7±0.8 | 91.51 ±1.4 |\n\nit can be seen that considering the similar parameter amount and modules with the baseline, our introduction of diffusion pre-training such as the U-net structure plays an important role. \nMoreover, as we discussed with review 2 by quantifying the anisotropy of data, we show that our method can achieve better performance on datasets with strong anisotropy"}, {"review_id": "vaX1caTeEl", "reviewer": "Reviewer_ejpR", "summary": "The authors present directional diffusion model (DDM) to learn graph / node representation. Compared to vanilla diffusion based representation learning techniques, DDM adds a batch based mean and variance, as well as preserving direction of diffusion. The authors then demonstrate that their model perform better than other models on representating learning by experimenting on various dataset and compare with other methods.", "questions": "- A lot of citations by the paper, including some baselines like Infograph and GraphMAE, come from arxiv. Are these methods reliable?\n- Directional diffusion preserves direction, so in recovery process, won't the representation fail to learn the direction of the entries? If it doesn't learn the directions, does directions don't matter?", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- It is a nice finding by the authors that compared to diffusion model for generative modeling, diffusion model for representation learning does not require sampling from final distribution, thus we don't need to know the limiting behavior of the diffusion process.\n- The experiments supports the authors' claim on the performance of the model.", "weaknesses": "- The authors should include a brief introduction on how diffusion models work on representation learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n> A lot of citations by the paper, including some baselines like Infograph and GraphMAE, come from arxiv. Are these methods reliable?\n> \nWe apologize for our problem, in section 5, the articles we compared and cited are all published in top-tier conferences, as follows： GIN-ICLR 201, DiffPool-NIPS 2018, GCC-KDD 2020, JOAO-ICML 2021, GraphMAE-KDD 2022, MVGRL-ICML 2022, GPT-GNN-KDD 2022, DGI-ICLR 2019, GRACE-ICML 2020, BGRL-ICLR 2022, InfoGCL - NIPS 2021, CCS-SSG nips 2021\nWe will correct the problem of citing the arxiv version of GIN, GraphMAE, GARGE, BGRL in the subsequent version.\n\n> Directional diffusion preserves direction, so in recovery process, won't the representation fail to learn the direction of the entries? If it doesn't learn the directions, does directions don't matter?\n> \nIn this paper, $\\mu$ and $\\sigma$ in section 3 are calculated in mini-batch during the model training stage. The core idea of this design is that, through random batch learning of multiple epochs, the model can both approximate the global (law of large numbers) and retain the signal-to-noise ratio in each learning process.\n\n> The author should include an introduction on how diffusion models work on representation learning.\n> \n\nThanks for your suggestions. Actually, Section 1, the introduction part of our article has discussed some notable and recent works about how diffusion models work in representation learning.  We would like to provide a thorough summary in this regard.\n\nSeveral methods based on diffusion models[1,2,3,4] have been proposed for effective representation learning. Notably, [4] have demonstrated the value of intermediate activations obtained from denoising networks, as they contain valuable semantic information that can be utilized for tasks like image representation and semantic segmentation. Their findings emphasize the effectiveness of diffusion models in learning meaningful visual representations. More recently, [5] has revealed that the restoration of data corrupted with specific noise levels provides an appropriate pretext task for the model to learn intricate visual concepts, and prioritizing such noise levels over other levels during training improves the performance of diffusion models.\n\n[1] Zhang Z, Zhao Z, Lin Z. Unsupervised representation learning from pre-trained diffusion probabilistic models[J]. Advances in Neural Information Processing Systems, 2022, 35: 22117-22130.\n\n[2] Preechakul K, Chatthee N, Wizadwongsa S, et al. Diffusion autoencoders: Toward a meaningful and decodable representation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 10619-10629.\n\n[3] Abstreiter K, Mittal S, Bauer S, et al. Diffusion-based representation learning[J]. arXiv preprint arXiv:2105.14257, 2021.\n\n[4] Baranchuk D, Rubachev I, Voynov A, et al. Label-efficient semantic segmentation with diffusion models[J]. arXiv preprint arXiv:2112.03126, 2021.\n\n[5] Choi J, Lee J, Shin C, et al. Perception prioritized training of diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 11472-11481."}]

Author rebuttal:

Meta-review: The reviewers originally had some concerns but these have mostly been addressed by the authors in the rebuttal and all reviewers now lean for acceptance. According to the reviewers feedback, please incorporate all the discussion in the updated manuscript, paying special attention to adding more context in the background section and being more precise about the metrics used in the evaluations.
Decision: Accept (poster)