SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Run Yang
SUFE and Baidu
luckyyangrun@163.sufe.edu.cn
Yuling Yang
SUFE
sibyllayang@163.sufe.edu.cn
Fan Zhou*
zhoufan@mail.shufe.edu.cn
Qiang Sun*
qiang.sun@utoronto.ca
Diffusion models have achieved remarkable success in diverse domains such as image synthesis, super-resolution, and 3D molecule generation. Surprisingly, the application of diffusion models in graph learning has garnered little attention. In this paper, we aim to bridge this gap by exploring the use of diffusion models for unsupervised graph representation learning. Our investigation commences with the identification of anisotropic structures within graphs and the recognition of a crucial limitation in the vanilla forward diffusion process when dealing with these anisotropic structures. The original forward diffusion process continually adds isotropic Gaussian noise to the data, which may excessively dilute anisotropic signals, leading to rapid signal-to-noise conversion. This rapid conversion poses challenges for training denoising neural networks and obstructs the acquisition of semantically meaningful representations during the reverse process. To overcome this challenge, we introduce a novel class of models termed directional diffusion models. These models adopt data-dependent, anisotropic, and directional noises in the forward diffusion process. In order to assess the effectiveness of our proposed models, we conduct extensive experiments on 12 publicly available datasets, with a particular focus on two distinct graph representation learning tasks. The experimental results unequivocally establish the superiority of our models over state-of-the-art baselines, underscoring their effectiveness in capturing meaningful graph representations. Our research not only sheds light on the intricacies of the forward process in diffusion models but also underscores the vast potential of these models in addressing a wide spectrum of graph-related tasks. Our code is available at https://github.com/statsle/DDM.
Unsupervised representation learning through diffusion models has emerged as a prominent area of research in computer vision. Several methods that leverage diffusion models, such as those proposed by Zhang et al. (2022); Preechakul et al. (2022); Abstreiter et al. (2021); Baranchuk et al. (2021), have been put forth for representation learning. Notably, Baranchuk et al. (2021) showed that the intermediate actiations obtained from denoising networks contain valuable semantic information. Their findings emphasize the effectiveness of diffusion models in learning meaningful
(b) Amazon-Photo
(c)IMDB-M
Figure 1: 2D visualization of the data using SVD decomposition. (a) Visualization of the node features in CIFAR-10, with different colors indicating different labels. (b) Visualization of the two classes in Amazon-Photo. (c) Visualization of the graph features in IMDB-M.
visual representations. More recently, Choi et al. (2022) showed that restoring data corrupted with certain noise levels offers a proper pretext task for the model to learn rich visual concepts.
Despite the increasing research on diffusion models in computer vision, there is still a noticeable shortage of studies exploring the use of diffusion models in graph learning. Previous works, such as those by Haefeli et al. (2022) and Jo et al. (2022), have primarily concentrated on employing diffusion models for generating discrete graph structures. However, the realm of graph representation learning, a fundamental and challenging aspect of graph learning, has yet to harness the potential of diffusion models. To effectively adapt and incorporate diffusion models into graph representation learning, and to facilitate progress in this domain, it is imperative to identify and understand the obstacles hindering the application of diffusion models.
To gain insights into the limitations of the vanilla diffusion models which are commonly adopted for image generation tasks (Ho et al., 2020), we first investigate the underlying structural disparities between images and graphs. In particular, we employ singular value decomposition (SVD) on both image and graph data, and visualize the projected data in a 2-dimensional plane, as shown in Figure 1. This figure illustrates that the projected data from Amazon-Photo and IMDB-M exhibit strong anisotropic structures along only a few directions, while the projected images from CIFAR-10 form a relatively more isotropic distribution within a circular shape centered around the origin. This observation strongly implies that graph data may feature distinctive anisotropic and directional structures that are less prevalent in image data. As we will subsequently demonstrate, the vanilla diffusion models with isotropic forward diffusion processes result in a rapid decrease in the signal-to-noise ratios (SNRs), which in turn diminishes their effectiveness in learning anisotropic structures. Therefore, it is imperative to develop new approaches that can effectively account for these anisotropic structures.
This paper presents directional diffusion models as a solution for better learning anisotropic structures. Our approach involves incorporating data-dependent and directional noise into the forward diffusion process, effectively mitigating the challenge of rapid signal-to-noise ratio deterioration. The intermediate activations obtained from the denoising network excel at capturing valuable semantic and topological information crucial for downstream tasks. Consequently, the proposed directional diffusion models offer a promising approach for generative graph representation learning. Numerically, we perform experiments on 12 benchmark datasets covering both node and graph classification tasks. The results consistently highlight the superior performance of our models when compared to state-of-the-art contrastive learning and generative approaches (Hou et al., 2022). Notably, for graph classification problems, our directional diffusion models even surpass supervised baseline methods, underscoring the considerable potential of diffusion models in the field of graph representation learning.
1. We contribute to the exploration of anisotropic structures in graph data, being among the pioneers in the literature. We demonstrate that the vanilla forward diffusion process with isotropic white noise leads to a rapid decline in signal-to-noise ratios for graph learning problems. This issue hampers the ability of denoising networks to extract fine-grained feature representations across a wide range of SNRs.
2. We propose novel directional diffusion models specifically designed for graph data, incorporating data-dependent and directional noise in the forward diffusion process. Our proposed models effectively address the issue of the rapid decline of SNRs, enabling better graph representation learning.
3. Numerically, our proposed directional diffusion models outperform state-of-the-art self-supervised methods and even supervised methods on 12 benchmark datasets. Additionally, we provide comprehensive ablation studies to gain a deeper understanding of the mechanisms underlying directional diffusion models.
Graph representation learning Graph representation learning aims to embed nodes or entire graphs into a low-dimensional vector space, where the structural and relational properties can be used for downstream tasks. Two prevalent paradigms for graph representation learning are contrastive learning and generative self-supervised learning. Contrastive learning approaches such as DGI (Velickovic et al., 2019), Infograph (Sun et al., 2019), GraphCL (You et al., 2020), GRACE (Zhu et al., 2020), and GCC (Qiu et al., 2020), have achieved promising results in some particular graph learning tasks. These methods leverage local-global mutual information maximization for node and graph representation learning. GraphCL learns node embeddings that are invariant to graph-level transformations, while GRACE and GCC use subgraph sampling and graph perturbation to create augmented pairs. Generative self-supervised learning aims to recover masked components of the input data through approaches such as GraphMAE (Hou et al., 2022), a masked graph autoencoder that focuses on feature reconstruction by utilizing a masking strategy and scaled cosine error. This method outperforms self-supervised learning baselines, and revitalizes the concept of generative self-supervised learning on graphs. GPT-GNN (Hu et al., 2020b) is a recent approach that introduces a self-supervised attributed graph generation task to pre-train a GNN so that it can capture the structural and semantic properties of the graph.
Denoising diffusion probabilistic models Denoising diffusion probabilistic models (Ho et al., 2020; Song et al., 2020), or simply diffusion models, are a class of probabilistic generative models that turn noise to a data sample and thus are mainly used for generation tasks (Dhariwal and Nichol, 2021; Rombach et al., 2022). Recently, diffusion models have been used as a representation learning toolbox in computer vision (Preechakul et al., 2022; Abstreiter et al., 2021; Baranchuk et al., 2021). For instance, Preechakul et al. (2022) proposed Diff-AE, a method that concurrently trains an encoder to discover high-level semantics and a conditional diffusion model that uses these representations as input conditions. Abstreiter et al. (2021) augmented the denoising score matching framework to enable representation learning without any supervised signal. Recently, diffusion models have also found applications in handling graph data. Haefeli et al. (2022) showed that diffusion models for graphs benefited from discrete state spaces. Jo et al. (2022) proposed the graph diffusion using stochastic differential equations. To the best of our knowledge, there have been no works for diffusion-model-based graph representation learning.
3 The effect of anisotropic structures
As discussed in the introduction section, significant structural differences exist between graphs and natural images. In addition to the analysis of the Amazon-Photo and IMDB-M datasets, we conducted similar examinations on all other graph benchmark datasets, and the detailed results can be found in the appendix. Moreover, it is important to highlight that the anisotropic structures, referred to as categorical directional dependence, are frequently encountered in natural language data (Gao et al., 2019; Li et al., 2020). It is intriguing to observe that diffusion models have not yet made significant strides in the realm of natural language processing. This further underscores the significance of
(a) Amazon-Photo
(b)IMDB-M
Figure 2: The signal-to-noise ratio curve along different diffusion steps.
investigating and tackling the challenges posed by anisotropic structures within the framework of diffusion models.
This section delves deeper into the examination of how anisotropic structures in graphs impact the effectiveness of vanilla diffusion models. In the vanilla forward diffusion process, isotropic Gaussian noise is sequentially added to the raw data point  $x_0 \sim q(x_0)$  until it transforms into isotropic white noise  $\mathcal{N}(0,\mathbf{I})^2$ . This approach is reasonable when the data follow isotropic distributions, as it gradually transforms the data into noise, generating a sequence of noisy samples with a wide range of SNRs. However, in cases where the data exhibits anisotropy, the addition of isotropic noise can quickly contaminate the data structure, causing the SNRs to rapidly approach zero. Consequently, denoising networks struggle to extract fine-grained feature representations.
To explore the impact of introducing isotropic noise on the learning of anisotropic graphs, we conduct an experiment to measure the SNRs for both node and graph classification tasks at each forward step. We observe how these SNRs change throughout the forward diffusion process. Initially, we pre-train a graph neural network (GNN) denoted as  $\mathbf{E}$  to serve as a feature extractor, projecting the graph data into a linearly separable space. Subsequently, we optimize the weight vector  $w\in \mathbb{R}^d$  in this linear space using Fisher's linear discriminant analysis. The weight vector  $w$  is then employed to compute the SNR at each forward diffusion step, where  $\mathrm{SNR} = w^{\mathrm{T}}S_Bw / w^{\mathrm{T}}S_Ww$ . In this equation,  $S_{B}$  represents the scatter between class variability, and  $S_{W}$  represents the scatter within-class variability. This SNR quantifies the discriminative power of the learned representations at different stages of the diffusion process.
We conducted this experiment on all graph benchmark datasets to assess the impact of isotropic noise on learning anisotropic graphs. Here, we present the results for IMDB-M and Amazon-Photo in Figure 2a, while the additional results can be found in the appendix. In Figure 2a, we observe that for anisotropic graph data and isotropic noise, the SNR rapidly decreases to 0 at around 50 steps for Amazon-Photo and 400 steps for IMDB-M. Furthermore, the SNR remains close to 0 thereafter, indicating that the incremental isotropic white noise quickly obscures the underlying anisotropic structures or signals. Consequently, the denoising networks are unable to learn meaningful and discriminative feature representations that can be effectively utilized for downstream classification tasks. In contrast, when utilizing our directional diffusion models, which incorporate a data-dependent and directional forward diffusion process (to be introduced later), the SNR declines at a slower pace. This slower decline enables the extraction of fine-grained feature representations with varying SNRs, preserving the essential information of the anisotropic structures.
Overall, these studies underscore the significance of considering anisotropic data structures when designing forward diffusion processes and the corresponding diffusion models, especially in the context of graph data.
$^{2}$ Bold symbols are used for matrices, but not for vectors.
Forward Process with Directional Noise
Figure 3: Directional noise vs white noise. We sample directional and white noise<sup>1</sup>, and subsequently add the noise in a sequential manner. The upper panel displays the samples with directional noise at the diffusion steps  $t = 0, 100, 500, 800, 1000$ , while the lower panel exhibits the samples with white noise at these identical diffusion steps. The two distinct colors indicate two different classes.
Forward Process with White Noise
4 Directional diffusion models
In this section, we start by introducing the requisite notation. Subsequently, we propose the directional diffusion models (DDMs), with a specific focus on their application to graph representation learning. Furthermore, we discuss how to extract feature representations from DDMs, a pivotal aspect for downstream tasks.
Notation We denote a graph by  $G = (\mathbf{V}, \mathbf{A}, \mathbf{X})$ , where  $\mathbf{V}$  is the node set,  $N = |\mathbf{V}|$  is the node number,  $\mathbf{A} \in \mathbb{R}^{N \times N}$  is the adjacency matrix (binary or weighted), and  $\mathbf{X} = (x_1, x_2, \dots, x_N)^{\mathrm{T}} \in \mathbb{R}^{N \times d}$  is the node feature matrix. Our goal is to learn a network, denoted as  $f: \mathbb{R}^{N \times d} \times \mathbb{R}^{N \times N} \to \mathbb{R}^{N \times d_h}$ , to encode graph features into representations  $\mathbf{H} = (h_1, h_2, \dots, h_N)^{\mathrm{T}} \in \mathbb{R}^{N \times d_h}$ , where  $h_i \in \mathbb{R}^{d_h}$  is the representation for node  $i$ . Mathematically, we have  $\mathbf{H} = f(\mathbf{X}, \mathbf{A})$ . These representations can subsequently be used for downstream tasks, such as graph and node classification.
Directional diffusion models In the preceding section, our investigation uncovered a pivotal factor contributing to the overwhelming performance of vanilla diffusion models in graph learning: the swift deterioration of signal-to-noise ratios. To tackle this challenge, we introduce the directional noise in the forward diffusion process, which involves transforming the isotropic Gaussian noise into an anisotropic noise by incorporating two additional constraints. These two constraints play a vital role in enhancing the efficacy of vanilla diffusion models.
Let  $G_{t} = (\mathbf{A}, \mathbf{X}_{t})$  be the noisy graph at the  $t$ -th forward diffusion step, where  $\mathbf{X}_{t} = \{x_{t,1}, x_{t,2}, \dots, x_{t,N}\}$  represents the learned features at the  $t$ -th step. Specifically, the node feature  $x_{t,i} \in \mathcal{R}^{d}$  of node  $i$  at time  $t$  is obtained as follows:
$$
x _ {t, i} = \sqrt {\bar {\alpha} _ {t}} x _ {0, i} + \sqrt {1 - \bar {\alpha} _ {t}} \epsilon^ {\prime}, \tag {1}
$$
$$
\epsilon^ {\prime} = \operatorname {s g n} \left(x _ {0, i}\right) \odot | \bar {\epsilon} |, \tag {2}
$$
$$
\bar {\epsilon} = \mu + \sigma \odot \epsilon \quad \text {w h e r e} \epsilon \sim \mathcal {N} (0, \mathbf {I}), \tag {3}
$$
where  $x_{0,i}$  is the raw feature vector of node  $i$ ,  $\mu \in \mathbb{R}^d$  and  $\sigma \in \mathbb{R}^d$  consist of the means and standard deviations of the  $d$  features across  $N$  nodes, respectively. The symbol  $\odot$  denotes the Hadamard product. During the mini-batch training,  $\mu$  and  $\sigma$  are calculated using graphs within the batch. The parameter  $\bar{\alpha}_t \coloneqq \prod_{i=0}^{t} (1 - \beta_i) \in (0,1)$  represents the variance schedule (Ho et al., 2020) and is parameterized by a decreasing sequence  $\{\beta_{1:T} \in (0,1)\}$ .
The generation parameters are provided in the appendix.
In contrast to the conventional forward diffusion process, our directional diffusion models integrate two additional constraints, as denoted in equations (2) and (3). The second constraint, (3), transforms the data-independent Gaussian noise into an anisotropic and batch-dependent noise. In this constraint, each coordinate of the noise vector shares the same empirical mean and empirical standard deviation as the corresponding coordinate in the data within the same batch. This constraint confines the diffusion process to the local neighborhood of the batch, preventing excessive deviation from the batch and preserving local coherence.
The first constraint, (2), aligns the noise  $\epsilon^{\prime}$  with the feature  $x_{0,i}$  to ensure that they share the same coordinate signs. This guarantees that adding noise does not result in noisy features pointing in the opposite direction of  $x_{0,i}$ . By preserving the directionality of the original feature, this constraint plays a crucial role in maintaining the inherent data structure during the forward diffusion process. These two constraints work in tandem to ensure that the forward diffusion process respects the underlying data structure and mitigates the rapid degradation of signals. Consequently, the SNR decays slowly, enabling our directional diffusion models to effectively extract meaningful feature representations across various steps. This, in turn, enhances the utility of these representations in downstream tasks by providing reliability and informativeness.
To illustrate the impact of directional noise, we refer to the experiments conducted in Section 3. Our newly proposed "directional noise" ensures a smoother decline of the SNRs throughout the diffusion process, confirming our initial intuition. To further visualize the differences between using directional noise and isotropic noise in the forward diffusion process, we conducted simulations on two ellipses and sequentially added noise, as depicted in Figure 3. The figure clearly illustrates the distinct behaviors exhibited by the two types of noise. With directional noise, the samples maintain a clear decision boundary, indicating the preservation of discriminative structures during the diffusion process. Conversely, samples with white noise quickly blend into pure noise, leading to the loss of discriminative information. This visual comparison unequivocally underscores the superiority of directional noise in preserving the structural information of the data during the forward diffusion process.
Model architecture We follow the same training strategy as in the vanilla diffusion models, where we train a denoising network  $f_{\theta}$  in the reverse diffusion process. Since the posterior of the forward process with directional noise cannot be expressed in a closed form, we borrow the idea from Bansal et al. (2022); Li et al. (2022) and let the denoising model  $f_{\theta}$  directly predict  $\mathbf{X}_0$ . The loss function  $\mathcal{L}$  is defined as the expected value of the Euclidean distance between the predicted feature representation  $f_{\theta}(\mathbf{X}_t, \mathbf{A}, t)$  and the original feature representation  $\mathbf{X}_0$ :
$$
\mathcal {L} = \mathbb {E} _ {\mathbf {X} _ {0}, t} \| f _ {\theta} \left(\mathbf {X} _ {t}, \mathbf {A}, t\right) - \mathbf {X} _ {0} \| ^ {2}. \tag {4}
$$
This loss function ensures that the model predicts  $\mathbf{X}_0$  at every step.
To parameterize the denoising network  $f_{\theta}$ , we employ a symmetrical architecture inspired by the successful UNet architecture in computer vision (Dhariwal and Nichol, 2021). Figure 4 illustrates our DDM framework, comprising four GNN layers and one multilayer perception (MLP). The first two GNN layers function as the encoder, responsible for denoising the target node by aggregating neighboring information. The final two GNN layers serve as the decoder, mapping the denoised node features to a latent code and smoothing the latent code among neighboring nodes. To address the potential issue of over-smoothing and account for long-distance dependencies in the graph, we introduce skip-connections between the encoder and decoder. The algorithm, along with the mini-batch training procedure, is presented in the appendix.
Learning representations For a given graph  $G = (\mathbf{A}, \mathbf{X})$ , the learned node-level representations are obtained from the activations of the denoising network  $f_{\theta}$  at user-selected time steps. It is important to note that we only utilize the activations from the decoder of  $f_{\theta}$  since they incorporate the encoder activations through skip connections. As depicted in Figure 4, at each time step  $k$ , we introduce  $k$  steps of directional noise following (1) and employ the denoising network  $f_{\theta}$  to denoise and encode the noisy data  $\mathbf{X}_k$ . The decoder of  $f_{\theta}$  maps the denoised node features to a latent code while smoothing the latent code among neighboring nodes. We extract the activations from the decoder of  $f_{\theta}$  and concatenate them to obtain  $\mathbf{H}_k = \{h_{k,1}, h_{k,2}, \dots, h_{k,N}\} \in \mathbb{R}^{N \times d_h}$ . The complete pipeline is presented in the appendix.
Figure 4: The pipeline of our model: (1) Adding directional noise to the original graph  $G_0$ . (2) Extracting the last two GNN's feature maps in the denoising network as the representation of the graph.
This section provides an evaluation of the directional diffusion models from two perspectives. First, we compare our models with existing state-of-the-art methods on various graph learning tasks, including node and graph classification tasks. This allows us to assess the effectiveness of our approach for graph representation learning problems. Second, we conduct several studies to gain a better understanding of the effect of our directional noise and evaluate the necessity of our design choices.
In all experiments, we follow a two-step process. First, we pre-train a DDM on the dataset in an unsupervised manner. Then, we extract feature representations from diffusion steps 50, 100, 200 using the pre-trained model. Although this approach is inspired by the experimental results and insights from Section 3, it is deliberately not fine-tuned for each dataset. Ideally, fine-tuning with carefully selected steps for each dataset could further improve the performance.
5.1 Graph classification