SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To demonstrate the effectiveness of our method, we conduct comparisons with state-of-the-art (SOTA) unsupervised learning methods, which include GCC (Qiu et al., 2020), Infograph (Sun et al., 2019), GraphCL (You et al., 2020), JOAO (You et al., 2021), MVGRL (Hassani and Khasahmadi, 2020), and GraphMAE (Hou et al., 2022). We also compare our approach with supervised learning methods, specifically GIN (Xu et al., 2018) and DiffPool (Ying et al., 2018). These experiments are conducted on seven widely-used datasets, namely MUTAG, IMDB-B, IMDB-M, PROTEINS, COLLAB, and REDDIT-B (Yanardag and Vishwanathan, 2015). Node degrees are used as initial node features for IMDB-B, IMDB-M, REDDIT-B, and COLLAB, while node labels are employed for MUTAG and PROTEINS, in accordance with prior literature (Hou et al., 2022). We extract graph-level representations at various steps, train linear SVMs using LIBSVM (Chang and Lin, 2011), and independently test them. The final predictions are obtained by majority vote, and we report the average accuracy and standard deviation, calculated after five runs. Additional details on hyper-parameters can be found in the appendix.
Table 1 presents the results, clearly indicating that our DDM achieves the most competitive, if not the best, performance across all benchmark datasets. Notably, our DDM outperforms even the supervised approaches in certain experiments, such as IMDB-B, COLLAB, and MUTAG. This remarkable performance can be attributed to two key factors. First, from a data perspective, these datasets have limited information in their node features, which can hinder the accuracy of supervised learning (Hou et al., 2022). By leveraging directional noise diffusion, our DDM effectively acts as a pseudo-infinite-step data augmentation technique, generating numerous samples while preserving the classification boundary. This augmentation significantly enhances the effectiveness of unsupervised learning. Second, from a model perspective, the DDM framework harnesses the power of directional
Table 1: Results in unsupervised representation learning for graph classification.
Dataset	IMDB-B	IMDB-M	COLLAB	REDDIT-B	PROTEINS	MUTAG
GIN	75.1±5.1	52.3±2.8	80.2±1.9	92.4±2.5	76.2±2.8	89.4±5.6
DiffPool	72.6±3.9	-	78.9±2.3	92.1±2.6	75.1±2.3	85.0±10.3
Infograph	73.03±0.87	49.69±0.53	70.65±1.13	82.50±1.42	74.44±0.31	89.01±1.13
GraphCL	71.14±0.44	48.58±0.67	71.36±1.15	89.53±0.84	74.39±0.45	86.80±1.34
JOAO	70.21±3.08	49.20±0.77	69.50±0.36	85.29±1.35	74.55±0.41	87.35±1.02
GCC	72	49.4	78.9	89.8	-	-
MVGRL	74.20±0.70	51.20±0.50	-	84.50±0.60	-	89.70±1.10
GraphMAE	75.52±0.66	51.63±0.52	80.32±0.46	88.01±0.19	75.30±0.39	88.19±1.26
DDM	76.40±0.22	52.53±0.31	81.72±0.31	89.15 ±1.3	75.47 ±0.50	91.51 ±1.45
Table 2: Results in unsupervised representation learning for node classification.
Dataset	Cora	Citeseer	PubMed	Ogbn-arxiv	Computer	Photo
GAT	83.0 ± 0.7	72.5 ± 0.7	79.0 ± 0.3	72.10 ± 0.13	86.93 ± 0.29	92.56 ± 0.35
DGI	82.3 ± 0.6	71.8 ± 0.7	76.8 ± 0.6	70.34 ± 0.16	83.95 ± 0.47	91.61 ± 0.22
MVGLR	83.5 ± 0.4	73.3 ± 0.5	80.1 ± 0.7	-	87.52 ± 0.11	91.74 ± 0.07
BGRL	82.7 ± 0.6	71.1 ± 0.8	79.6 ± 0.5	71.64 ± 0.12	89.68 ± 0.31	92.87 ± 0.27
InfoGCL	83.5 ± 0.3	73.5 ± 0.4	79.1 ± 0.2	-	-	-
CCA-SSG	84.0 ± 0.4	73.1 ± 0.3	81.0 ± 0.4	71.24 ± 0.20	88.74 ± 0.28	93.14 ± 0.14
GPT-GNN	80.1 ± 1.0	68.4 ± 1.6	76.3 ± 0.8	-	-	-
GraphMAE	84.2 ± 0.4	73.4 ± 0.4	81.1 ± 0.4	71.75 ± 0.17	88.63 ± 0.17	93.63 ± 0.22
DDM	83.4 ± 0.2	74.3 ± 0.3	81.7 ± 0.8	71.29 ± 0.18	90.56 ± 0.21	95.09 ± 0.18
noise and ensures that the learned representations capture discriminative information by preventing the rapid decay of signal-to-noise ratios.
5.2 Node classification
To assess the quality of the node-level representations produced by our method, we conduct evaluations of DDM on six standard benchmark datasets: Cora, Citeseer, PubMed (Yang et al., 2016), Ogbn-arxiv (Hu et al., 2020a), Amazon-Computer (Zhang et al., 2021), and Amazon-Photo (Zhang et al., 2021). We follow the publicly available data-split schema and employ the evaluation protocol used in the literature. Graph-level representations are extracted at different diffusion steps, and an independent linear classifier is trained for each step. The final predictions are determined through majority voting, and we report the mean accuracy on the test nodes. Additional details regarding hyperparameters can be found in the appendix.
We compare DDM with state-of-the-art generative unsupervised models, specifically GPT-GNN (Hu et al., 2020b) and GraphMAE (Hou et al., 2022). Furthermore, we include the results of contrastive unsupervised models for comparison, which include DGI (Velickovic et al., 2019), MVGRL (Hassani and Khasahmadi, 2020), GRACE (Zhu et al., 2020), BGRL (Thakoor et al., 2021), InfoGCL (Xu et al., 2021), and CCA-SSG (Zhang et al., 2021). As shown in Table 2, DDM achieves competitive results across all benchmark datasets. This underscores the capacity of the generative diffusion method to learn meaningful node-level representations and highlights the effectiveness of DDM in node-level tasks. Notably, the node features used in node classification are text embeddings, showcasing the efficacy of our directional noise in continuous word vector spaces.
5.3 Understanding the directional noise
The aforementioned studies provide evidence that our approach either outperforms or is on par with existing state-of-the-art (SOTA) methods. To gain a deeper understanding, we conduct a comprehensive investigation into the impact of different types of noise. Additionally, we analyze the effect of directional noise by removing the two constraints specified in equations (3) and (2).
We examine the representations extracted from models trained with directional noise and isotropic white noise at each reverse step. The results, as depicted in Figure 5, reveal significant differences between the two approaches. With white noise, only the representations corresponding to the early steps of the reverse process contained useful discriminative information, while the representations for later steps are mostly uninformative. This is in stark contrast to the case of directional noise, where the learned representations consistently preserve sufficient information for downstream classification tasks.
In our additional experiments, we consistently observe that directional diffusion models outperform vanilla diffusion models across all datasets, particularly in node classification tasks. This superior performance can be attributed to the nature of node classification datasets, which often use word vectors as node features. These word vectors have higher feature dimensionality and greater anisotropy. The effectiveness of our directional approach is further supported by these findings, reinforcing its value in graph representation learning.
To examine the impact of different schedulers, we compare the performance of the vanilla diffusion model using isotropic white noise and DDM under the cosine and sigmoid scheduler in Chen (2023). Table 3 collects the results. Different schedulers indeed influence the model performance. However, the performance of the vanilla diffusion model with isotropic white noise heavily relies on the hyperparameters of the scheduler. Yet, irrespective of the scheduler employed, our proposed data-dependent anisotropic noise consistently yields superior performance.
Lastly, we conduct an ablation study to examine the effects of the two constraints. Table 4 presents results, where "w/o R" indicates the removal of the constraint (2) and "w/o S&R" indicates the removal of both constraints. As shown in Table 4, the introduction of anisotropic Gaussian noise generated through (3) led to a significant improvement compared to isotropic Gaussian noise. Furthermore, the inclusion of constraint (2) provided an additional and indispensable improvement. This finding further confirms the importance of making the noise in the forward process data-dependent and anisotropic.
Table 3: Results for different schedulers.
Noise schedule function	Noise type	CiteSeer	PubMed	MUTAG
cosine (s=0, e=1, τ = 1)	DDM	0.715	0.824	0.867
cosine (s=0, e=1, τ = 1)	White Noise	0.371	0.453	0.692
sigmoid (s=0, e=3, τ = 1)	DDM	0.710	0.806	0.877
sigmoid (s=0, e=3, τ = 1)	White Noise	0.581	0.434	0.691
Table 4: An ablation study on the two constraints.
Dataset	w/o S&R	w/o R	Full
CiteSeer	34.37±0.5	60.77±0.2	74.3 ± 0.3
PubMed	73.07±0.7	77.60±0.4	81.7 ± 0.8
IMDB-M	49.80±0.53	50.87±0.49	52.53±0.31
COLLAB	80.50±0.36	81.04±0.17	81.72±0.31
MUTAG	82.89±1.16	87.25±1.12	91.51±1.45
This paper unveils the presence of anisotropic structures in graphs, which pose challenges for vanilla diffusion models in graph representation learning. To address this limitation, we introduce directional diffusion models, a novel class of diffusion models that leverage data-dependent and anisotropic noise to better handle anisotropic structures. Through experiments conducted on 12 benchmark datasets, we demonstrate the effectiveness of our proposed method.
There are several promising avenues for future research. One direction is to develop methods that can automatically determine the optimal set of diffusion steps for each dataset, further enhancing the performance of our directional diffusion models. This could involve techniques such as adaptive selection of diffusion steps. Additionally, exploring the application of our method to computer vision and natural language processing tasks holds great potential for advancing these domains. By adapting and extending our directional diffusion models to these areas, we may leverage their inherent strengths to improve representations and enable effective learning tasks such as image recognition, object detection, sentiment analysis, and language understanding.
(a) Citeseer
(c) PubMed
Figure 5: Comparing the accuracy in downstream tasks when using representations extracted from models trained with directional noise and white noise at every step of the reverse process.
(b) MUTAG
(d)IMDB-M
FZ is supported by National Natural Science Foundation of China (12001356), Shanghai Education Development Foundation and Shanghai Municipal Education Commission (Chenguang Program), Zhejiang Lab (Open Research Project NO. 2022RC0AB06), Shanghai Research Center for Data Science and Decision Technology. Fz is Supported by the Fundamental Research Funds for the Central Universities: "High-Quality Development of Digital Economy: An Investigation of Characteristics and Driving Strategies (Grant Numbers 2023110139). QS is supported in part by a NSERC Discovery Grant, a New Frontiers in Research Fund, and a Data Sciences Institute Catalyst Grant.
Abstreiter, K., Mittal, S., Bauer, S., Scholkopf, B., and Mehrjou, A. (2021). Diffusion-based representation learning. arXiv preprint arXiv:2105.14257.
Bansal, A., Borgnia, E., Chu, H.-M., Li, J. S., Kazemi, H., Huang, F., Goldblum, M., Geiping, J., and Goldstein, T. (2022). Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392.
Baranchuk, D., Rubachev, I., Voynov, A., Khrulkov, V., and Babenko, A. (2021). Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126.
Chang, C.-C. and Lin, C.-J. (2011). Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):1-27.
Chen, T. (2023). On the importance of noise scheduling for diffusion models. arXiv preprint arXiv:2301.10972.
Choi, J., Lee, J., Shin, C., Kim, S., Kim, H., and Yoon, S. (2022). Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472-11481.
Dhariwal, P. and Nichol, A. (2021). Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794.
Gao, J., He, D., Tan, X., Qin, T., Wang, L., and Liu, T.-Y. (2019). Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009.
Yang, Z., Cohen, W., and Salakhudinov, R. (2016). Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pages 40-48. PMLR.
Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., and Leskovec, J. (2018). Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31.
You, Y., Chen, T., Shen, Y., and Wang, Z. (2021). Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121-12132. PMLR.
You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., and Shen, Y. (2020). Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812-5823.
Zhang, H., Wu, Q., Yan, J., Wipf, D., and Yu, P. S. (2021). From canonical correlation analysis to self-supervised graph neural networks. Advances in Neural Information Processing Systems, 34:76-89.
Zhang, Z., Zhao, Z., and Lin, Z. (2022). Unsupervised representation learning from pre-trained diffusion probabilistic models. arXiv preprint arXiv:2212.12990.
Zhu, Y., Xu, Y., Yu, F., Liu, Q., Wu, S., and Wang, L. (2020). Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131.