Cross-Domain Graph Data Scaling: A Showcase with Diffusion Models

Authors: Wenzhuo Tang, Haitao Mao, Danial Dervovic, Ivan Brugere, Saumitra Mishra, Yuying Xie, Jiliang Tang
Primary area: learning on graphs and other geometries & topologies
Keywords: Graph Foundation Model, Graph Data Augmentation, Diffusion Models

Abstract:
Models for natural language and images benefit from data scaling behavior: the more data fed into the model, the better they perform. This 'better with more' phenomenon enables the effectiveness of large-scale pre-training on vast amounts of data. However, current graph pre-training methods struggle to scale up data due to heterogeneity across graphs. To achieve effective data scaling, we aim to develop a general model that is able to capture diverse data patterns of graphs and can be utilized to adaptively help the downstream tasks. To this end, we propose UniAug, a universal graph structure augmentor built on a diffusion model. We first pre-train a discrete diffusion model on thousands of graphs across domains to learn the graph structural patterns. In the downstream phase, we provide adaptive enhancement by conducting graph structure augmentation with the help of the pre-trained diffusion model via guided generation. By leveraging the pre-trained diffusion model for structure augmentation, we consistently achieve performance improvements across various downstream tasks in a plug-and-play manner. To the best of our knowledge, this study represents the first demonstration of a data-scaling graph structure augmentor on graphs across domains.

Decision: reject
Reviewer scores: 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730063119274

thread_type:
official_review

<thread_object>
creator: reviewer_y1x3

<content>
summary:
The paper proposes a model called UniAug, based on a graph diffusion model. The approach consists of two steps: first, pretraining on diverse graphs from different domains, and second, using the model to generate synthetic structures for downstream tasks. Experimental results on approximately 30 cross-domain graphs demonstrate the empirical effectiveness of the method.

strengths:
1. The paper is well-written and easy to follow, presenting a simple yet effective approach that uses diffusion models to augment structural data for cross-domain graphs.

2. Addressing the challenge of designing models for cross-graph learning is highly valuable and relevant.

3. The experiments are comprehensive, covering approximately 30 graphs across more than five domains.

weaknesses:
1. The authors claim to propose a graph foundation model for cross-domain graphs. However, the method appears to be more of a universal graph structural augmentor that utilizes graph diffusion models for generating additional structures, rather than a foundation model capable of inference across various graph types, like [1,2,3]. Additionally, although the authors demonstrate that pretrained diffusion models can generate synthetic structures to enhance performance on various downstream tasks, the concept of using graph diffusion models to create structures is not entirely new [4,5], which may limit the novelty of the approach.

2. The experimental results are not entirely convincing. The authors use node degrees as features for self-supervised baselines, which can significantly impair model performance when replacing original node features with node degrees, whereas using original node features for the proposed method. Although the authors elaborate on this issue in Appendix D, the experimental descriptions still confuse me, and I cannot find strong evidence that the proposed UniAug outperforms existing SSL methods under comparable settings (i.e., using the same feature setup).

3. The applicability of the method may be limited in certain scenarios. When applying it to downstream tasks, such as node and graph classification, the authors use node labels to guide the generation of graph structures. However, the method may struggle with graphs that lack sufficient label information. It would be helpful to understand whether the approach can be effectively applied to downstream graphs with limited or no label data.

References:

[1] One for all: Towards training one graph model for all classification tasks. ICLR, 2024.

[2] GraphAny: A Foundation Model for Node Classification on Any Graph. Arxiv, 2024.

[3] Zerog: Investigating cross-dataset zero-shot transferability in graphs. KDD, 2024.

[4] SimDiff: Simple Denoising Probabilistic Latent Diffusion Model for Data Augmentation on Multi-modal Knowledge Graph. KDD, 2024.

[5] Data-Centric Learning from Unlabeled Graphs with Diffusion Model. NeurIPS, 2023.

questions:
1. The authors present experimental results with varying pretraining scales in Figure 3. Although there is a general performance improvement between the SMALL, FULL, and EXTRA scales, there are instances where the model pretrained on smaller graph datasets outperforms those pretrained on larger datasets, such as on the Enzymes and Erdos datasets. Could the authors provide a detailed explanation for these observations?

2. Can the proposed method be applied to more practical scenarios, such as few-shot or zero-shot learning?

3. The proposed method leverages a diffusion model to generate additional structural information. While the model demonstrates empirically desirable performance, I am curious whether there is a theoretical understanding that supports its efficacy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730168945106

thread_type:
official_review

<thread_object>
creator: reviewer_sw2z

<content>
summary:
This paper explores a meaningful issue: How to effectively leverage the increasing scale of data across domains for graph learning? To achieve effective data scaling, this paper proposes a universal graph augmentation framework, UniAug. This framework pre-trains a discrete diffusion model on massive graphs across domains to learn the structural patterns, and conducts structure augmentation with the help of the pre-trained diffusion model. Experiments shows that this framework can leverage the data scaling laws and achieve performance improvements across various downstream tasks.

strengths:
+ The research topic of this paper—How to effectively leverage the increasing scale of data across domains for graph learning, is highly significant and meaningful, representing a critical issue that urgently needs to be addressed in the current field.
+ This paper collects thousands of graphs from varied domains with diverse patterns to explore the potential of data scaling for graph learning.
+ This paper provides a solution based on the diffusion model from a structural augmentation perspective. This augmentation paradigm strategically circumvents feature heterogeneity and fully utilizes downstream inductive biases in a plug-and-play manner.

weaknesses:
+ The authors should clearly illustrate the scale and domain variety of the pre-training data collection in the paper and make it public for the community to conduct further research on the issue. Meanwhile, providing code examples is also essential.
+ The authors should consider whether excessively introducing additional datasets for pre-training is appropriate and whether it might produce negative effects.
+ Related work about data scaling on graphs should be include in the paper [1, 2, 3].
1. [1] Liu J, Mao H, Chen Z, et al. Neural scaling laws on graphs[J]. arXiv preprint arXiv:2402.02054, 2024.
2. [2] Ma Q, Mao H, Liu J, et al. Do Neural Scaling Laws Exist on Graph Self-Supervised Learning?[J]. arXiv preprint arXiv:2408.11243, 2024.
3. [3] Wang Z, Li Y, Ding B, et al. Exploring Neural Scaling Law and Data Pruning Methods For Node Classification on Large-scale Graphs[C]//Proceedings of the ACM on Web Conference 2024. 2024: 780-791.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706283757

thread_type:
official_review

<thread_object>
creator: reviewer_9f3c

<content>
summary:
The paper proposes a graph structure augmentation framework built on discrete diffusion models. They first pre-train a structure-only discrete diffusion model on graphs from multiple domains. During downstream adaptation, the model generates synthetic graph through guided generation, using a task-specific head to align these generated structures with the target task. This framework aims to enhance performance across various tasks, including node classification, link prediction and graph prediction. The experiment also shows that the downstream performance increases when the amount of pre-training data increases.

strengths:
- The paper demonstrates an interesting cross-domain transfer ability, where the pre-trained model can be adapted to various downstream tasks at different levels (node, edge, and graph).
- The presentation is clear, with well-explained methods and experimental results, making it easy to follow the proposed approach and findings.

weaknesses:
- I have some concerns about the comparison of UniAug with baseline pre-training methods. The authors mention that they used the same pre-training dataset for UniAug and baselines. They also calculated node degrees as inputs and replaced node features with node degrees for downstream testing. However, these baseline models were not originally designed to be trained and tested in this way. As shown in Table 13, when evaluated in their original semi-supervised or self-supervised settings, these baselines achieve results similar to UniAug. This is different from the larger improvement suggested in Table 2. A more accurate comparison would keep the baselines in their original configurations, and it would also help to see similar comparisons for link prediction and node classification tasks.
- Additionally, a discussion of pre-training time for UniAug compared to the baselines would improve the analysis. This can offer a better understanding of UniAug’s efficiency compared to other methods.
- A more detailed description of the sizes of Small, Full, and Extra pre-training datasets should be provided. From the description, the Extra dataset includes 1,000 more subgraphs from the GitHub Star dataset than the Full dataset. This addition results in large improvements for link prediction tasks but shows little effect on graph classification tasks. Thus I am curious whether the improvements come from the increased amount of data or from the diversity added by the new subgraphs. A more detailed breakdown of the dataset sizes and an analysis of data diversity versus quantity would help clarify the reasons behind these improvements.

questions:
Please see the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731952084110

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to the reviewers

comment:
We thank the reviewers for their valuable suggestions and concerns. Below, we summarize key experimental results added during the rebuttal period:

1. **Comparison with self-supervised baselines on link prediction and node classification**: To provide a more comprehensive evaluation, we have included both semi-supervised and self-supervised results for graph classification in Appendix D. Additionally, we present self-supervised results for link prediction and node classification in the following tables. The best results are highlighted in bold, with the last column showing the average ranking. Our findings demonstrate that UniAug consistently outperforms the baselines in terms of average ranking, particularly on heterophilic node classification datasets.

|  |  | Cora | Citeseer | Pubmed | Power | Yeast | Erdos | Flickr |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | MRR | MRR | MRR | Hits@10 | Hits@10 | Hits@10 | Hits@10 | A.R. |
| Self-supervised | MVGRL[1] | 29.13±3.90 | 51.32±4.12 | 15.21±2.35 | 31.71±3.78 | 23.74±5.74 | 36.21±2.81 | 8.42±2.18 | 4.29 |
|  | GRACE[2] | 31.77±4.31 | 49.13±3.95 | 16.88±1.74 | 28.21±5.04 | 23.96±4.31 | 33.90±2.12 | **9.87±0.98** | 4.00 |
|  | BGRL[3] | 33.59±2.14 | 51.91±5.01 | 16.93±2.03 | 33.71±3.21 | 25.91±3.12 | 37.95±1.73 | 8.52±1.85 | 2.57 |
|  | GraphMAE[4] | 32.98±5.01 | 52.71±5.39 | **18.83±1.30** | 32.81±2.12 | 26.51±2.92 | 35.63±3.61 | 7.01±3.86 | 2.86 |
| GDA | UniAug | **35.36±7.88** | **54.66±4.55** | 17.28±1.89 | **34.36±1.68** | **27.52±4.80** | **39.67±4.51** | 9.46±1.18 | 1.29 |

| ACC ↑ |  | Cornell | Wisconsin | Texas | Actor | Chameleon* | Squirrel* | A.R. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Self-supervised | MVGRL[1] | 56.19±2.42 | 50.64±5.89 | 61.70±3.94 | 31.37±0.83 | 32.34±2.11 | 35.32±1.32 | 4.00 |
|  | GRACE[2] | 56.39±2.11 | 53.83±3.56 | 63.54±2.57 | 28.14±0.81 | 35.71±1.95 | 33.65±2.51 | 4.17 |
|  | BGRL[3] | 56.67±2.13 | 59.80±4.08 | 65.78±2.66 | 29.80±0.31 | 37.01±2.89 | 34.77±2.01 | 2.67 |
|  | GraphMAE[4] | 57.31±2.11 | 58.27±2.91 | 58.34±3.57 | 28.97±0.27 | 36.75±1.78 | 39.13±2.01 | 3.17 |
| GDA | UniAug | **68.11±6.72** | **69.02±4.96** | **73.51±5.06** | **33.11±1.57** | **43.84±3.39** | **41.90±1.90** | 1.00 |

2. **Data scaling with fixed distribution coverage**. In the manuscript, we conducted a data scaling experiment using three pre-training datasets: SMALL, FULL, and EXTRA. We acknowledge that these datasets differ in both scale and diversity. To isolate the effect of data quantity, we clustered the pre-training set into 10 clusters based on graph-level representations (Section 3.2) and performed stratified sampling within these clusters. Using this approach, we created three subsets containing 25%, 50%, and 75% of the total graphs and pre-trained UniAug on each subset. The results, summarized in the following figures with links for [graph classification](https://drive.google.com/file/d/15wiJj9awBqxYbvJirY99O3RLwZsUR8pv/view?usp=sharing) and [link prediction](https://drive.google.com/file/d/19WicL43j9YEwWHbXZsBI9b93OjOTznJl/view?usp=sharing), reveal a clear trend of performance improvement as the size of the pre-training set increases. Combined with the experiments on the SMALL, FULL, and EXTRA sets, these findings highlight that UniAug benefits from both increasing the scale and enhancing the diversity of the pre-training data.

3. **Few-shot graph classification**: To demonstrate the effectiveness of UniAug in scenarios with limited labeled data, we include results for 5-shot graph classification following [5]. These results show that UniAug achieves significant performance improvements over the self-supervised baselines, underscoring its robustness and adaptability in few-shot settings.

| 5-shot ACC ↑ | Proteins | Enzymes |
| --- | --- | --- |
| GIN | 58.17±8.58 | 20.34±5.01 |
| InfoGraph | 54.12±8.20 | 20.90±3.32 |
| GraphCL | 56.38±7.24 | 28.11±4.00 |
| JOAO | 57.21±6.91 | 35.31±3.79 |
| GraphMAE | 58.03±5.35 | 33.91±6.58 |
| UniAug | **66.85±4.71** | **48.37±4.77** |

[1] Hassani, Kaveh, and Amir Hosein Khasahmadi. "Contrastive multi-view representation learning on graphs." ICML, 2020.

[2] Zhu, Yanqiao, et al. "Deep graph contrastive representation learning." arXiv, 2020.

[3] Thakoor, Shantanu, et al. "Large-Scale Representation Learning on Graphs via Bootstrapping." ICLR, 2022

[4] Hou, Zhenyu, et al. "Graphmae: Self-supervised masked graph autoencoders." KDD, 2022.

[5] Liu, Zemin, et al. "Graphprompt: Unifying pre-training and downstream tasks for graph neural networks." WWW, 2023.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The UniAug framework proposed in this paper shows some potential in cross-domain graph learning tasks, but several key issues remain. First, there is inconsistency in the comparison with baseline methods, particularly regarding the differences in feature usage, which affects the fairness and comparability of the results. Second, although the proposed method demonstrates some performance improvements in the experiments, its novelty is questioned, as similar data augmentation methods based on diffusion models have already been explored in previous research. While the authors have provided responses, they have not adequately addressed the related concerns. Therefore, I recommend rejecting the submission.