A Large-scale Training Paradigm for Graph Generative Models

Authors: Yu Wang, Ryan A. Rossi, Namyong Park, Huiyuan Chen, Nesreen K. Ahmed, Puja Trivedi, Franck Dernoncourt, Danai Koutra, Tyler Derr
Primary area: learning on graphs and other geometries & topologies
Keywords: Large-scale Training, Graph Generative Model, Diffusion model

Abstract:
Large Generative Models (LGMs) such as GPT, Stable Diffusion, Sora, and Suno are trained on a huge amount of texts, images, videos, and audio that are extremely diverse from numerous domains. This large-scale training paradigm on diverse well-curated data enhances the creativity and diversity of the generated content. However, all previous graph-generative models (e.g., GraphRNN, MDVAE, MoFlow, GDSS, and DiGress) have been trained only on one dataset each time, which cannot replicate the revolutionary success achieved by LGMs in other fields. To remedy this crucial gap, we propose a large-scale training paradigm that uses a large corpus of graphs (over 5000 graphs) from 13 domains, leading to the development of large graph generative models (LGGMs). We empirically demonstrate that the pre-trained LGGMs have superior zero-shot generative capability to existing graph generative models. Furthermore, our pre-trained LGGMs can be easily fine-tuned with graphs from target domains and demonstrate even better performance than those directly trained from scratch, behaving as a solid starting point for real-world customization. Inspired by Stable Diffusion, we further equip LGGMs with the Text-to-Graph generation capability, such as providing the description of the network name and domain (i.e., "The power-1138-bus graph represents a network of buses in a power distribution system.") and network statistics (i.e., "The graph has a low average degree, suitable for modeling social media interactions."). This Text-to-Graph capability integrates the extensive world knowledge in the underlying language model, offering users fine-grained control of the generated graphs. We release the code, the model checkpoint, and the datasets at https://github.com/KINDLab-Fly/LGGM.

Decision: accept (poster)
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730470562271

thread_type:
official_review

<thread_object>
creator: reviewer_vfbb

<content>
summary:
The paper presents a novel approach to large-scale graph generation with the introduction of a Large Graph Generative Model (LGGM). The authors design an innovative training paradigm for graph generative models, training over 5,000 graphs sourced from 13 distinct domains. Remarkably, this work is the first to explore a large-scale training paradigm specifically tailored to graph-structured data generation, marking an exciting achievement for the graph modality. By addressing the complexities of large-scale graph generation, this research expands the capabilities of generative modeling for graph-based data, offering substantial contributions to the field.

strengths:
1.  The paper addresses a novel problem by extending the scope of large-scale generative models to the graph domain. This exploration into graph generative modeling at scale is a significant contribution, introducing fresh perspectives and potential for advancements in the field.
2. The implementation is both innovative and thorough, covering several essential aspects, including Pre-training Evaluation, Fine-tuning Evaluation, Graph Classification, and Text-to-Graph Generation. These multiple dimensions demonstrate a robust approach.
3. The comprehensive dataset scale strengthens the results, making the findings more generalizable across graph-structured data.

weaknesses:
1.  The approach relies primarily on existing discrete denoising diffusion techniques, which, while effective, lacks methodological innovation specific to large-scale graph generation. This reliance on existing methods may limit the theoretical foundation and originality of the model within the context of large-scale graph generation.
2. The paper includes a comparison with only a single baseline, which is insufficient to thoroughly evaluate the model’s performance.

questions:
1. Are there any experiments or analyses conducted to evaluate the model's performance in relation to scaling law expectations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730578246475

thread_type:
official_review

<thread_object>
creator: reviewer_62fq

<content>
summary:
This paper introduces a large-scale training paradigm for graph generative models (LGGMs), pre-trained on over 5000 graphs from 13 domains. The work aims to mirror the success of large generative models like GPT and Stable Diffusion in the graph domain, offering both zero-shot generation capabilities and domain-specific fine-tuning. The authors propose a diffusion-based architecture that incorporates text-to-graph generation through language model integration and demonstrates empirical improvements across multiple domains. While the approach shows promise in expanding graph generation to multiple domains, the paper suffers from overclaiming and limited technical innovation in its core methodology.

strengths:
1. The paper is generally easy to follow and the motivations are clear.
2. The integration of text-to-graph generation capabilities represents a new direction for controlled graph generation.
3. The paper provides an interesting roadmap for scaling up graph generation models with practical applications.

weaknesses:
1. **Fundamentally Weak Theoretical Foundation**:
   * The theoretical analysis is superficial and merely combines existing results without meaningful insights. While Section 3.3 claims novel expressiveness results, the proofs follow directly from prior work [1,2] and ignore fundamental limitations established in recent literature [3]. The paper fails to provide any theoretical justification for why cross-domain pre-training should help, or how the sampling strategy preserves important graph properties. Most importantly, there's no analysis of how the multi-domain training affects model capacity and expressiveness.

2. **Limited Technical Innovation and Poor Design Choices**:
   * The core methodology is essentially DiGress [1] with small modifications. The sampling strategy for handling large graphs is ad-hoc and lacks theoretical guarantees, while all four pre-training tasks are directly borrowed from existing work without significant adaptation to graph-specific challenges. The architecture choices appear arbitrary rather than fundamental, and the text-to-graph component is a straightforward application similar to existing approaches TAPE[4] and GraphGPT[5] without addressing the unique challenges of graph generation. The text-to-graph mechanism is particularly concerning - it simply concatenates LLM embeddings with graph features without addressing the fundamental challenges of aligning textual and structural representations. This naive approach ignores significant prior work on text-attributed graphs, such as [4,5], which have developed sophisticated techniques for text-graph alignment. 

3. **Empirical and Evaluation Limitations**:
   * The experimental evaluation has several critical flaws. The paper primarily compares against a single baseline (DiGress) while ignoring crucial comparisons with recent graph generative model works like GCC [6] and GraphMAE [7]. The evaluation metrics don't adequately capture global structural properties, and there's no analysis of computational efficiency or scaling behavior. The ablation studies are insufficient to justify architectural choices, and more importantly, I would not really call 5000 graphs a large graph dataset.

[1] Vignac et al., "Digress: Discrete Denoising Diffusion for Graph Generation", ICLR 2023

[2] Chen et al., "Efficient and Degree-guided Graph Generation", 2023

[3] Liu et al., "On the Expressivity of Graph Neural Networks", ICLR 2024

[4] He et al., "Harnessing explanations: Llm-to-lm interpreter for enhanced text-attributed graph representation learning." ICLR 2024.

[5] Tang, Jiabin, et al. "Graphgpt: Graph instruction tuning for large language models." SIGIR 2024.

[6] Qiu et al., "GCC: Graph Contrastive Coding", KDD 2020

[7] Hou et al., "GraphMAE: Self-supervised Masked Graph Autoencoders", KDD 2022

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693807059

thread_type:
official_review

<thread_object>
creator: reviewer_9vk9

<content>
summary:
This paper proposes a large-scale graph generative model (LGGM). The proposed LGGM is pretrained over 5000 graphs from 13 domains and thereby demonstrates better zero-shot generation on graphs from unseen domains. Extensive experiments showcase the superior performance of the proposed LGGMs compared to baselines (e.g., DiGress) trained on specific domains. The proposed LGGM is equipped with the Text-to-Graph generation capability, which integrates extensive world knowledge in the underlying language model and offers users fine-grained control of the generated graphs. This is the very first one exploring the potential of the large-scale training paradigm on graph-structured data.

strengths:
- The paper is well-written. Concepts and proposed techniques are clearly explained and clarified.
- The proposed method is the very first one exploring the potential of the large-scale training paradigm on graph-structured data.
- The proposed LGGM shows exceptional zero-shot capability and novel text-to-graph generative ability.
- The experimental results show significant improvement and potential.

weaknesses:
- The proposed LGGM builds upon the DiGress architecture, with the primary differences being cross-domain pretraining and text-to-graph features. These aspects are not particularly novel in this field. Additionally, the expectation of enhanced zero-shot generative ability and fine-tuned performance from cross-domain pretraining is not surprising.
- The computational complexity and scalability of the discrete denoising diffusion process raise concerns, particularly when applied to large-scale real-world networks with high-dimensional node and edge categories. A discussion on how these challenges might be addressed would be beneficial.
- See questions below.

questions:
- What specific architecture is utilized for the reverse process mentioned in Line 193? More detailed information regarding this architecture should be provided to enhance understanding.
- It remains unclear whether the performance improvements observed over DiGress are primarily due to the cross-domain pretraining or any enhancements in the pretraining architecture. The authors should clarify how each component contributes to the overall performance, ideally through ablation studies or additional experiments.
- The authors mention that the default hyperparameter settings from DiGress are used for implementing LGGM during cross-domain pretraining. How do the authors ensure that these hyperparameters (best for certain domain training) are also optimal for cross-domain pretraining? Is there any empirical evidence or rationale behind their choice?
- As the model scales to accommodate larger graph corpora, is there a scaling law that governs the architecture’s performance? A discussion on how the model's behavior and hyperparameter effectiveness change with the size of the graph data would provide valuable insights.
- The textual descriptions are limited to graph structure and domain type. As the architecture has diffusion modeling on node/edge categories, is it possible to extend to text-to-graph generation conditioned on certain categories, which will be more useful and applicable in real-world scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730778751867

thread_type:
official_review

<thread_object>
creator: reviewer_ypc2

<content>
summary:
To mitigate the gap between graph generative models and general generative models (especially for texts / images), the paper propose a new generative training paradigm (LGGMs) for graphs from 13 domains. The pretrained LGGMs achieves better performance than existing graph generative models on the zero-shot tasks. With proper finetuning, the LGGMs can be applied to targeted domains with good performance. Furthermore, the LGGMs can be also used for text-to-graph scenarios by combining a text encoder.

strengths:
- The proposed LGGM framework can be trained on graphs from 13 different domains, which covers most of the graph scenarios. 
- The framework can be used both on zero-shot generation setting and fine-tuning setting. 
- The paper proposes to utilize the text encoder to enable LGGMs to use the text-to-graph capability.

weaknesses:
- The performance of zero-shot scenarios differs in different domains. On some domains such as FB, the LGGM does not work well. 
- The paper emphasizes that the LGGM is trained on over 5000 graphs. The number is not such large for a generative pre-trained model. 
- In the experiment of augmenting graph classification, the performance of LGGM is not promising (especially on the ENZYMES dataset).

questions:
1. The time and space complexity is quadratic to the number of nodes. How can the method be applied to graphs with large numbers of nodes?
2. The number of used training graphs is over 5000. What if the model (pre-)trains on millions of graphs?
3. In line 866 (Appendix C), the equation number is missing.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces an approach to training graph generative models using a large corpus of graphs from multiple domains. 

The strengths of this paper include its comprehensive coverage of various domains, the introduction of zero-shot and fine-tuning capabilities, and the innovative text-to-graph generation feature. However, the paper also has some weaknesses, such as performance variability across domains, limited training data size, technical contributions, and scalability concerns. The authors have partially addressed these issues by clarifying the novelty of their cross-domain pretraining approach, discussing potential scalability solutions, and providing additional experimental results.

In light of the authors’ responses and the overall contribution of the paper, I suggest a weak accept. Authors are encouraged to include some of the discussion in the revised paper.