Title: Generative Graph Pattern Machine
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generative Pre-Training

Graph Transformer

Substructure

Scalability
Abstract: Graph neural networks (GNNs) have been predominantly driven by message-passing, where node representations are iteratively updated via local neighborhood aggregation. Despite their success, message-passing suffers from fundamental limitations---including constrained expressiveness, over-smoothing, over-squashing, and limited capacity to model long-range dependencies. These issues hinder scalability: increasing data size or model size often fails to yield improved performance. To this end, we explore pathways beyond message-passing and introduce Generative Graph Pattern Machine (G$^2$PM), a generative Transformer pre-training framework for graphs. G$^2$PM represents graph instances (nodes, edges, or entire graphs) as sequences of substructures, and employs generative pre-training over the sequences to learn generalizable and transferable representations. Empirically, G$^2$PM demonstrates strong scalability: on the ogbn-arxiv benchmark, it continues to improve with model sizes up to 60M parameters, outperforming prior generative approaches that plateau at significantly smaller scales (e.g., 3M). In addition, we systematically analyze the model design space, highlighting key architectural choices that contribute to its scalability and generalization. Across diverse tasks---including node/link/graph classification, transfer learning, and cross-graph pretraining---G$^2$PM consistently outperforms strong baselines, establishing a compelling foundation for scalable graph learning. The code and dataset are available at https://github.com/Zehong-Wang/G2PM.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces G²PM (Generative Graph Pattern Machine), a Transformer-based framework for graph representation learning that bypasses traditional message-passing GNNs. G²PM tokenizes graphs into sequences of substructures (via random walks) and employs masked substructure modeling (MSM) for generative pre-training. The method demonstrates scalability, outperforming existing approaches on tasks like node classification, graph classification, and cross-domain transfer.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1- The study presents a comprehensive experimental evaluation, encompassing:\n    - node classification benchmarks, including large-scale datasets like OGBN-ArXiv\n    - graph classification tasks spanning molecular and social networks\n    - Cross-domain transfer experiments\n    - In-depth ablation studies\n\n2- The results demonstrate consistent performance improvements with model scaling, achieving gains up to 60 million parameters—supporting the paper’s core hypothesis.\n\n\nWeaknesses:\n\n1- Theoretical Foundations: Although the empirical results are compelling, the theoretical underpinnings could be strengthened through: \n   - formal examination of the expressive power of substructure sequences \n   - theoretical analysis comparing the approach to message-passing methods\n\n2- Computational Efficiency: Several key aspects related to scalability are not thoroughly addressed: \n   - The memory consumption when applied to large graphs \n   - The computational overhead compared to message-passing baselines \n   - How the length of walks affects overall computational cost\n\n3- The Related Work section lacks a clear and explicit discussion of how G²PM fundamentally differs from and advances beyond previous substructure-based approaches in a way that highlights its unique contributions.\n\nQUESTIONS:\n1- You use a fixed tokenizer based on random walks. Would a learnable tokenizer—such as one trained via reinforcement learning or self-supervision—further improve performance?\n\n2- For billion-edge graphs, what modifications would make G²PM practical (e.g., graph sampling, attention approximations)?\n\n3- The paper relies on random walks for substructure extraction, which have known limitations in capturing certain graph properties. How does G²PM address scenarios where key higher-order patterns (like large cliques or complex cycles) aren't sufficiently captured by walk-based sampling?\n\n4- Could you provide more insight into why positional embeddings show no benefit in your framework?\n\nLIMITATIONS:\n---", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces G²PM, a generative pre-training framework for graph representation learning based on Transformers. The key idea is to model graphs as sequences of substructures and apply Masked Substructure Modeling (MSM), where masked substructures are reconstructed conditionally on visible ones. The proposed approach enables the application of autoregressive transformer models to graph data by leveraging substructure-aware sequential representations.\n\nSTRENGTHS AND WEAKNESSES:\n[Strenghts]\n\n- **Clear and well-motivated contribution**: The paper is grounded in a well-defined need—the incorporation of semantic granularity into graph representations—to improve the effectiveness of Transformer-based models on graphs. The integration of substructures into a sequential modeling framework is conceptually novel.\n- **Thorough experimentation**: The authors provide comprehensive empirical results across various benchmarks, demonstrating the effectiveness of the proposed method.\n\n[Weaknesses]\n\n- **Missing prior work on graph-to-sequence representations**: The manuscript overlooks several prior works [1–5] that represent graphs as sequences suitable for Transformer-style training. Notably, the random walk approach employed in this paper closely resembles the method proposed in [5]. While the scope of this work focuses on pre-training, these related methods should still be acknowledged in the “Graph Transformers” and “Random Walks on Graphs” sections of the Related Work.\n- **Representation invariance and completeness**: The sequential representation is based on sampling and random walks. It is unclear whether the resulting representation is lossless. Specifically, how does the model capture the attachment points between substructures? A discussion on the limitations or assumptions of the representation would improve transparency.\n- **Limited benefit of cross-domain pretraining**: Although the paper aims to establish a graph pre-training framework, cross-domain pre-training occasionally degrades performance. For example, in Table 5, the performance on the HIV dataset is worse than in Table 3. The authors should elaborate on potential causes (e.g., domain mismatch or lack of generalization) and whether this highlights a limitation of their pre-training framework.\n\nQUESTIONS:\n**Scope of downstream tasks**: The current architecture removes the decoder and adds a linear prediction head to the encoder. This setup supports node-level and graph-level prediction but not generation. Is it possible to extend this framework to graph generation tasks?\n\nLIMITATIONS:\nThe paper only discusses the future direction and does not include any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a Transformer-based pre-training framework for graphs based on masked subgraph prediction. Subgraphs are generated through random walk tokenization and the model is shown to perform well across a range of scaling and ablation experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n\n- The authors undertake a comprehensive set of ablations and experiments. These clearly highlight the benefits of using subgraphs generated from their tokenization with a transformer encoder across a range of tasks. \n\n- The cross-domain transfer experiments are very compelling and show how the approach could make sense as a form of foundation model when compare to standard message-passing based approaches. \n\n- The paper is well-written and clearly motivated. \n\nWeaknesses \n\n- the technical development is limited in that the model is based on GPM and the innovation comes from the masked modeling component which in itself is well established. This lack of innovation is further highlighted by the fact that the performance improvement over GPM is negligible across a lot of tasks and can reduce performance. Given that the paper mainly adapts standard self-supervised learning techniques to GPM and doesn't see a huge performance boost, the scope of the paper could appear limited.\n\n- The self-supervised step can introduce a lot of additional computational time. Having some idea of the tradeoff between this and the performance improvement could be helpful to practitioners.\n\nQUESTIONS:\nAs well as addressing the two points in weaknesses, I have a couple of minor questions.\n\n[Line 128] How do you decide the ordering of the graph substructure tokens? Does this matter?\n\nDo the substructures from random walks significantly differ across datasets? Would this effect cross-domain adaption?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors claim that existing pretrain graph representation methods suffer from fundamental limitations due to MPNNs: constrained expressive power, over smoothing, over-squashing, and poor capacity for modeling long-range dependencies. Therefore, this paper aims to extend the success of Transformer-based generative pre-training to the graph domain, with the goal of enabling scalable graph representation learning. The authors propose G^{2}PM, which first applies random walk to extract node sequence for each node, and then obtain the nodes’ representation via a transformer. Finally, based on constructed substructures sequence, mask-prediction self-supervised loss is used to learn node representations. Experimental results on multiple datasets seem to validate the effectiveness of the pre-training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is easy to read and understand.\n2. The experiment evolve node classification from 7 datasets and graph classification from 7 datasets.\n3. The proposed G^{2}PM seems to have somewhat scalability from Figures 2 and 3, which is attractive.\n4. The paper conducts lots of experimental analysis about model architecture design in Section 3.\nWeakness:\n1. The proposed method is not a novelty. According to my understanding, the proposed G²PM obtains the substructure sequence of each node through random walks and then computes the node representation using sequence modeling methods (i.e., Transformer, RNN). Based on the constructed node representations and the sampled sequences, G²PM further learns the node representations using a general mask prediction method. Thus, the main innovation of the method is tokenizing substructures via random walks, which is simple and straightforward.\n2. In the scalability experiment of Figure 2, 60M is not a large model size. If the model size continues to increase, can the performance of the method still improve consistently?\n3. In the experiment of Model architecture, the author claims that deeper encoders enhance performance while increasing decoder capacity provides limited or negative returns. I personally think that the selected range of encoder and decoder layers is too small to support the argument. Moreover, which dataset is used for the experimental results in Figure 5(b)?\n4. To demonstrate the pretrain effectiveness, it is necessary to experiment with a link prediction task.\n5.“Model Layer. Table 5a shows” in line 181 has a linking error.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nThe authors claim in the limitation section that the current method is merely a masked-token prediction paradigm, and its effectiveness and scalability for next-token prediction remain unknown. Exploring next-token prediction is particularly important, as autoregressive methods have shown greater potential than autoencoding methods in the fields of NLP and CV for building foundation models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}