GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Authors: Qifang Zhao
Weidong Ren
Tianyu Li
Hong Liu
Xingsheng He
Xiaoxiao Xu
Venue: ICML 2025 poster (2025)
Keywords: Graph
GraphGPT
GPT
Foundation Model
Pre-train
Primary area: deep_learning->foundation_models

Abstract: We introduce *GraphGPT*, a novel self-supervised *generative pre-trained* model for graph learning based on the *Graph Eulerian Transformer* (**GET**). First, we propose **GET**, which combines a standard transformer encoder or decoder architecture with an innovative graph-to-sequence transformation method. This method converts graphs or sampled subgraphs into sequences of tokens representing nodes, edges, and attributes in a reversible manner using Eulerian paths. We pre-train **GET** using either of the two self-supervised tasks: next-token prediction (NTP) and scheduled masked-token prediction (SMTP). The pre-trained model is then fine-tuned for downstream tasks such as graph-, edge-, and node-level prediction. Despite its simplicity, GraphGPT achieves performance comparable to or surpassing state-of-the-art methods on multiple large-scale Open Graph Benchmark (OGB) datasets. It demonstrates exceptional results on the molecular property prediction dataset PCQM4Mv2 and the protein-protein interaction dataset ogbl-ppa. Notably, generative pre-training enables scaling GraphGPT to 2 billion parameters while maintaining performance gains — a breakthrough that overcomes the scalability limitations of traditional Graph Neural Networks (GNNs) and prior graph transformers (GTs). To advance research in graph foundation models and facilitate scientific discovery in chemistry, materials science, and related fields, we have released the
source code (https://github.com/alibaba/graph-gpt) and model checkpoints (https://www.modelscope.cn/organization/Alibaba-DT).

GraphGPT:
Generative Pre-trained Graph Eulerian Transformer
Qitang Zhao
Weidong Ren"
Tianyu Li!
Hong Liu
Xingsheng He
Xiaoxiao Xu
Abstract
1. Introduction
The deep learning revolution sparked by AlexNet (Krizhevsky et al., 2012) has driven remarkable progress in computer vision (CV) and natural language processing (NLP). The graph learning community similarly shifted from traditional machine learning to deep learning with the rise of graph neural networks (GNNs) (Kipf & Welling, 2017; Hamilton et al., 2017; Zhang & Chen, 2018; Wu et al., 2020).
We introduce GraphGPT, a novel self-supervised generative pre-trained model for graph learning based on the Graph Eulerian Transformer (GET). First, propose GET, which combines stan- dard transformer encoder decoder architecture with an innovative graph-to-sequence transforma- tion method. This method converts graphs or sam- pled subgraphs into sequences of tokens repre- senting nodes, edges, and attributes in a reversible manner using Eulerian paths. We pre-train GET using either of the two self-supervised tasks: next- token prediction (NTP) and scheduled masked- token prediction (SMTP). The pre-train pre-trained model is then fine-tuned for downstream tasks such as graph-, edge- and node-level prediction. Despite simplicity, GraphGPT achieves performance comparable or surpassing state-of-the-art meth- ods on multiple large-scale Open Graph Bench- mark (OGB) datasets. It demonstrates excep- tional results on the molecular property prediction dataset PCQM4Mv2 and the protein-protein inter- action dataset ogbl-ppa. Notably, generative pre- training enables scaling GraphGPT to 2 billion parameters while maintaining performance gains breakthrough that overcomes the scalability limitations of traditional Graph Neural Networks (GNNs) and prior graph transformers (GTs). To advance research in graph foundation models and facilitate scientific discovery in chemistry, mate- rials science, and related fields, we have released the source code and model checkpoints?.
Today, transformers dominate CV (Dosovitskiy et al., 2021; Liu et al., 2021) and NLP (Devlin al., 2019; Radford et al., 2018), scaling to billions of parameters (Liu et al., 2021; Brown et al., 2020) and achieving superhuman perfor- mance on benchmarks like ImageNet (Deng et al., 2009) and GLUE (Wang et al., 2019). These advances underpin trans- formative applications such as ChatGPT (Open-AI, 2023) and Midjourney (Midjoumey, 2023).
Despite progress, GNNs remain constrained by smoothing (Rusch et al., 2023) and over-squashing (Alon & Yahav, 2021), limiting their scalability and capacity to leverage large-scale graph data. Recent efforts to adapt transformers to graphs (Ying et al., 2021; Kim et al., 2022; Luo al., 2023; Müller et al., 2024) show promise but face critical challenges: 1). Structural Bias: Most graph transformers (GTs) rely on handcrafted features or GNN modules to encode graph topology, compromising general- ization. 2). Task Limitations: GTss excel at graph-level tasks but struggle with edge- and node-level objectives (Müller et al., 2024). 3). Pre-Training Gap: Unlike NL.'s suc- cess with self-supervised pre-training (Radford et al., 2018; Devlin al., 2019), GTs lack effective frameworks for gen- erative pre-training (Min et al., 2022; Müller al., 2024).
In this work, we propose GraphGPT, novel model for graph learning comprising three key innovations. 1). GET Backbone: A transformer- based architecture that operates on graph-equivalent token sequences via Eulerian paths, 2). Self- SuperSuue Pre-Training: Utilizing NTP and SMTP tasks (Radford et al., 2018; Chang et al., 2022), and 3). Task-Agnostic Fine-Tuning: Adapting the pre- trained model to supervised graph-, edge- and node-level tasks.
Alibaba Inc., Hangzhou, China. Qifang Zhao <james.zqf @ waw.zigaaaaaabaaaaaaaaaaaaaaaaaa alibaba-ine.com.. Xiaoxiao Xu <xiaoxiao xuxx @uxx@alibaba com.com>.
Correspondence to:
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s). https///ghhhhh.cmmmmabbbbagggggp
Our contributions are summarized as follows:

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Graph Eulerian Transformer (GET): We introduce GET, novel architecture that leverages Fulerian semi-Eulerian paths to losslessly and reversibly con- vert graphs into token sequences. By integrating sub- graph sampling and node identity encoding, GET effi- ciently processes graphs of arbitrary sizes. A standard transformer encoder or decoder then applied to these sequences, eliminating the need for specialized archi- tectural modifications.
classificatinn/regresiion link prediction, and node classifi- ba reformatting task objectives into sequence-base based inputs. This unified approach maximizes the transfer of pre- trained knowledge.
2.2. Graph to Sequence of Tokens
To convert graphs into token sequences, we employ distinct strategies based on graph size:
Generative Pre-Training Framework: GraphGPT is pre-trained using NTP or SMTP tasks, offering three advantages: a) Captures structural and semantic graph patterns without handcrafted features or domain- specific architectures, 6) Scales to over 2 billion param- eters with sustained performance gains, and c) Enables effective graph generation through its sequence-basee formulation.
Small graphs (e.g., molecular graphs) are directly seri- alized using the method in $2.2.1.
Large graphs (with up to billions of nodes/edges) are first decomposed into subgraphs via the sampling pro- cess described $2.2.2. Node identity preservation ($2.2.3) ensures structural consistency during this de- composition. These subgraphs are then serialized using $2.2.1's guidelines.
Unified Task Formatting: We design a novel method to reformat graph-, edge-, and node-level tasks into se- quences compatible with transformers. This approach allows downstream tasks to fully exploit pre-trained representations while unifying pretext and target task frameworks.
2.2.1. SERIALIZING GRAPHIS WITH (SEMI-)EULERIAN PATHS
We propose a lossless, reversible graph serialization method based on traversing all edges and nodes via (semi-)Eulerian paths. This approach guarantees:
State-of-the-Art (SOTA) Performance: Extensive periments on OGB datasets demonstrate GraphGPT's superiority: it achieves SOTA results in graph- and edge-level tasks (e.g., molecular property prediction on PCQM4Mv2 and protein-proteni interaction on ogbl-ppa), while delivering competitive performance in node-level tasks.
Complete representation of nodes and edges in the sequence.
Bijective mapping between the graph and its serialized form
Algorithmic Foundation: The problem aligns with the Chi- nese Postman Problem (Mei-Ko, 1962; Edmonds & Johnson, 1973), which seeks the shortest path traversing all edges. For graphs lacking (semi-)Eulerian properties, we apply Fulerization (Edmonds & Johnson, 1973; Daubechies & Hughes, 2009), duplicating minimal edges to create an Eu- lerian multigraph.
2. Approach
2.1. Overview
GraphGPT employs three-stage framework: 1). Graph- to-Sequence Transformation of GET: The input graph is converted into sequence of tokens via (serin)Eulerian paths, ensuring lossless, reversible mapping between the graph and its sequential representation. This transforma- tion preserves node, edge, and attribute information while enabling compatibility with transformer architectures. 2). Self-Supervised Pre-Training: standard transformer back- bone (c.g.. Llama; Touvron et al. (2023)) processes these se- quences using the tasks NTP SMTP (Radford et al., 2018; Chang et al., 2022). These tasks enable the model to learn structural and semantic graph patterns without task- supervision. 3). Task-Specific Fine-Tuning: The pre-trained model is adapted to downstream tasks including graph
Implementation:
1. Connectivity Check:
Check if the graph is connected. If not, link discon- nected components by adding synthetic edges between randomly selected nodes. For example, given graph with disconnected components A, B, and C, connect and B via random node pair, then B and C similarly.
Label these edges with a dedicated JEDGEJJMM token and default attribute tokens.
Forr a quick recap, connected graph with every node of even degree is Fulerian, and with exactly two nodes of odd degree semi-Falerian. In this paper, "aaerinFalerrian refers both Fulerian and semi-Fulerine unless specified otherwise.
'Given fixed starting node and predetermined node indices, the Fulerian path generated by NetworkX (Hagberg al., 2008) guaranteed unique.
2

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
F
A
B
C
A
C
16
14
16
17
18
B
Serr-Euuerinn path
Re-index
R4
R1
R1
R1
R;
R;
71
R;
1
2
3
short
[p]
6t
6c
6t
Et
A
Eulerizatio
tycle re-dere
[p]
e E
e1
E
e1
e
13
14
15
16
14
16
17
Atach
Sequence lem:
13
[p]
[p]
[p]
[p]
[p]
16
[p]
17
[p]
D
Attach
[p]
74
[p]
R4
[p]
n1
[p]
[p]
n1
[p]
[p]
long
[p]
[p]
[p]
et
[p]
6c
[p]
6t
[p]
[p]
[p]
E
[p]
[p]
61
[p]
6 [p]
e
[p]
6
[p]
[p]
[p]
Sequence e len: 2(m
1)
+
prolonged
13
14 12g
15
16
14
12
6x
61
16
n1
17
18
Sequence len: [(m
1)
Figure 1. Overview of Graph-to-Scquena Tokenization. (Left) The process of converting (sub)graph into a token sequence via (semi-)Eulerian path. Dashed lines indicate duplicated edges added during Eulcrization enable full edge traversal. (Right) Threc methods (short, long, prolonged) for integrating node/edge attributes into the Eulerian sequence. For simplicity, we assume one node attribute (rt, and one edge attribute (ei) per edge. Special tokens include padding token [p] edge type e (e.g.. incoming/outgoing direction). Sequence parameters: m is length of the Eulerian sequence, and / 2 #edge-attrs + #node-attrs (here, / = 4).
This ensures the graph becomes connected, enabling Eulerian path generation.
N 256), ensuring uniform training across all index tokens.
Cyclic re-indexing critical for datasets like Triangles ($3.2.1), where test graphs have significantly more nodes than training graphs (e.g., test graphs up to 100 nodes vs. training graphs 25 nodes). With- out re-indexing, higher-index tokens (e.g., 25 255) remained undertrained, degrading performance.
Path Identification:
Check Eulerian properties using established criteria (West et al., 2001).
If non-Eulerian, perform Eulerization to enable path traversal.
4. Attribute Attaching:
Randomly sample one valid path from possible candi- dates, introducing stochasticity as a data augmentation strategy akin to computer vision techniques (Perez & Wang, 2017). This stochasticity forces the transformer to learn invariance across different paths of the same graph and empirically reduces overfitting.
Discrete attributes: Direct tokenization.
Continuous attributes: Digit-wise tokenization (e.g.. 3.14 (3..,1,4].
Edge directionality: Distinet tokens for incoming/out- going edges (e.g., [-+], (+)).
3. Node Re-indexing:
Three attribute integration strategies (Fig. 1): short, long, and prolonged formats.
Assign indices 1,... , based nodes' first appearance in the path (e.g., Fig. 1: node F + O, A 4 1).
Theoretical Guarantee:
Introduce cyclic re-indexing: i' = (i N,,, where is random integer and N (hyperparameter) exceeds the maximum node count. Without eyclic re- re-indexing, Eulerian paths would always start with low-index to- kens (e.g., 0, 1, 2), leading to skewed token frequency distributions. Cyclic re-indexing randomizes start- ing indices (e.g., selecting from {0, 1,,,,, 255) for
The serialization is lossless and reversible (up isomor- phism) per the Eulerian Path Theorem: reconstructing edges from adjacent tokens recovers the original graph structure (Grohe & Schweitzer, 2020). For example, in Fig. I's se- quence, connecting consecutive tokens (13 + 14 + 15 + ...) reconstructs all edges, yielding a graph isomorphic to the input.
3

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
2.2.2. SUBGRAPH SAMPLING
mirrors variable-length encodings like utf-8 (Allen al., 2012, Chapter 2), where characters are represented by 1-4 bytes (vocabulary size 256).
Directly serializing large graphs into sequences via the method in $2.2.1 produces excessively long sequences that exceed transformer context limits. While truncating such sequences is possible, this approach faces two critical issues:
Our ablation studies ($3.5.3) demonstrate this method's effectiveness preserving node identity.
Computational Overhead: Eulerization and path iden- tification for large graph computationally expen- sive.
2.3. Modeling with the Transformer Decoder/Encoder
We demonstrate how the transformer architecture processes graph token sequences under a unified pre-training and fine- tuning paradigm for diverse graph tasks.
Inconsistent Training: Sequence fragmentation intro- duces mismatches between pre-training and fine-tuning data formats.
2.3.1. PRE-TRAINING WITH THE NTP OR SMTP TASKS
Self-supervised pre-training has proven critical for success in NLP (Devlin et al., 2019; Radford et al., 2019) and CV (He et al., 2022; Chang et al., 2022). We adopt two foun- dational generative tasks: NTP, which enables SOTA per- formance in NLP (Brown et al., 2020), and SMTP, which extends masked prediction with scheduled masking rates.
To address these challenges, we adopt subgraph sam- pling-a scalable strategy that decomposes large graph into smaller, manageable subgraphs before serialization.
Implementation:
We use the ShaDow Hopp sampler (Zeng et al., 2021) to extract localized subgraphs centered on randomly selected nodes edges.
Implementation:
Masking (SMTP only): For node-level masking, all occurrences a masked node in the Eulerian sequence are hidden to prevent leakage (e-g., two occurrences of node A in Fig. 1 masked concurrently).
Sampler parameters (e.g.. hop depth, neighbor count) are preconfigured to ensure generated sequences fit within the transformer's context window. These param- eters are dataset- and hardware-dependent (see App. A.1 for configuration details).
Mask Scheduling (SMTP only): Following Chang et al. (2022), we adopt the same linear scheduling function, which empirically balances training stability and per- formance.
2.2.3. NODE IDENTITY ENCODING
Preserving global node identities during subgraph sampling essential to avoid information loss. While unique token- based encoding (via learnable embeddings) is theoretically viable, it becomes impractical for graphs with billions of nodes due to:
Mulei-Token Prediction (NTP and SMTP): For se- quences encoded in short or long formats (Fig. 1), we predict all non-padding tokens per column simulta- neously, similar to Gloeckle al. (2024).
2.3.2. FINE-TUNING ON DOWNSTREAM GRAPH TASKS
Vocabulary Explosion: A 10-billion-node graph would require a vocabulary of size 10
We adapt the pre-trained transformer to supervised tasks by reformatting the sequence-based inputs, ensuring alignment with pre-training:
Memory Constraints:
Corresponding embedding ma- trices become prohibitively large.
Graph-Level Tasks (e.g., classification/regrssion))) Ap- pend special [GSUM] token the sequence.
Solution: Multi-Token Node Encoding. We propose en- coding each node as a combination of tokens, reducing vocabulary size exponentially. For example: 1ode- graph can be uniquely represented with two tokens from size vocabulary (105 x 10 10. Graph partition- ing via METIS (Karypis & Kumar, 1997) enables this by dividing the graph into 10 clusters, each containing ~ 10 nodes.
2. Edge-Level Tasks (e.g., link prediction): Append to- kens of the target edge's source and destination nodes.
3. Node-Level Tasks (c.g., node classification): Append the target node's token to the sequence.
The final token's 's output is fed to a randomly initialized multilayer perceptron (MLP). Fig. 2 (Appendix) illustrates the implementation.
Trade-offs: Increasing & (e.g., l = 5) allows smaller vo- cabularies (100" 10)0) but lengthens sequences. This

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
The transformer weights initialized from pre-trained checkpoints, and the MLP layers initialized randomly. All parameters are updated during fine-tuning.
Empirical results demonstrate that SMTP pre-training achieves superior comparable performance across all benchmarks. Unless otherwise specified, reported GraphGPT results utilize SMTP pre-training.
This formulation ensures seamless knowledge transfer from pre-training, mirroring successes in NLP (Brown et al., 2020; Wei et al., 2022).
3.2. Graph-Level Tasks
3. Experiments
Table I. Results of the graph regression task the PCQM4Mv2 dataset. The metric is mean absolute error (MAE), the smaller the better. 86% of the valid dataset is added training after selection. Superscript numbers indicate source references, while subscript numbers correspond model vari- ants in Table 11 (Appendix). The best results are in bold, and second-bes are underlined. This notation convention applies to all subsequent tables.
3.1. Datasets
Recent advances in AI for scientific discovery (Wang al., 2023) motivate our evaluation of GraphGPT on large-scale scientific graph datasets spanning physics, chemistry, and bioinformatics. To demonstrate its versatility across graph tasks, we select benchmarks for graph-, edge- and node- level objectives:
MAE I
Models
Params
Valid
Test
Graph-level: PCQM4Mv2 (quantum chemistry), oghg- molpeba (molecular property prediction) and Triangles (triangles counting).
GCN'
0.1379
0.1398
2.0M
GIN
0.1195
0.1218
3.8M
GNN
GCN--VN
0.1153
0.1152
4.9M
GIN--VN
0.1083
0.1084
6.7M
ogbl-ppa (protein-protein associations) and ogbl-citation2 (citation networks).
Edge-level:
TokenGT"
0.0910
0.0919
48.5M
Graphformer
0.0864
N/A
48.3M
GT
GPS-Deep
0.0852
0.0862
138. IM
ogbn-proteins (protein interaction net- works) and ogbn-arxiv (paper categorization).
Node-level:
GPS++ (no 3Dy
0.0818
N/A
40.0M
GPTrans-L.
0.0809
0.0821
86.0M
GraphGPT-M
0.0827
N/A
37.7M
Dataset statistics detailed in Table (Appendix A).
GraphGPT-BB.
0.0807
N/A
113.6M
Ours
GraphGPT-BB B
0.0793
N/A
227.3M
GraphGPT-BB Bas 0.0792
0.0804
453.4M
PCQM4Mv2 contains 3.7 million organic molecules from PubChemQC (Nakata & Shimazaki, 2017). Nodes represent atoms (9D attributes: atomic number, chirality, etc.), and edges denote chemical bonds (3D
Kipf Welling (2017), Pxu al. (2019), AGilmer al. (2017), "Kim (2022),
'Ying al. (2021), "Ramplsck al. (2022), "Masters ct. (2023), *Chen al.
(2023b)
attributes: bond type, stereochemistry, conjugation).
We evaluate GraphGPT on two molecular datasets where tasks involve predicting quantum chemical properties solely from 2D molecular graphs a practical alternative to relying on 3D equilibrium structures. Specifically, PCQM4Mx2 predicts the HOMO-LUMO energy gap, and molg-aa predicts 128 binary molecular properties.
ogbg-molpeba is smaller molecular dataset (Wu et al., 2017) with the same nodeledge attributes.
Triangles (Knyazev al., 2019) contains 45k graphs (no node/edge attributes).
ogbl-ppa: Nodes are proteins from 58 species; edges represent functional associations (Szklarezyk et al., 2019).
On PCQM4Mv2, GraphGPT achieves a test MAE of 0.0804, significantly outperforming the previous SOTA (0.0821, Chen et al. (2023b))
Compared to GTs like TokenGT, Graphformer, GPS, and GPTrans-whidh require handcrafted features intricate architectures encode structural information Information-GrrphhGPP attains superior performance without manual feature engi- neering. It also surpasses GNNs by substantial margin (Table 1).
ogbl-citation2: A directed citation network with 3 million papers (nodes) and >> 30 million edges.
Undirected, weighted graph of 132,534 proteins (nodes) with 8D edge attributes encoding as- sociation strengths.
ogbn-proteins.
Analysis (Tables and 2):
Citation network 169,343 papers; tasks involve predicting 40 subject categories.
orbn-arxiv:
Lossless Serialization: The Fulerian path-based serializa-
5

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
tion and generative pre-training enable GraphGPT to fully capture structural and semantic graph information.
This task is challenging even for in-distribution (ID) graphs and considerably harder for out-of-distribution (OOD)
graphs.
Scalabiliry: While GTs with fewer parameters often plateau when scaled (Shi et al., 2022), GraphGPT shows consistent improvement up to 200M parameters. The log-log scaling law plot for both pre-training loss and supervised fine-tuning loss is shown Fig. (Appendix).
Pre- Training Setup: We augment pre-training with diverse datasets, i.e., Reddit-threads (Rozemberezki et al., 2020), Erdös-Rényi random graphs (Erdos et al., 1960), and Inter- nal real-world graphs (See Table 9, Appendix A).
Parameter Efficiency: GraphGPT's larger parameter count may reflect its capacity to implicitly learn features that other GTs encode manually. Generative pre-training also allocates model capacity to generation tasks, potentially limiting dis- criminative performance of models at smaller scales.
Analysis (Table 3):
Pre-Training Efficacy: GraphGPT achieves comparable ac- curacy to GTs on ID graphs and superior OOD generaliza- tion (lower variance). This demonstrates that generative pre-training effectively encodes structural knowledge trans- ferable to downstream tasks.
Limitations: Pre-training on additional external large-scale molecular datasets yielded diminishing returns, suggesting saturation in 2D structural information. Incorporating 3D molecular data could help address this limitation.
Impact of Graph Types: Pre-training on real-world graphs (e.g., internal datasets) outperforms random Erdōs-Rényi graphs, suggesting meaningful structural patterns in real- world data enhance model learning.
Transfer Learning: When fine-tuned on ogbg-molpcba, our PCQM4Mv2-pretraine model achieves results exceeding powerful GNNs (GCN, GIN) and matching SOTA GTs (Table 2).
Dataset Diversity: Combining Triangles with diverse datasets (Reddit-threads, internal graphs) yields better per- formance than pre-training on Triangles alone. This high- lights the importance of diverse pre-training data for learn- ing generalizable structural patterns.
Table Results of the graph classification task on ogbg- molpeha dataset. All the baseline results are from the OGB leader- hoard or the corresponding papers. indicates the model is pre- trained on PCQM4M-v2 v2 dataset.
Attributed Graphs: Models pre-trained on attributed graphs (PCQM4Mv2, ogbl-ppa, ogbn-proteins) and fine- tuned on Triangles achieve significant improvements: /4.3%%/...%%%/
vs. 32.6% (baseline GET without This confirms that structural knowledge is ob- tained even when pre-training includes node/edge attributes.
Average Precision (%) f Valid
Models
Params
Test
GCN
20.2010.24
20.59
0.57M
GIN
22.66.0.22
23.05
1.92M
VIN
29.1710.15
30.65 30.65_o.00
6.1M
3.3. Edge-Level Tasks
NGIN-VNN
30.07
30.59_o.58
44.19M
We evaluate GraphGPT on link prediction using the ogbl- ppa and ogbl-citation2 datasets. Results are summarized in Table 4.
PDF
30.31
31.15_o.20
3.84M
Graphormer-L""
31.4010.22
32.27 20.24
119.5M
EGT-Larger
29.6110.24
N/A
110.8M
LargeLarrge
31.5010.10
N/A
118.3M
Performance Superiority: GraphGPT significantly outper- forms all baseline methods, including GNNs, heuristic mod- cls, and fatent-factor approaches, across both datasets. This underscores the effectiveness of generative pre-training and sequence-based modeling for edge-level tasks.
GPTrans-T.
32.43.0.22
N/A
86.0M
GraphGPT-M*
30.134..26
31.62_0.24
37.7M
GraphGPT-B|, 12
31.28
113.6M
GraphGPT--
31.81
32.54
227.3M
Scalability: GraphGPT scales seamlessly to 2 billion param- eters, achieving sustained performance gains with increas- ing model size. This motivates future exploration of even larger architectures and datasets.
'Kipf& Welling (2017), Xu al. (2019), "Brossard al. (2020), *Gilmer al. (2017), 'Zhang& & Li (2021), "Yang el al. (2023), "Ying el al. (2021), "lussai et al. (2022), PPurk al. (2022), 1"Chen et (2023h)
3.2.1. GRAPH STRUCTURE UNDERSTANDING
Transformer Efficacy: To our knowledge, GraphGPT is the first transformer-bact model to achieve SOTA results on ogbl-ppa and ogbl-citation2, demonstrating the viability of sequence-driven architectures for large-scale edge-level tasks.
To evaluate GraphGPT's ability to leam structural patterns through generative pre-training, we use the Triangles dataset with the task of counting triangles. The dataset is split into: 1). Training/Validation: 30k and 5k small graphs (< 25 nodes); 2). Testing: 5k small graphs (Test-small) and 5k large graphs (25-100 nodes, large..
6

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Table Results of the graph classification task the Triangles dataset. Superscript numbers indicate source references, while letters denote specific pre-training datasets. We report averaged metrics from 10 independent runs to ensure statistical reliability. The baseline results are from Müller (2024).
Table 4. Results of the link prediction task on the oghl-ppa and ogbl-citation2 datasets.
ogbl-ppa HR @ 100 (%)
ogbl-citation2 MRR (%) f
Models
Common Neighbor
27.65=0.00
51.47
Adamic Adar
32.45=0.00
51.8940..0
Accuracy (%) f
Models
Params
Resource Allocation!
49.33
51.98.20.00
T-small
T-large
Node2Vec
22.26_0.63
61.41
GIN!
71.53
33.5410.30
0.15M
Matrix Factorization
32.29=0.94
51.86=4.43
Transformer
12.08 10.31
10.0110.04
0.2M
GCN
18.67=1.32
84.74.0
Transformer-LapPE
78.2910.25
10.6442..4
0.2M
GraphSAGE
16.55_2.40
82.60.0.30
Transforme:RRWEE*
99.40-2.110
54.76.77.24
0.2M
SEAL
48.80_a.16
87.67 #2
Graphormer'
99.09
42.3416.48
0.2M
AGDN'
4..23=1.50
85.4920.22
GET-B
1..60
1..99
113,5M
SIEG
63.22=1.74
10.18
GraphGPT-B'
92.1610.22
26.51
113.5M
MPLP
15.24
90. 00.722
GraphGPT-B²
81.38:0.227
37.68
113.5M
RefinedGAE"
73.74 00.92
84.55
GraphGPT-B
99.08
38.8013.60
113.5M
GraphGPT-B
90.93
40.7941.40
113.5M
GraphGPT-M
65.4 65.44
92.82
GraphGPT-B
68.76 0.667
93. 93.05.2 0.200
GraphGPT-BB BB"
64.28
17. 17.38+0.61
113.5M
GraphGPT-XXL.
76.55
N/A
GraphGPT-B
86.14
26.94444.80
113.5M
GraphGPT-B-
86.5712.74
23.4511.44
113.5M
Zhou (2009), "Grver & Leskovec (2016), 'Mnih & Salakhuidinov (2008),
GraphGPT-B"*
81.83+0.81
39.62+1.84
113.5M
*Kipf Welling (2017), "Hamilton et al. (2017). 'Zhong et al. (2021), "Sun et al. (2025), "Shi al. (2024), "Dong et al. (2024), "Ma et al. (2024)
GraphGPT-B
98.6844.118
50.07
113.5M
GraphGPT-BB>c
98.26+0.30
52.33
113.5M
GraphGPT-B
89.98
33.4512.51
113.5M
local aggregation with feature propagation.
GraphGPT-M**h
95.07 0.667
51.72+1.12
33.7M
GraphGPT-B
98.63
58.96 1.90
113.5M
3.5. Ablation Study
Xu al. (2019), VVaswani et (2017), 'Rampásek et al. (2022), "Ying et (2021)
We analyze the impact of three core components of GraphGPT: pre-training, node re-indexing and node identity encoding.
Pre trained with: "Triangles (45K), "Reddit ihreads (0.2M), 'Internal dataset (3.IM),
4 Random graphs (3.1M), PCQMMM-2 (3.7M). fogbl-ppa (1), "ogbo-proteins (1).
3.5.1. -RAINIIGG
3.4. Node-Level Tasks
The self-supervised NTP or SMTP tasks are central to GraphGPT's success. As shown in Table 6, pre-training de- livers performance improvements of 10-100% across graph- edge-, and node-level tasks. These gains highlight its role in enabling the model learn intrinsic graph structural pat- terns and capture semantic relationships inherent in node and edge attributes.
We evaluate GraphGPT on two node-level benchmarks: ogbn-proteing predicts 112 binary protein function labels, and ogbn-arxiv classifies arXiv papers into 40 subject cate- gories. Results are summarized in Table 5.
ogbn-proteins: GraphGPT surpasses well-tuned GNN base- lines (GCN, GraphSAGE, GAT) and significantly outper- forms graph transformers (GTs), Remarkably, GraphGPT achieves competitive performance with input subgraphs of 40 nodes, while SOTA GNNs like AGDN (Sun et al., 2025) require subgraphs with > 22, 000 nodes.
3.5.2. NODE RE-INDEXING
As illustrated in Fig. 1, we re-index the nodes based on their order the (semi-)Eulerian path. To evaluate the effectiveness of this approach, we conduct experiments on the 7.
ogbn-arxiv: GraphGPT delivers performance comparable or approaching SOTA graph transformers and optimized GNNs.
dataset, with results summarized in Tab.
The strong performance with minimal neighborhood sam- pling suggests that generative pre-training effectively en- codes global structural and semantic graph information into node token embeddings and transformer parameters. This contrasts with traditional GNNs, which rely on extensive
While node re-indexing increases pre-training loss, it con- sistently improves performance on downstream tasks across various model sizes. This technique acts as a form of data augmentation, preventing the model from memoriz- ing graph-specific artifacts such as arbitrary node label-

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Table Results of the node classification task the ogbn-proteins and ogbn-arxiv datasets.
Table 6. Ablation study of pre-training on the datasets of various types of tasks. Superscripts D/E stand for transformer decoder/en- coder. means both molpeba and PCQM4Mv2 datasets are used for SMTP pre-training, and f indicates that the model is further trained using PCQM4M-v2's regression task. For the PCQM4Mv2 datasct, the metric is MAE, the lower the better.
oghn-proteins ROC-AUC (%)
oghn-arxiv Accuracy (%)
Models
GCN
77. 40.49
73.53
GraphSAGE
82.21 82.21
73.00
GAT
40.46
73.30
DATASETS
PRE-TRAINING
TEST
VALID
DRGAT
N/A
74.16#0.07
X
N/A
0.0978
AGDN
88. 88.65
73.41+0.25
PCQM4Mv2
x
N/A
0.0856
DeeperGCN?
85.80
71.9210.16
NTP
N/A
0.0875
SMTP
N/A
0.0807
GraphGPS
77.15+0.64
71.23+0.59
NAGphormer"
72.17+0.45
70.88
X
12.80
13.31
Exphormer
77.62+0..3
72.32+0.36
X
25.80
26.33
GOAT
79.41
72.76.0.29
OGBG-MOLPCHA
NTP
23.85
27.77
SMTP
27.56
28.74
NodeFormertiz
77.86
67.78
SMTP
27.20
28.49
SGFormer
79.92:0.48
72.76+0.33
SMTP* + FT
28.07
29.01
Polynormer!
79.5340.67
73.4040.22
x
41,28
40.14
GraphGPT-S
83.5640.16
70.83 4.033
x
42.13
41.57
GraphGPT-M 84.0210.21
71.20+0.34
OGBL-PPA
NTP
55.56
54.87
GraphGPT-B 80.33
72.10+0.30
SMTP
55.68
54.93
LLoo al. (2024), Kiipf& Welling (2017), Hammilton al. (2017), "Vaswani et al. (2017), "Zhang et (2023), "Sun al. (2025), et (2023), 'Rampásek et (2022). "Chen al. (2023a), SShrzzad al. (2023), "Kong al. (2023), (P et.l.
X
57.52
61.19
x
53.20
$6.39
OGBN-PROTEINS
NTP
75.61
80.47
SMTP
83.56
87.73
(2022), 1u et al. (2024), DDDeng (2024)
Table 7. Ablation study of node re-indexing on the ogbg-molpcba dataset with model sizes. means pre-training.
ing- ing-and and thereby enhancing generalization.
Furthermore, re-indexing enables constrained decoding of node tokens during graph generation with GraphGPT, re- ducing the search space for valid outputs.
PARAMS
RE-INDEX
PT Loss
TEST
VALID
X
0.0844
23.10
25.25
4.48M
V
0.0874
23.85
27.77
X
0.0689
22.70
26.21
3.5.3. NODE IDENTITY ENCODING
12M
V
0.0750
25.17
28.57
Node identity encoding (see $2.2.3) representing nodes' identity in large graphs as multiple tokens-in critical for edge- and node-level tasks. Using GraphGPTT-min (a lightweight variant to conserve computational resources), we demonstrate that this method significantly enhances per- formance (Table 8). Further implementation details are provided in Appendices A and F.
Dataset size. Performance on some small- to medium-sized datasets (c.g.. ogbn-arxiv) lags behind traditional GNNs. This can be mitigated by expanding datasets with semanti- cally aligned data.
Computational Cost. Pre-training large-scale graphs (ogbn-proteins, ogbl-ppa) or extensive small graphs (PCQM4Mv2) with 50M+ parameters is resource-intensive. For example, pre-training GraphGPT-B (100M+ parame- ters) on PCQM4My2 with x 10 tokens requires ~ 63 V100 GPU hours, and fine-tuning incurs ~ 3 V100 GPU hours per epoch in the distributed data parallel setting with GPUs.
4. Limitations
We critically assess the limitations of GraphGPT to contex- tualize applicability and inspire future improvements.
Transferabillit. GraphGPT's reliance on dataset-specific pre-training limits its ability generalize across domains with divergent semantics (e.g., social networks vs. molec- ular graphs). However, it demonstrates robust cross- dataset structural understanding ($3.2.1) and effective intra- domain transferability, as evidenced by molecular data ex- periments ($3.2).
While GraphGPT is less practical for small datasets due to compute-performent trade-offs, it excels with large-scale data. Emerging techniques like quantization (Dettmers et al., 2022; Frantar et al., 2022), distributed training frameworks (Rasley al., 2020; Shoeybi et al., 2019), and transformer
8

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
scale benchmarks. By unifying pretext and downstream tasks into a sequence-based paradigm. GraphGPT demon- strates strong transferability in capturing both structural graph patterns and domain-specific knowledge (e.g., molec- ular properties). Notably, scaling GraphGPT to billions of parameters yields consistent performance gains, highlight- ing its potential as a foundation for graph-centric foundation models.
Table Ablation study of node identity encoding (NTE) on the ogbl-ppe ppa and pront-roteins datasets.
DATASETS
PARAMS
NIB
TEST
VALID
44.38
45.08
OGBI--PPA
14.75M
55.56
54.87
60.22
65.66
OGBN-PROTEINS
10.76M
75.61
80.47
Looking ahead, GraphGPT's architecture is inherently scal- able-capabl of expanding to hundreds of billions of pa- rameters- and offers promising avenues for integration or alignment with large language models (LLMs), bridging graph reasoning and textual intelligence.
optimizations (Dao, 2024) are poised to alleviate these costs.
Future Directions. These limitations highlight opportu- nities for research in cross-domain transfer, data-efficient training, and scalable architectures.
Acknowledgements
5. Related Works
We thank colleagues Guoshuai Wang and Tiange Xu for their insightful discussions and feedback. We appreciate the constructive feedback from the anonymous reviewers, which significantly improved the clarity of this paper. We thank our families for their unwavering support and encouragement throughout this project.
Graph Neural Networks (GNNs) GNNs have dominated graph learning for decades, with numerous variants achiev- ing strong performance across tasks (Wu et al., 2020). However, they face fundamental limitations such as over- smoothing and over-squashing (Rusch et al., 2023; Alon &
Yahav, 2021), 1), which hinder their scalability and ability to model long-rang range dependencies.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
Graph Transformers (GTs) Inspired by transformers' suc- cess in NLP and CV, recent work has adapted these archi- tectures to graphs (Ying et al., 2021; Rampásek et al., 2022; Müller et al., 2024). While GTs achieve competitive results on large-scale graph-level tasks (Müller et al., 2024), they typically rely on handcrafted structural features or GNN- based modules to encode graph topology-either in input representations (Ying al., 2021; Kim al., 2022; Masters et al., 2023) attention mechanisms (Ying al., 2021; Chen et al., 2022; Luo et al., 2023).
References
Allen, D., Anderson, D., Becker, J., Cook, R., Davis, M., Edberg, P., Everson, M., Freytag, A., Iancu, L., Ishida, R., et al. The unicode standard. Citeseer, 2012.
Pre-training and fine-tuning The self-supervised pre- training and supervised fine-tuning paradigm, popularized by transformers (Vaswani et al., 2017), revolutionized NLP (Devlin et al., 2019; Radford et al., 2018). Scaling this approach with web-scale data (Brown et al., 2020) and tech- niques like instruction tuning (Wei et al., 2022) or rein- forcement learning from human feedback (Ouyang et al., 2022) further advanced the field. In CV, self-supervised methods like MAE He et al. (2022) and MaskGIT (Chang et al., 2022) demonstrated that masked prediction tasks (e.g., reconstructing masked image patches) enable transformers to achieve SOTA results.
Alon, U. and Yahav, E. On the bottleneck of graph neural networks and its practical implications. In In- ternational Conference on Learning Representations, 2021. URL https: / http:///ppneeeeeeeeeeeeee net.:eeuweet/Eoru forum?
1d=1800PhOCVH2.
Brossard, R., Frigo, O., and Dehaene, D. Graph convo- lutions that can finally model local structure. CoRR, abs/2011.15069, 2020. URL http::s:://axxi...o abs/2C11.15065
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
6. Conclusions
We introduce GraphGPT, a novel model built on the GET backbone, which achieves SOTA or SOTA perfor- mance across graph-, edge,, and node-level tasks on large-
Chang, H. Zhang, H., Jiang, L., Liu, C., and Freeman, W. Maskgit: Masked generative image transformer. In
9

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 11305-11315. IEEE, 2022. doi: 10. 1109/CVPR52688.2022 202..0000..222.0...0. URL https://doi. org/10 1tte//001119////5522220 2022. .112220115 01103.
Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L.. 8- bit optimizers via block-wise quantization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenRe- view.net, 2022. URL. https:/ /pppppeeeeeeeeeee wetnnrevieweeeeee net/ forum? saauumiiiddhhpppp/22253
Chen, D., O' O'Bray, L.,, and Borgwardt, K. M. Structure- aware transformer for graph representation learning. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato. S. (eds.), International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Balti- more, Maryland, USA, volume 162 of Proceedings of PMLR,
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio. T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume / (Long and Short Pupers), pp. 4171-41866 Association for Computational Linguistics, 2019. doi: 10. 10.18653/1/119-44233 URL https://doi... org/ 10.18653/v1/119-11
Machine Learning Research, pp. ppwoceddings....m mlr. press/
: / 2022. http:: v1 chen22r html.
Chen, J., Gao, K., Li, G., and He, K. Nagphormer: A tok- enized graph transformer for node classification in large graphs. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a. URL https: //openreview net / forum? ii wwwddd-KKeee111
Dong, K., Guo, Z., and Chawla, N. V. Pure message passing can estimate common neighbor for link prediction. In Globersons, A., Mackey, L.., Belgrave, D., Fan, A., Pa- quet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Chen, Z. Tan, H., Wang, T., Shen, T., Lu, T., Peng, Q. Cheng, C., and Qi. Y. Graph propagation trans- former for graph representation learning. In Proceed- ings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR. China, pp. 3559-3567. ijcai.org, 2023b. doi: 10.24966/UCCAA.2223333966 URL https: adwi.org//i. 10. 111111.2444933 /tt99963/jcaaaa http://iacaaa..2/33333 2023/396.
Dosovitskiy. A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Con- ference on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httph::/p///cc..... htt.lhttt////pnrreeeeee /ttpn///eeeeeeeeeeeeeeeeeeee net/ forum? id-YicbE FNNTy..
Dao. T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth In- terationnal Conference on Learning Representations, 2024. URL httph: http:////peeeeeeeeeeeeee http:/pewwwweeeeeeeeeeeeeeeeeeee id-mžn2Xyh9Ec.
Daubechies, I. and Hughes, S.
Graph theory, 2009. httptt::// /wbb math. princeton.edd/ math_a SechhallveeeSSe 5/otee11/ pdf.
Edmonds, J. and Johnson, E. L. Matching, euler tours and the chinese postman. Mathematical programming, 5: 88 124, 1973.
Deng, C., Yue, Z., and Zhang, Z. Polynormer: Polynomial- expressive graph transformer in linear time. In The Twelfth International Conference on Learning Represen- tations, ICLR 2024. Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //tp////eeeeee..eeee/ee/e Set//oormmiin hww.mmmmiddddmmmmma.......mm
Erdos, P., Rényi, A., et al. On the evolution of random graphs. Publ. math. inst. hung. acad. sci, 171): 1960.
Fey, M. and Lenssen, J. E. Fast graph representation learning with Py Geometric. In ICLR Workshop on Repre- sentation Learning on Graphs and Manifolds, 2019.
Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference Computer Vision and Patlern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEBE Com-
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: accurate post-training quantization for genera- tive pre-trained transformers. CoRR, abs/2210.1 17323, 2022. doi: 10.48550/ARXXIV22110 17323, URL https: 2210 17323.
puter Society, 2009. doi: 10.11099CVPR.22009552000 URL https: / /doi wwc...org11...0 5206848,
2009.
10

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for quan- tum chemistry. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML. 2017, Sydney, NSW, Aus- tralia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1263-1272. PMLR, 2017. URL http:/ / /ptppeeeeeiggsss.eeeeeeep.e...se mww. press/ v70/gi 1mer17a html.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W. Vinyals, O., and Sifre, L.. Training compute-optimal large language models. CoRR, abs/2203. 15556, 2022. doi: 10.45500ARXIV2203.55555 URL https: /doi. org 00lg/11.4855505 48550/855///00 wttp:///xxxv....0 15556.
Gloeckle, F., Idrissi, B. Y., Rozière, B., Lopez-Paz, D., and Synnaeve, G. Better & faster large language mod- els via multi-token prediction. In Forty-first Interna- tional Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024, OpenReview.net, 2024. URL ttttps::/operreeeet wwwp:rrvvieeeeeeforrrm.... id j11.
Hou, 7., Liu, X., Cen, Y., Dong, Y., Yang, H., Wang, C., and Tang, J. Graphmae: Self-supervised masked graph autoencoders. In Zhang, A. and Rangwala, H. (eds.), KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 18, 2022, pp. 594-604 ACM, 2022. doi: 10.1145753367883533922) URL https: h/dpi.org///l...// 11..0..111555333 35393321.
Grohe, M. and Schweitzer, P. The graph isomorphism problem. Communications of the ACM, 13811)1128-334 2020.
Grover, A. and Leskovec, J. node2vec: Scalable feature learning for networks. In Krishnapuram, B., Shah, M., Smola, A. J., Aggarwal, C. C., Shen, D., and Rastogi, R. (eds.), Proceedings of the 22nd ACM SIGKDD In- ternational Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13- 17, 2016, pp. 855-864. ACM, 2016. doi: 10.1145/ 2939672.2939754. URL attps://doi.oo 1145/2939672.2999 29397544.
Hu, W., Fey, M., Ren, H., Nakata, M., Dong, Y., and Leskovec, J. Ogb-Isc: A large-scale challenge for machine learning on graphs. arXiv preprint trXiv:2103.094300 2021.
Hussain, M. S., Zaki, M. J., and Subramanian, D. Global self-attention as replacement for graph convolution. In Zhang, A. and Rangwala, H. (eds.), KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining, Washington, DC, USA, August 14 - 18, 2022, pp. 655-665. ACM, 2022. doi: 10. 1145/3534678.5539299 URL https: /ttt::://dii.rre org/ 10. 01144/33336788 3539296.
Guo, C., Sablayrolles, A., Jégou, H., and Kiela, D. Gradient- based adversarial attacks against text transformers. In Moens. M., Huang, X., Specia, L., and Yih, S. W. (eds.), Proceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2021, Vir- tual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 5747-5757. Association for Com- putational Linguistics, 2021. doi: 10.18653/V1/2021.
Jin, W., Li, Y., Xu, H., Wang, Y., Ji, S., Aggarwal, C., and Tang, J. Adversarial attacks and defenses on graphs. SIGKDD Explor., 22(2):19 34, 2020. doi: 10. 1145/3447566.444566, URL httph::///// / //t..oog/////o//o 10. 11 00.11155344755..35 34475664.66.
EMNLP-MAIN.4E 464. URL https:::pp.........t...::t : add...oog/10.0 18653/v1 /2011/1/2222222 .ww.:02211mmmmmmmmmmmmmmmmmmmm
in. 464
Hagberg, A., Swart, P., and S Chult, D. Exploring net- work structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab,(LANL), Los Alamos, NM (United States), 2008
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint 2iti::2001083660 2020.
Hamilton, W. L., Ying, Z., and Leskovee, J. Inductive representation learning on large graphs. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pp. 1024-1034, 2017.
Karypis, G. and Kumar, V. Metis: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices. Technical report, University of Minnesota Twin Cities, Department of Computer Science and Engincering, 1997.
Kim, J., Nguyen, D.,, Min, S., Cho, S., Lee, M., Lee, H., and Hong, S. Pure transformers are powerful graph learners.

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Advances in Neural Information Processing Systems, 35: 14582-14595, 2022.
Luo, Y., Shi, L., and Wu, X. Classic gnns are strong base- lines: Reassessing gnns for node classification. In Glober- sons, A., Mackey, L.,, Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neu- ral Information Processing Systems 38: Annual Confer- ence on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10- 15, 2024, 2024.
Kipf, T. I.. and Welling, M. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: // openreview. beenreview.net/formne formm 1i= L0-5
Ma, W., Wang, Y., Wang. X., and Zhang, M. Recon- sidering the performance of GAE in link prediction. CoRR, 0as/2411.03884 2024. doi: 10.48550/ARXIV. 2411.03845. URL https: http://ti.ooggg..../o 48550/ 03845.
Knyazev, B., Taylor, G. W., and Amer, M. R. Understanding attention and generalization in graph neural networks. In Wallach, H. M., Larochelle. H., Beygelzimer. A., d' A'Ahéé Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 4204 4214, 2019.
Masters, D., Dean, J., Kläser, K., Li, Z. Maddrell- Mander, S.,, Sanders, A., Helal, H., Beker, D., Fitzgib- bon, A. W., Huang, S., Rampásek, L.,, and Beaini, D. GPS++ reviving the art message passing for molecu- lar property prediction. Trans. Mach. Learn. Res., 2023, 2023, URL httptttt////ppttteee /ttp:////pp/eeeeee///m//mee//eem net. fertierummtt/ forum?
Kong, K., Chen, J., Kirchenbauer, J., Ni, R., Bruss, C. B., and Goldstein, T. Goat: A global transformer on large- scale graphs. In International Conference on Machine Learning, pp. 17375 17390. PMLR, 2023.
i id=moVEUgJaHO.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q. (eds.), Advances in Neural Information Process- ing Systems 25. pp. 1097-1105. Curran Associates, Inc., 2012.
Mei-Ko, K. Graphic programming using odd or even points. Chinese Math, 1:237-277, 1962.
Midjourney, I. Midjourney, 2023. URL https: /www. midjourney. com.
Min, E.. Chen, R., Bian, Y., Xu, T., Zhao, K., Huang, W., Zhao, P., Huang, J., Ananiadou, S., and Rong, Y. Transformer for graphs: An overview from architec- ture perspective. CoRR, abs/2202.008555, 2022. URL 08455.
Li, G., Xiong, C., Qian, G., Thabet, A. K., and Ghanem, B. Deepergen: Training deeper gens with general- ized aggregation functions. IEEE Trans. Pattern Anal. Mach. Intell., 1014-:3333244344 2023. doi: 10.1109/ 2PAMM.2023.330000
https:/ http:///aaxvv.../aassoeoeo
URL https: http:://dco..oooee 1109/TPAM/ 2023. 0.002...30630..
Mnih, A. and Salakhutdinov, R. R. Probabilistic matrix fac- torization. Advances in neural information processing systems, pp. 1257-1264 2008
Liu, 7.,, Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference
Müller, L., Galkin, M., Morris, C., and Rampásek, L. At- tending to graph transformers. Trans. Mach. Learn. Res., 2024, 2024. URL https: http://ppnreveeee renerrvview.... net/ forum?id-Hhbqiiiee
Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9992-10002. IEEE, 2021. doi: 10.1499/1 2021.00986. URL https:://// htt///i..orgg1..111 1109/ ICCV48922 22..222.22111 00986.
Nakata, M. and Shimazaki, T. Pubchemqe project: A large- scale first fiss-princippls electronic structure database for data- driven chemistry. J. Chem. Inf. Model., 1300--3880 2017. doi: 10.1002/acs.ccim..bbb 7b00083.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019. URI. https: //openreview net /free/ffrruee forum? d=Bkg 6R1CQY7.
URL https:
/ waa.i.org//a. 1021/ace acs. jaim. 7b00083.
OGB Team. Dataset ogbl-ppa. https: /tops://ogb.. stan edu/ doca/
edw/doca/ pocoss/lnnkkpee wwikpropn htoblloollllooo -ppa,
Luo, S., Chen, T., Xu, Y., Zheng, S., Liu, T.-Y., Wang, L., and He, D. One transformer can understand both 2d & 3d molecular data. In International Conference on Learning Representations, 2023.
2020. Accessed: 2025-05-20;
Open-Al. Chatgpt. https: // chat owe.ti.ppna....mm com/ 2023.
12

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-277 277444, 2022.
Ireland, October 19-23, 2020, pp. 3125-3132. ACM, 2020. doi: 10. 10.1145/3340531.341225 URL https: wtt..o////...111/344333333311
Rusch, T. K., Bronstein, M. M., and Mishra, S. A survey on oversmoothing in graph neural networks. CoRR, abs/2303.10992 2023. doi: 10.48550/ARXIV, 2303.10993. URL https:: a arxiv. 2303. 10993.
Park, W., Chang, W.-G., Lee, D., Kim, J., and seung- hwang. GRPE: Relative positional encoding for graph transformer. In /CLR2022 Machine Learning for Drug Discovery, 2022. URL https: //pppreveiieeeee//ee//e/eeeeeeee net/ Ceooummid-GNAARRR.
48550/
Sennrich, R., Haddow, B., and Birch, Neural machine translation of rare words with subword units. In Pro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10. 1/66-1111 URL https://doi.co 10. 1t..18633vvl1 201553/v1p1666/ 6-1162.
Perez, L. and Wang. J. The effectiveness of data augmen- tation in image classification using deep learning. arXiv preprint ar arXiv:111..0462) 2017.
Qiu, J., Chen, Q. Dong, Y., Zhang, J. Yang, H., Ding, M.. Wang. K., and Tang. GCC: graph contrastive coding for graph neural network pre-training. Gupta, R., Liu, Y., Tang, J., and Prakash, B. A. (eds.), KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp. 1150-1160. ACM, 2020. doi: 10. 1145/3339448.33033666 URL. https://///. /wt.doi.orr/////////oo/... 10. N0.1145/339442 3403168.
Shao, R., Shi, Z., Yi, J., Chen, P., and Hsieh, C. On the adversarial robustness of vision transformers. Trans. Mach. Learn. Res., 2022, 2022. URL https:// openreview net fow.mforuiioooooo 1dd=1114411E1kk
Shi, I.., Hu, B., Zhao, D., He, J., Zhang, 7., and Zhou, J. Structural information enhanced graph representation for link prediction. In Wooldridge, M. J., Dy, G., and Natarajan, S. (cds.). Thirty-Eighth AAAI Confer- ence on Artificial Intelligence, AAAI 2024, Thirty-Sixte -ixth Conference on Innovative Applications of Artificial In- telligence, IAAI 2024, Fourteenth Symposiun Edu- cational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 14964- 14972. AAAI Press, 2024. doi: 10. 10.1609/AAAI..3811 29417. URL https: www.do.orgogoooooo 16t.:/....661000 1609/ aaaii. 118113.29417
Radford, A., Narasimhan, K., Salimans, T., and Sutskever,
I. Improving language understanding by generative pre- training, 2018.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.. et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551, 2020.
Shi, Y., Zheng, S., Ke, G., Shen, Y., You, J., He, J., Luo, S., Liu, C., He, D., and Liu, T. Benchmarking graphormer large-scale molecular modeling datasets. CoRR, abs/2203 004800 2022. doi: 10.48550/ARXIV. 2203.04810. URL https: arXiv 2203. 04810.
Rampásek, L., Galkin, M., Dwivedi, V. Luu, A. T., Wolf, G., and Beaini, D. Recipe for a general, powerful, scal- able graph transformer. In NeurlPS, 2022.
Shirzad, H., Velingker, A., Venkatachalam, B., Sutherland, D. J., and Sinop, A. K. Exphormer: Sparse transform- ers for graphs. In International Conference on Machine Learning, pp. 31613-31632. PMLR, 2023.
Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deep- speed: System optimizations enable training deep learn- ing models with over 100 billion parameters. In Proceed- ings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Shoeybi. M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. CoRR, http//999000053. 2019. URL http: 909 ..00000000 08053.
Rozemberezki, B., Kiss, O.,, and Sarkar, R. Karate club: An API oriented open-source python framework for un- supervised learning on graphs. In d' Aquin, M., Dietze, S., Hauff, C., Curry, E.,, and Maureox, P. (eds.), CIKM 200 The 29th ACM International Conference on Information and Knowledge Management, Virtual Event,
/ l/arxiv.ogg/ablllb.
Sun, C., Zhang, M., Hu, J., Gu, H., Chen, J., and Yang, M. Adaptive graph diffusion networks: com- pact and expressive gnns with large receptive fields.
13

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Artif. Intell. Rev. 58(4):107, 2025. doi: 10.1007/
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jer- nite, Y.. Plu, J., Xu, C., Scao, T. L.. Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Compu- tational Linguistics. URL https: htww://wwwc/mccoo/ooo aclweb. org/ aaaaantooloooo ogy/ 2020 emn 1 -epdd-.s..... 6.
$10462-025-111144 URL https:tttt////....m /tt//oo.ooooo org//0. 1007/s10462
225-11114-22
Sun, I.,, Dou, Y., Yang, C., Zhang, K., Wang, J. Yu, P. S., He, L., and Li, B. Adversarial attack and defense on graph data: A survey. IEEE Trans. Knowl. Data Eng., 35(8):
7693-7711, 2023. doi: 10.1109/TKDEE.022.290000 URL attpss://ost http://dt..oo//..///// httpg//1..110330 3201243.
2022.
Szklarczyk, D., Gable, A. L., Lyon, D., Junge, A., Wyder, S., Cuerta-Cepass J., Simonovic, M., Doncheva, N. T., Morris, J. H., Bork, P., et al. String v11: protein-protein association networks with increased coverage, support- ing functional discovery in genome-wide experimental datasets. Nucleic acids research, 47(DI):D607-D61 2019.
Wu, Q.. Zhao, W. Li, Z., Wipf, D. P., and Yan, J. Node- former: A scalable graph structure learning transformer for node classification. Advances in Neural Information Processing Systems, 35:27387-27401, 2022.
Wu, Q. Zhao, W., Yang. C.,, Zhang, H., Nie, F., Jiang, H., Bian, Y., and Yan, J. Simplifying and empowering transformers for large- graph representations. Advances in Neural Information Processing Systems, 36, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13977, 2023.
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Ge- niesse, C., Pappu, A. S., Leswing, K., and Pande, V. S. Moleculenet: A benchmark for molecular ma- chine learning. CoRR, abs/1703.00564, 2017. URL http:://axxiv.org/babe. 1701/.10...00 00564.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems, pp. 1998-6008, 2017.
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Philip, S. Y. comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning sys- tems, 32(1):4-24, 2020.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: //ppereeee.....e net/ forum? 1
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How pow- erful are graph neural networks? In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpt::/p////....... /oppneeveeeeeeeeeeeeeeeeeeeeeeee net/ forum? id id-ryGs6iA5k)
Wang, H., Fu, T,, Du, Y., Gao, W., Huang, K., Liu, 7 Chandak, P., Liu, S., Katwyk, P. V., Deac, A., Anand- kumar, A., Bergen, K., Gomes, C. P., Ho, S., Kohli, P., Lasenby, J., Leskovec, J., Liu, T., Manrai, A., Marks, D. S., Ramsundar, B., Song, L.., Sun, J., Tang, J., Velick- ovic, P., Welling, M., Zhang, L.. Coley, C. W., Ben- gio, Y., and Zitnik, M. Scientific discovery in the age of artificial intelligence. Nat, 620(7972)477-60, 2023. doi: 10. 1000000484186666 100.8/4441156-002--022222222 URL https:tpt.. //doi. org /00. /0. 1038 / 10044411666000 00500//4110003-00005530030 02175588662202--2222 -22-02221-222222-----2---22222
Yang, M., Feng, W., Shen, Y., and Hooi, B. Towards better graph representation learning with parameterized decom- position & filtering. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Inter- national Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 39234 39251. PMLR, 2023. URL https: hattps://proceedinng mir www.prrsssseee v202 www.yannnnnnnm.mm
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: http///perreeeeeeeeeeee ew, n forum? id Ed=gEZrGGozdqR.
Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., and Liu, T.-Y. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 011288877-8888 2021.
Zeng, H., Zhang, M., Xia, Y., Srivastava, A., Malevich, A., Kamnan, R., Prasanna, V., Jin, L., and Chen, R. De- coupling the depth and scope of graph neural networks.
West, D. B. et al. Introduction to graph theory, volume 2. Prentice hall Upper Saddle River, 2001.
14

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Advances in Neural Information Processing Systems, 34: 19665-19679, 2021.
Zhang. J., Zhang, H., Xia, C., and Sun, L. Graph-bert: Only attention needed for learning graph representations. CoRR, abs/2001 ,05140, 2020. URL https: //axxva...vv org/abs/ 2001 40.
Zhang, L., Yan, X., He. J., Li, R., and Chu, W. DRGCN: dynamic evolving initial residual for deep graph convolu- tional networks. In Williams, B., Chen, Y., and Neville, J. (eds.). Thirty-Seventh AAAI Conference Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference Inno- vative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Ar- tificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pp. 11254-11261. AAAI Press, 2023. doi: 01111/9//AAALL33333 r
URL https:
Zhang. M. and Chen, Y. Link prediction based on graph neu- ral networks. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, (eds.), Advances in Neural Information Processing Systems 31: Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Mon- tréal, Canada, pp. 5171-5181, 2018.
Annual Conference
Zhang, M. and Li, Nested graph neural networks. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Infor- mation Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual. pp. 15734-15747, 2021.
Zhang, M., Li, P., Xia, Y., Wang, K., and Jin, L. La- beling trick: A theory of using graph neural networks for multi-node representation learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 1021, virtual, pp. 9061-9073 2021.
Zhou, T., Lü, L.. and Zhang, Y.-C. Predicting missing links via local information. The European Physical Journal B. 71:623-66 630. 2009.
15

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
A. Datasets
The detailed statistics the datasets are in the Tab,
Table Statistics graph-/odge-/nodde datasets. Here "BC" stands for binary classification, p in Random Graph datasets means the probability of creating the edge between node pair.
datasets
# of graphs
avg # of nodes
avg # of edges
task-type
metrics
PCQM4Mv2
3,746,619
14.14
14.56
regression
MAE
ogbg-molpcba
437,929
26.0
28.1
multi-label BC
AP
reddit-threads
203,088
23.9
24.9
BC
ROC-AUC
Triangles
45,000
20.9
32.7
multi-class classification
ACC
Internal dataset
3,100,000
24.8
54.7
N/A
N/A
Random Graph
3, 3,100,000
67.1
74.8
N/A
N/A
oghl-ppa
576,289
30,326,273
BC
HR@ 100
ogbl-citation2
2,927,963
30,561,187
BC
MRR
ogbn-proteins
132,534
39,561,252
multi-label BC
ROC-AUC
ogbn-arxiv
169,343
1,166,243
multi-class classification
ACC
A.1. Subgraph Sampling
The subgraph sampling configurations for different datasets of large graphs are shown in the Tab. 10.
Table Details subgraph sampling oghl-ppa and oghn-proteins datasets. 'seq-len' means the average length of the Fulerian sequences. Edge-ego means sampling subgraph around the central edge, and Node-ego means sampling around the central node.
dataset
sampling
depth (d)
# neighbors (ni
seq-len
14
90
ogbl-ppa
edge-cgo
30
280
14
60
ogbl-citation2
edge-ego
20
90
30
130
20
20
50
tooteproteins
node-ego
120
60
200
30
30
ogbn-arxiv
node-ego
40
40
B. Models
We list the model specifics in the Tab. 11. We experiment with eight different scales of models.
C. Implementation Details
C.1. Graphs to Sequences of Tokens
The implementation uses Py Torch as the primary framework. For graph preprocessing tasks such as subgraph sampling, we utilize corch-geometics (Fey & Lenssen, 2019). When required, we employ NetworkX (Hagberg et al., 2008) to paths. custom tokenizer converts these paths into token sequences, with
Eulerize (sub)graphs and identify dataset-specific vocabularies constructed for each case.
16

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Table 11. Statistics of GraphGPT models of different sizes. The GraphGPT-Base of the scale Bert-Base (Devlin al., 2019).
Model-size
Hidden-size
# of layers
# of heads
Params (excluding embed)
Mini
256
4
4
4.2M
(Small)
512
4
8
16.8M
M (Medium)
512
33.6M
B / B12 (Base)
768
12
12
113.2M
B24 (Base24)
768
24
12
226.5M
Bas (Base48)
768
48
12
453.0M
I. (large)
1024
24
16
402.7M
XXI. aagg)
1600
25
2.0B
C.2. Model Backbone
We employ a transformer architecture based on Llama (Touvron et al., 2023), implemented via the Hugging Face Transform- ers library (Wolf et al., 2020), as the backhone for NTP pre-training. For SMTP pre-training, we modify the architecture by replacing the causal attention mask with a bidirectional attention mask to create an encoder. We initialize parameters randomly, and train models various scales (see Table 11).
C.3. Training
The models are pre-trained and fine-tuned on A800-80G GPU clusters using DeepSpeed's Stage-2 strategy with mixed precision (FP16/FP32) or BF16 (Rasley al., 2020). We employ the AdamW optimizer (Loshchilov & Hutter, 2019) with learning scheduler. To maximize computational efficiency in pre-training stage, we pack multiple graph sequences into single entries, optimizing context window utilization (Raffel et al., 2020) for certain graph datasets. Dataset-specific configurations are detailed in their respective sections below.
The variance inherently low for most large-scale datasets (e.g., PCQM4Mv2, ogbl-ppa), where reporting variance is standard practice only when significant. For these datasets, 3-5 runs consistently yield minimal variance (as shown in tables). For the Triangles dataset, variance partica-prticularlly on out-of-distribution (OOD) test data. So we conducted 10 runs to ensure robustness.
The pre-training and fine-tuning paradigm is illustrated in Fig. 2.
Pre-training with NTP
Fine-tuning
Pre-training with SMTP
Fine-tuning
15
17
<005>
heaph-/nodeeeddg---v tasks
<eos
graph-noode/eddee-VV tasks
Trans remmer Decoder
Transfor Fremerrnerrrr E Encoder
13
14
15
16
14
16
17
<COS>
[m]
[m]]
15
[m]
[m]]
[m]
17
18 [m]
13
16
>
14
15
17
<eos>
<gsum
13
14
14
16
14
14
15
16
17
18
<eos>
14
13
14
15
14
16
17
<208>
13
14
15
16
16 17
18
<eos>
16
P2
Ps
P6
Pr
Ph
Po
Pro
PE
Po
Ps
P4
P4
P4
Pso
Figure 2. Pre-training and fine-tuning illustrations.
We also utilize clusters other types of GPUs, for example, Nvidia V100-32G, I.20, I.40 etc.
17

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
C.4. Vocabulary
In NLP, the vocabulary is typically constructed by tokenizing text data using the byte-pair encoding (BPE) algorithm (Sennrich et al., 2016). The resulting unique tokens form the vocabulary, which usually comprises frequent subwords from the text corpus.
In contrast, our GraphGPT employs a fundamentally different vocabulary construction approach. The vocabulary is split into two distinct parts: 1). structural and special lokens, which are dataset-agnostic and transferable across different datasets. 2). semantic tokens, which encode -ptasct-specific information, such as node and edge attributes.
An example is provided in Appendix F. In the graph sequence: i). tokens like '1', '2', etc., represent structural tokens; it). Tokens such as hogbl-ppathode#0#117 and logbl-ppa#node#) #1959' are semantics tokens; iii). Special tokens like <gsum>> and <cos>> (Fig. 1) denote graph-specific functions (c.g., graph summary and und-of-sqquena markers).
For the datasets ppa/cipp/cittiin........ ogbn-proteins/arxin we set & = 2 (S2.2.3), resulting in vocabulary sizes of 41, 634 25, 687 and 31, / 25, 600, respectively.
D. Graphs to Sequences of Tokens
In this section, we show some examples of turning graphs to sequences of tokens.
D.1. Molecular Graphs to Tokens
Below one example of 2D molecular graphs in the ogbg gool-baopebb dataset in torch- gecce-eommtricc data format (Fey & Lenssen, 2019).
Data (x=[4, 9]. edge_index=[2, 6]. edge_ attr= 31, y=[128])
The graph has 4 nodes and 3 edges. The source and destination nodes of the edges recorded in "edge_index" and its dimension is (2, 2 number_oo _f_f________ of_edees) for undirected graphs. 'x' is the node attributes of dimensions, and 'edge_attr stores the edge attributes of 3 dimensions.
The node and edge attributes of the graphs are numbers. If we directly discretize them into tokens, i.e., using one token to represent each unique number, the numbers that appear few times in the dataset cannot be well-trained. the same time, the vocabulary may blow up. Therefore, we split them into single digits and represent them with the combination of the following tokens. They are dataset agnostic, and can be shared across different datasets.
<->, <.>, <0>, <1>, <2>, <3> <4>,
<6> <7>, <8>, <9>
The resulting vocabulary is 556 for both molppaa and PCQM4Mv2.
Below shows the tokens from one of the possible (semi-) Fulerain paths of the above molecular graph.
[',,,
'<7>', (gbg-mollpba+foode#221
'<l>', 'ogbg- '<>>' "ogbg-molpebatedge
molpcba#node+3#11. '<5>', "ogbg- gebg-mmopcbaannde##66 #0#1', '<l>', '2', '3', ogbg-mollpbaanodee#tre 1 <5>'; mobgg-molpcaa#node #2#1', '<4>', gggg-mollpbaa#node#55 ggbbsmolpcbaa#ooee#HH
'<5>'. "oggg-molpcbaa hoge--moopcbaanoee@aeee
'<3>',
'<2>'. '2' "ogbg- #00g '<5>' ooggg- 22 '<3>' "ogbg- gebg--molpbbaandde333" #6#1', '<l>'. '4' gebgsmoopobathode+00t '<5>' mgbg=molppbafnode#2211 '<4>', 'ogbg- holg--moopccaatoeesHttt molp #6 <<>>]
'<5>' "ogbg-molpcbathooee
'<>>' 'ogbg " '<3>', 'ogbg-
In the sequence of tokens above, for the node '1', can deduce that its 9 dimensional attributes are (7,0, 0, 0,000, Node 'I' is connected to '2' with edge attributes (1, 0). We set 0 as the the default value of the attributes in this dataset, and do not encode it into tokens.
In the (sumi-)Eulerian path, a node may appear several times. We append its attributes tokens to one of its appearances randomly. This can prevent the model from copying the attributes from the previous appearance, and also shorten the resulting sequence.
18

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
For graph obtained from Eulerization, an edge may present several times in the path. We apply the same logic insert the edge attributes tokens.
As in the above sequence, node '2' appears two times, and its node attributes tokens are appended after its second appearance. There is no tokens encode the edge attributes of edge between '2' and '3', which implies the edge attributes are default value (0, 0, 0).
D.2. Subgraphs to Tokens
In edge/node-level tasks, we usually have one big graph. In this section, we use ogbl-ppa and how to sample subgraphs from the big graph, and then transform the subgraph to sequences of tokens.
datasets to show
The whole ogbl-ppa dataset is summarized torch-geometric format as follows.
Data [num_ _ed___odessss = =762889, edge edge_index index [22. 42463862], x=[576289, 58])
has 576289 nodes and 21231931 edges in the training data. "x' is the one-hot representation of the species that the node (protein) belongs to.
We sample a subgraph from it as below.
Data ( [num_nodes moodes=0,, root_n_ine =[2]. lige_index==s 84], x=[30, 21)
has 30 nodes, 42 edges as in 'edge_index'. 'x' is the node attributes of dimensions, and encodes the node identity as described in Sec. 2.2.3. We partition the nodes (proteins) based on the associated species. The number of proteins inside each species varies from 616 41017. Finally we use 58 tokens for species and 41017 tokens for the local indices. Combined with the tokens for the structure and the special tokens, the total vocabulary is 41231.
Here 'root i_id id' records the two seed nodes, and the subgraph is sampled centered around them. The resulting tokens from one of the possible (semi-)Eulerian paths are:
['1. ' '3' -p #0#1' "pg #1 '4' '5', -ppa #node#0#17', "gbb-ppa+trod+442444 spa#node#1#3566 '6', '8', "pgal-ppa#node #0#17', logbl-ppa#nee # 44
'6', '7' -pbbl-ppa#nod #0#17 'ogbl-
'6', '9', 'oghl -o
'oghl node '6', oobl- "ogbl-ppa#node
'10'
#0#27', 1
#1#6850', '11' "o
'6', '6','12,
ogbl pgbl-ppaadooe
'6', '4' 'ogbl -pa
"ggbl-ppattnood+#1111133
' '13' "ogbl=ppa= 'ogbl-ppal
'ogbl-ppa#indeo0#77 "ogbl=ppaanode++19977"" '4' ogbll 1 '2'
#pa#nodc#0#17". "ggbl-ppaaHoode#HHH166 '5 -gg.b---aaaaanoooooo111 oppal-ppa
#node#1#8368 '2' '6'
o '1', '18"
'6', '17'
'10'.
ogbl-ppa#nod
ogb '6' '19' ogbl-ppa#node
#0#17', - saatlppppanooch111557111 2 '20',
ogbl-ppa#nod #0 oppll
#07e#1#7761 '2' 21
-pd #0
#0#17' "oghi-ppastoodeett88833
'2'
ppa#node11#16222 o "ogb 1
'22', 'ogbl ogbl-
'2' '23' 'ogbl-ppa#
#0de#'#17'_ oogl
'2' '24' 'ogbl-ppa#hnodeo##11 oggll
'6' '25', #001
node#0#17 'ogbl holll-ppppnooetttttppoe 0g "ggbl=ppa+ttode+++119555 '2',
'27' 'ogbl-ppatt nope #0# opbl--pa#odd
'19', '25'. '2', '26',
'6' '28' 'ogbl-ppatt
'6', '29' 'ogbl ogbl--paafnode0##77 'ogbl-
noa# # '6', '30' ggbl-ppaannde+###17 "ogbl 1#19700" -pa# #1#204741
'6', '27' '1' 'ogbl mood-ppa#ne
In the ablation study on node identity encoding in Sec. 3.5.3, an example of the subgraph sampled from ogbl-ppa without identity encoding is shown below.
Data (num_nodess= n0des=30. root [oor___id=(2) id= =22 136), x=[30, 1))
19

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Different from the subgraph with node identity encoded in 'x', its node attribute 'x' contains only the information of the node's (protein) hosting species. It cannot be used to uniquely identify the nodes. The vocabulary decreases from 41231 to 214.
The resulting tokens from one of its possible
paths is below.
[' ' '3', '4', -pa #0 '5', '6 '7, '5' '8' '9', '1'
ogbl -pea #0#17', '10', "ogbl -paatnoddd
#0= #0#
111" 'ogbl poadnode #0#17'
'3' oggl ppal=paaaodooo0117 '11" '12", ' '5' 113" 'oobl -paa#node #0#17',
'5' '14". -ppa #0#17', '5' '9' 100, ' ppp #0##7
3', '15", =ppl- '3' '16", "ggbl-paa#ode=0#117
'3' '2',
-ppl--pan =0020= #0#20' '17' 'ogbl ppabll-ppa#oo #0#27", 1 '18' "ogbl
#0#20" '1' '19 00bl ' "9" "obbl-ppttno==+4117
'20',
10 3' '11', '3 '5' 10" '12 oogbl ppal
node#0#17' ' '22". '13" '24'. '5' '25', -ppbl
'3
'17', '18 '3' '23
'23', "pbll-pa=#od++0##11
'21', -ppal-ppa# #0#17' '20'. '5' '66'. ogbl--paatndee###11 '5' '22', '24', oogbl nog '28', "ogbl-ppaannde '7', ooghl -pat-oode '28", '5" oogl- #0#17', 55 '30', 'ogbl =00 '5' '19, '5', '12', '2'', '1]
'23'. '5', '27', '6' "gbl--ppa+oooc+0#11
node#0#17', .
'29',
In the following, we use the ogbn-proteins dataset as the example. The entire dataset is a large graph as below.
Data ( num ham___odd==322344 dige_index=12, 79 79122504]. edge_ attr tuttr=[[77 [19 (791225044 8], (132533, 1]. y y=[132534, 112))
It has 132,534 nodes and 39,561,252 edges. ooeccsspecess stores the species' numeric id that the node (proteins) belongs to.
One sampled subgraph in the torch-geometric data format is:
Data maum_nodes- = 10, root_n_id=0, edge_index =[2, 22], edge_attr= 8]. y=110, 112], x=[10, 21)
It has 10 nodes, 11 edges as in edge_index' Edge attributes is stored in 'edge_attr" aatte_at'''oe of dimension 8. 'x' is the node attributes of 2 dimensions, and it encodes the node identity as described in Sec. 2.2.3. Its first dimension (token) represents the species, and the second is local numbering of each protein inside its species. Similar to the ogbl-ppa dataset, the identity encoding of 132,534 nodes occupies 25, 56,65 tokens in the vocabulary, and the total vocabulary is 25.620.
'y' records the labels for the supervised node-level task. 'root_n_id" represents the target node, and the subgraph is sampled centered around it.
The resulting tokens from one of the possible (almii) paths are follows.
['1, honee-pprrteinssoooooootee
(ggh 'ogbn-proteins: 'ogbn-
edge#7#1', '<1>', '<6>' '<4>' ' oroeh proteins#node #1#6896 tood
sedd-ppootinnstce#####t '<3>' '<4>' '<0>' '3',
: 'ogbn-proteinstedge "<5>' '<1>' '<''' '<3>', '5'," oggnn
#4#1 "4#1' '<3>' '<タ>', '<8>' '4', 1 Oggnn-proteinnscccee#4:: proteins#node #0#3702 'ogon-prooteinssHooee4444333333
#04300 'ogbn-proteinst
'ogbn-proteinn #ecer###edd
'<2>', '<l>' '<>>' '6', '7', 'ogbn <4>' '<l>' '<8>', '8', 'ogbn- tnd oghn-proceinsshooeellll '7', p hoobn-ppoocnnsmoo+o0000000 1ode#1#89'
'<4>' '8>>, "ggbn-proteinnsfeeee##11 -proteins#edge+7*1 '<>>'
'ogbn-proteins# '3>', '<2>', <<>>. '6', 'ogbn-
oggnn-proeinnstcdge+7 <<2>', '<>>' '0>'' '9', oog #0#3702 "ogbn-protein
proteins#nod ogbn
20

---

To reduce the number of tokens in the resultant sequences further, we multiply the number with 1000 and then minus it by 1. GraphGPT: Generative Pre-trained Graph Eulerian Transformer
#1#7055', aggn-proteiins#edeeeee '<l'', '<6>', '<>'' '10', "ogbn-proteing #node#0#3702- Loghn-proted
'ogbn- proe- '<>', <6>', '<">' '4', ' "ogbl-protteinsecdee##l '<2>', '<0'' <<>', '3']
The original edge attributes are 8-dimensional vector decimal numbers from 0.001 to 1. We split them into single digits and represent them with the combination of digits tokens as in App.
So do not need to encode "!' any more. At the same time, we treat the value 0.001 (0 after the above transformation) the default value and do not encode it with tokens.
E. Graph-Level Task
E.1. PCQM4M-v2
SMTP pre-trained loss vs #Params (non-embedding)
Supervised fine-tuned loss #aarams (non-embeddling
x
2.8 x 10-1
Valld loss
Valid loss
2.7 x 10
x.6
2.5 x 10-1
8 10
109
10
10a
10"
10a
10a
Parameters (non embedding)
Parameters (non-embedding)
Figure 3. Log-log plot of pre-training loss and supervised fine-tuning loss versus the number of non-embedding parameters for the BaSSSSallMMdiimmBBa/BBBBBaaa model configurations (sce Table 11) on the PCQM4M-v2 dataset.
The pre-training and tin-ttnnig configurations for PCQM4M-v2 are in Tab. 12. The log-scale scaling law plot for both pre-training loss and supervised fine-tuning loss is shown in Fig. 3.
F. Eege-Level Task
F.1. ogbl-ppa
We use two tokens for the node identity encoding introduced in Sec. 2.2.3. Specifically, we use the species to partition the nodes, so the first token represents the species, and the second is the local indices of proteins inside each species.
The pre-training and fine-tuning configurations for ogbl-ppa are listed in Tab. 13. The loss of pre-training versus the number of tokens is shown in Fig. 4.
The fine-tuning data consists of subgraphs induced by the positive edges for training and equal negative edges randomly sampled.
In general, a larger model results in lower pre-training loss, and better results down-stream fine-tuning tasks.
F.2. oghl-citation2
The pre-training and fine-tuning configurations for ogbl-citation2 are listed Tab. 14. The subgraph sampling is edge-ego with and n 14/30 as in Tab. 10. The pre-training losses the ogbl-citation2 dataset with different model sizes and subgraph sampling settings are shown Fig. 5.
21

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
OGBL-PPA Pre-training with SMTP
mini
small
1.8
medium
base
1.6
large
xxlarge
Pre-training loss
1.4
1.2
0.4
5
10
15
Billion of tokens
Figure 4. fre-train loss versus tokens of ogbl-ppa dataset for models mani//mm///mmcchnn//ss///rrrruaa in Tab. 11.
OGBL-CITATIONZ Pre-training with SMTP
4.5
medium-d1n14
medium-d1n30
base-d1n14
base-d1nn3
Pre-training loss
3
2.5
1.5
1
0
5
10
15
20
Billion of tolens
Figure 5. Pre-train loss versus tokens of ogbl-citation2 dataset for models medium/bain as in Tab. 11.
22

---

model-size GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Table 12. Pre-train and fine-tune configurations for the PCQM4M-v2 dataset. L.SI means layersscale-mittaaizaiion, EMA is exponential moving average, MPE stands for max max-posiiee maax-poitionn- embedding, position-embeddingg and TWE means tie-wort wwod-embeddding
pre-train
fine-tune
Mini Small Medium Base Base24 Base48
batch-size
1024 //0////12//122/////
/1////////////1999// 1/1/1/1/4/4 10 tokens
1024
total
32 epochs
warmup
10 tokens
9.6 epochs
scheduler
Warmup linear decay
Warmup cosine decay
max-Ir
3 x 10
6/6/ 6 /////////111/ 10
min-Ir
automatic [0.9, 0.99] x 10 10 I 0.02
Adam-betas
[0.9, 0.95
Adam-cps
x 10-6
max-grad-norm
weight-decay
0.1
attention-drope dropout
0.1
path-dropout
0/0/0/0.05/0.1//.2
emhed-dropout
mip-dropout
LSI-val
NA
EMA
NA
hidden-act
gelu 1024
MPE
TWE
FALSE
NA
G. Node-Level Task
G.1. ogbn-proteins
The configurations of pre-training and fine-tuning in Tab. 15. The subgraph sampling is node-ego with d 20 and 1 as in Tab. 10. The node identity is encoded with two tokens similar to the ogbl-ppa in Sec. 3.3 (see App. F for details).
G.2. ogbn-arxiv
The configurations of pre-training and fine-tuning in Tab. 16. The subgraph sampling is node-ego with 1 and 40 as in Tab. 10. The node identity is encoded with two tokens.
H. Question and Answering
QL. Evaluating GraphGPT on real-world citation networks (e.g., PubMed, Cora) or social networks (e.g., Twitter, Facebook graphs) could be great.
AI. We evaluated GraphGPT on large-scale real-world citation networks: ogbn-arxiv (169K nodes, 1.17M edges) and ogbl-citation2 (2.93M nodes, 30.6M edges). These datasets are significantly larger than traditional benchmarks like Cora (2.7k nodes, 5.4K edges) and PubMed (19.7K nodes, 44.3K edges), aligning with our focus on scaling to massive graph data.
chose these datasets because GraphGPT's performance benefits from large-scale pre-training data to learn inductive biases (c.g., node permutation invariance). For instance, pre-training on the small Triangles dataset (45K graphs) yielded poor fine-tuning results (32.6%), whereas scaling pre-training data improved performance to 99% (Section 3.2.1). This mirrors the trend in Vision Transformers (ViT), which outperform CNNs only with sufficiently large datasets (Dosovitskiy et al., 2021).
While GNNs may outperform GraphGPT on small datasets like Cora or PubMed, our goal is to demonstrate scalability for large-scale graphs a critical challenge in modern applications.
Q2. While GraphGPT enables lossless and reversible graph-to-seq transformation, how well does do this in
23

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Table 13. The Pre-training and fine-tuning configurations for the oghl-ppa dataset. For XXL. model, use fp16 in the pre-train stage, and bl16 in the fine-tune stage for numerical stability.
pre-train
fine-tune
model-size
Mini Small Medium Base Large LXLarge
batch-size
1024
8192
total
x 10 tokens
8/8/8/8/16/ 16 epochs 2.4/2.42..422.4/4.. 4.8 epochs
warmup
10 tokens
Ir scheduler
Warmup & linear decay
Warmup & cosine decay
max-Ir
3 x 10
3/3/3/3/3/2 10 automatic
min-Ir
0
Adam-betas
[0.9, 0.95
[0.9, 0.99]
Adam-eps
1 x 10
1 x 10
max-grad-norm
5
weight-decay
0.1
attention-dropout
0.1
path-dropout
0/000/0/0.1/0.1
0/00///..05/0.1/0.2
embed-dropout
mlp-dropout
LSI-val
NA
NA/NA/1/1/1/NA
EMA
NA
hidden-act
gelu 1024
MPE
TWE
FALSE
real-world noisy graphs?
A2. While not the focus of this paper, we tested GraphGPT on an internal noisy graph dataset (3.1M graphs, avg. 24.8 nodes, 54.7 edges) for edge denoising.
Using sumi-super node classification task, GraphGPT achieved 10-20% F1 score improvement over baselines. We formulated the task analogously to Part-of-Speech tagging in NLP, leveraging token-level embeddings. The 'long' variant outperformed 'short' (see Fig. 1) likely due its edge-agnostic token embeddings of nodes.
Results were robust enough for online deployment.
Q3. What are the run-time comparisons with GNNs?
A3. We evaluated the run-time of GraphGPT versus GNNs using the PCQM4M-v2 dataset on single V100-32G GPU, The GNN baselines (adopted from Hu et al. (2021)) were implemented using the official GitHub repository
Results shown in Table 17. Run-time remains nearly constant across GraphGPT models ranging from 0.62M 33.95M parameters. This consistency stems from 10 bottleneck during CPU-based data preprocessing. This time-consuming preprocessing phase involves determining if graph is Eulerian, Eulerizing non- non-Eulerian graphs, and generating Eulerian paths.
Overall, GraphGPT's run-time is comparable to GNNs when model sizes are similar.
Q4. What's the computational cost of GraphGPT models?
A4. We have included computational cost details for the PCQM4Mv2 dataset in $4. The cost of other datasets are in the Tab. 18.
Q5. How robust is the model to adversarial graph perturbations?
A5. Adversarial robustness is a promising research area across NLP, CV, and graphs (Guo et al., 2021: Shao et al., 2022; Jin al., 2020; Sun et al., 2023). While not our primary focus, preliminary results on noisy graphs (Q2) suggest robustness through large-scale training. A deeper study would bridge GraphGPT's transformer architecture with adversarial graph
Phttps://gtt thub. com/
http://ppee//sccccc///e/m sc/ppcgmmmv2
24

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Table 14. Pre-train and fine-tune configurations for the oghl-citation2 dataset. We 6 in hoth the pre-training and fine-tuning stages for numerical stability, One epochs contains 10% randomly sampled positive edges and negative edges. For a given positive edge of head and tail node, we randomly sample node as the tail node, and then form negative edge with the head node.
pre-train
fine-tune
model-size
Medium Basc
batch-size
1024
4096/2048
total
2 10 tokens
32 epochs
warmup
10 tokens
9.6 epochs
Ir scheduler
Warmup & linear decay
Warmup & cosine decay 10-5 automatic set
max-Ir
10-4
min-Ir
Adam-betas
[0.9. 0.95]
[0.9, 0.99] 1 10-10
Adam-eps
10-
max max-grad mux-grad-norm
weight-decay
0.1
attention -ropput
0.1
path-dropout
0
0.05
embed-dropout
0
mlp-dropout
0
LSI-val
N/A
EMA
N/A
hidden-act
gelu
MPE
512 FALSE
1024
TWE
N/A
defenses, encouraging future direction.
Q6. Can GraphGPT generate graphs that match real-world constraints (e.g., chemical validity)?
A6. While generation is not the primary focus, preliminary experiments show GraphGPT can generate valid molecules after pre-trained on PCQM4M-v2.
However, generation quality depends on hyperparameters (e.g., temperature, top-p, iteration count T). Unconditional/condi tional generation and diversity control require further study, which is planned for future work.
Q7. Can you compare with other pre-trained-based graph models to highlight the advantages of GraphGPT?
A7. While models like GraphBERT (Zhang et al., 2020), GraphMAE (Hou et al., 2022), and GCC (Qiu et al., 2020) employ graph pre-training, they primarily target small-scale datasets. GraphGPT's evaluation focuses on large-scale OGB leaderboard benchmarks, where existing pre-trained models lack competitive entries. Our comparisons align with state-ofthh-as baselines dominating these leaderboards, emphasizing scalability and performance on real-world graph tasks.
Q8. Why there is need to introduce stochasticity when doing path identification?
A8. GraphGPT lacks some inductive biases inherent to GNNs (e.g., node permutation invariance). Randomly sampling Eulerian paths per epoch forces the model to learn invariance between different paths of the same graph, akin to how ViT (lacking CNN's inductive biases) benefits from large-scale data and data augmentation. Empirically, this reduced overfitting on molecular datasets.
Q9. Size to performance ratio.
A9. We clarify parameter counts and performance across datasets below:
Graph-Level (Tab. 1-2): GraphGPT's parameter counts are comparable to prior SOTA (e.g., 113.6M vs. 86M for GPTrans).
Edge-Level (Tab.4): For ogbl-ppa, GraphGPT-B (145.3M) is a bit worse than Refined-GAB (295.80 (9), but GraphGPT-XXL (2B) achieves the highest performance. For ogbl-citation2, GraphGPT-M (46.8M) and GraphGPT-B (133.IM) outperform
MPLP (749.8M).
25

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Table 15. Configurations of pre-training with SMTP and fine-tuning for the ogbn-prootims dataset.
pre-train
fine-tune
model-size
Small Medium Base
batch-size
256
128 16/16/8 epochs 4.8 4.8/4.8 4.8/4.8/2.4 epochs
total
2 X 10a tokens
warmup
10" tokens
Ir scheduler
Warmup & linear decay
Warmup & cosine decay 10 automatic set [0.9, 0.99] 1 x 10
max-Ir
10
min-Ir
Adam-betas
[0.9, 0.95
Adam-eps
10-6
max-grad-norm
weight-decay
0.1
attention-dropout
path-dropout
0
emhed-dropout
0.1/0.2/0.1
mlp-dropout
LSI-val
N/A
EMA
N/A
0.999
hidden-act
gelu 512
MPE
TWE
FALSE
N/A
Node-Level (Tab.5): GraphGPT requires larger parameters on ogbn-proteing and ogbn-arxiv. This may reflect insufficient pre-training data for these tasks, leading to suboptimal parameter utilization.
Q10. It is unclear what intended by model scalability in $3.3; additionally scalability seems to not be the answer to the problem if we take into account costs and computational resources required to solve the tasks.
A/0. Our investigation of model scalability serves two critical purposes:
Studying performance limits reveals fundamental insights of data. Even small performance gains can reduce real-world validation costs (OGB Team, 2020).
This study aligns with foundational NLP scaling law research (Kaplan al., 2020; Hoffmann et al., 2022), aiming to catalyze similar investigations for graph-structured data.
Q1I. How do you evaluate the correctness of the response? Do you query the Transformer model again with additional information if the response is not correct?
AII. The model directly outputs predictions via the task head during inference. Results are evaluated using standard metries (e.g., MAE, accuracy) for downstream task. Each test/valid instance is processed once; no iterative querying performed.
Q12. How is the prompt structured? How do you express the task to solve?
A/2. We do not use prompts. Instead, tasks are encoded via specialized tokens appended to the input sequence and processed by an additional MLP head during fine-tuning as discussed in $2.3.2. Fig. 2 illustrates the implementations.
26

---

GraphGPT: Generative Pre-trained Graph Eulerian Transformer
Table 16. Configurations of pre-training with SMTP and fine-tuning for the ogbn-arxiv dataset.
pre-train
fine-tune
model-size
Small Medium Base
batch-size
256
128
total
10" tokens
4 epochs
warmup
x 10 tokens
1.2 epochs
Ir scheduler
Warmup linear decay
Warmup & cosine decay
max-Ir min-Ir
x 10-4
3/3/2 2 x 10
automatic [0.9, 0.99]
Adam-betas
[0.9, 0.95]
Adam-eps
x 10
1 10
max-grad-norm
weight-decay
0.1
altention-dropout
0.1
path-dropout
0/0/0.1
embed-dropout
0.1
mlp-dropout
0 N/A
LSI-val
EMA
N/A
0.9997
hidden-act
gelu 1024
MPE
TWE
FALSE
Table 17. Run-time comparison between GraphGPT variants and GNNs on the PCQM4My2 dataset. Time per epoch is measured
minutes.
model
# params
time per epoch
GIN
3.76 M
9.25 min
GIN-virtual
6.66 M
11.2 min
GCN
1.96 M
8.0 min
GCN-virtual
4.85 M
9.6 min
GraphGPT-Tiny
0.62 M
20 min
GraphGPT-Mini
4.39 M
21 min
GraphGPT-Smal
17.17 M
20 min
GraphGPT-Mediun
33,95 M
20 min
GraphGPT-Bass
113.85 M
46.7 min
Table 18. Computational cost details of the main datasets in the paper. 'PT' means pre-training and 'FT' stands for fine-tuning. Time is measured hours. The model size 'Base' as Tab. with number parameters about 110M. The corresponding hyper-parameters can he found in Tab. 13, 14, 15, 16.
dataset
model size
PT time
FT time
GPU-PT
GPU-FT
ogbl-ppa ogbl-citation2
58,73
112.62
8 Nvidia L.20
V100-32G
B
72 h
100.3 h
8 Nvidia L,20
Nvidia L.0
ogbn-proteins ogbn-arxiv
27.1
3.1
8 Nvidia L20
V100-32G
9.25
4.3 h
8 Nvidia 1.20
V100-32G
27


Reviews:
[{"review_id": "1NCh9Q18uj", "reviewer": "Reviewer_ozCk", "summary": "The paper \"GraphGPT: Generative Pre-trained Graph Eulerian Transformer\" proposes GraphGPT, a self-supervised generative pre-trained model for graph learning. The core contribution is the Graph Eulerian Transformer (GET), which enables transformers to process graph-structured data efficiently by converting graphs into sequence representations using Eulerian paths.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback. Our replies are as follows:\n\nQ1. Authors mentioned that \"GraphGPT scales to over 2 billion parameters with sustained performance gains.\", but some plots about the scaling laws would make this claim stronger.\n\nA1: We appreciate the suggestion to strengthen our scaling analysis.\n\nUnlike NLP data, graph datasets lack uniformity, making it impractical to pre-train a single model across diverse domains (e.g., social networks vs. molecular graphs). As a result, GraphGPT is pre-trained and fine-tuned separately for different domains.\n\nFor datasets like PCQM4Mv2, we observe performance saturation at 227M parameters (Table 1), while for ogbl-ppa, we scale up to 2B parameters. The pre-training loss decreases steadily with increasing model size (Figure 3, Appendix, page 19), mirroring trends in NLP scaling studies (e.g., Fig.1 of Llama1 [1]). Fine-tuning results for three model sizes are reported in Table 4 for brevity, but we will include results for all six sizes (4M–2B parameters) in the Appendix.\n\nWhile a comprehensive scaling law analysis (e.g., estimating model/data scaling exponents) is beyond this paper’s scope, we will add logarithmic plots of pre-training loss and fine-tuning metrics versus non-embedding parameter counts to the Appendix, analogous to NLP scaling plots (e.g., Figure 1 of [2]).\n\nQ2. Yes. But evaluate GraphGPT on real-world citation networks (e.g., PubMed, Cora) or social networks (e.g., Twitter, Facebook graphs) could be great.\n\nA2: We evaluated GraphGPT on large-scale real-world citation networks: ogbn-arxiv (169K nodes, 1.17M edges) and ogbl-citation2 (2.93M nodes, 30.6M edges). These datasets are significantly larger than traditional benchmarks like Cora (2.7K nodes, 5.4K edges) and PubMed (19.7K nodes, 44.3K edges), aligning with our focus on scaling to massive graph data.\n\nWe chose these datasets because GraphGPT’s performance benefits from large-scale pre-training data to learn inductive biases (e.g., node permutation invariance). For instance, pre-training on the small Triangles dataset (45K graphs) yielded poor fine-tuning results (32.6%), whereas scaling pre-training data improved performance to 99% (Section 3.2.1). This mirrors the trend in ViT, which outperform CNNs only with sufficiently large datasets [3].\n\nWhile GNNs may outperform GraphGPT on small datasets like Cora or PubMed, our goal is to demonstrate scalability for large-scale graphs—a critical challenge in modern applications. We will clarify this rationale in the final version.\n\n\nQ3. No mention of computational resources.\n\nA3: We have included computational resource details for the PCQM4Mv2 dataset in Section 4.2 (Limitations: Computational Cost). \n\nTo address this feedback, we will expand this discussion in the revised version to provide comprehensive resource metrics (e.g., GPU types, training time, memory usage) for all key experiments, including ogbn-proteins/arxiv and ogbl-ppa/citation2 datasets. \n\nThis information will be added to the Appendix to ensure transparency.\n\nQ4. Three runs might be too few for high-variance tasks.\n\nA4: We clarify that variance is inherently low for most large-scale datasets (e.g., PCQM4Mv2, ogbl-ppa). For these datasets, 3–5 runs consistently yield minimal variance (as shown in tables). (It is common practice not to report the variance for PCQM4Mv2.)\n\nFor the Triangles dataset, variance is higher—particularly on OOD test data. So we conducted 10 runs to ensure robustness. As shown in Table 3, GraphGPT pre-trained on large-scale data achieves superior performance with reduced variance (e.g., 58.96 ± 1.9 vs. 54.76 ± 7.24).\n\nTo improve clarity, we will explicitly state the number of runs in table captions or footnotes where applicable.\n\nQ5. The structure of the paper is kind of messy.\n\nA5: We appreciate your input and welcome specific suggestions to improve clarity. \n\nCould you clarify which aspects of the structure are most problematic (e.g., section organization, flow of technical details in Sections 2.2–2.3, or appendices)? \n\nFor instance, if the nested content in Sections 2.2 and 2.3 is unclear, we will reorganize subsections to enhance logical progression.\n\nWe are committed to refining the structure to improve accessibility for readers.\n\nQ6. Too many footnotes in the table.\n\nA6: We apologize for the confusion caused by the table’s formatting. \n\nTo conserve space and ensure clear citations, we used numerical superscripts to reference source papers (similar to [4]) and subscripts to denote model sizes (detailed in Appendix Tab. 10).\n\nWe agree that this notation requires clarification and will incorporate explicit definitions of these notations directly into the table captions in the revised version.\n\n[1] Touvron et al., LLaMA 2023\n\n[2] Kaplan et al., Scaling laws 2020\n\n[3] Dosovitskiy et al., ViT. (ICLR2020)\n\n[4] Hussain et al., Triplet Interaction (ICML2024)", "claims_and_evidence": "Authors mentioned that \"GraphGPT scales to over 2 billion parameters with sustained performance gains.\"​, but some plots about the scaling laws would make this claim stronger.", "methods_and_evaluation_criteria": "Yes. But evaluate GraphGPT on real-world citation networks (e.g., PubMed, Cora) or social networks (e.g., Twitter, Facebook graphs) could be great.", "theoretical_claims": "Correct", "experimental_designs_or_analyses": "1. No mention of computational resources. \n2. Three runs might be too few for high-variance tasks.", "relation_to_broader_scientific_literature": "1. GNNs struggle with long-range dependencies due to repeated message passing, leading to over-smoothing and over-squashing. GraphGPT circumvents this limitation by tokenizing graphs into sequences via Eulerian paths, enabling transformer-based models to process entire graphs without localized message passing.\n\n2. GraphGPT extends self-supervised pretraining techniques from NLP (e.g., BERT, GPT-3) to graphs. It introduces Next-Token Prediction (NTP) and Scheduled Masked-Token Prediction (SMTP), adapting masked language modeling (MLM) techniques for graphs.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The structure of the paper is kind of messy, which may make audience harder to follow. Also the the model has too many footnotes in the table, I spend some time to figure out the meaning of each footnotes. It would be helpful to mention the meaning in the caption, or refer to the definition of footnotes.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L0cHaJ1Z3E", "reviewer": "Reviewer_ndAQ", "summary": "The authors in this paper introduce GraphGPT for graph learning that leverages Graph Eulerian Transformer (GET). The proposed model uses a graph-to-sequence transformation method based on Eulerian paths, enabling it to convert graphs into token sequences for transformer-based processing.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the valuable feedback. Our responses are as follows:\n\nQ1. GraphGPT excels with large-scale data but lacks comparisons with GNNs.\n\nA1: We compared GraphGPT with multiple GNN baselines (e.g., GCN, GIN, GCN-VN, GIN-VN) in all experiments. These baselines are standard in graph learning literature (e.g., [1, 2, 3]).\n\nResults are mostly organized in tables with GNN baselines, followed by graph transformer baselines, then GraphGPT.\n\nWe will clarify this in table captions in the final version.\n\nQ2. While GraphGPT enables a lossless and reversible graph-to-seq transformation, how well does it do this in real-world noisy graphs?\n\nA2: While not the focus of this paper, we tested GraphGPT on an internal noisy graph dataset (3.1M graphs, avg. 24.8 nodes, 54.7 edges) for edge denoising. \n\nUsing a semi-supervised node classification task, GraphGPT achieved 10-20% F1 score improvement over baselines. We formulated the task analogously to POS tagging in NLP, leveraging token-level embeddings. The \"long\" variant outperformed \"short\" (see Fig. 1) likely due to its edge-agnostic token embeddings of nodes.\n\nResults were robust enough for online deployment.\n\nQ3. Include runtime comparisons with GNNs.\n\nA3: Runtime comparisons are typically parameter-count based in literature. However, we will add runtime benchmarks for GNN baselines (from cited papers) and GraphGPT in the appendix.\n\nQ4. Yes, but there is very little validation to test if the lossless property will also hold true in large, noisy, real-world datasets.\n\nA4: The lossless property is theoretically guaranteed by Eulerian path theory, independent of noise.\n\nEmpirical performance on noisy graphs (as in A2) demonstrates practical robustness.\n\nQ5. This is a novel approach with scalability to billions of params. However, there is lack of clarity on the computational costs given the scalability and the robustness of the proposed model/method.\n\nA5: Computational costs for PCQM4M-v2 are discussed in Section 4. We will expand appendix details for other datasets. \n\nRobustness: most results show low variance, indicating robustness across runs. For adversarial/noisy robustness, see A2 and A7.\n\nQ6. There are some minor typos in the paper, for example, in Section 1, \"(semi- \\n )Eulerian paths\" that can be revised.\n\nA6: Corrected and will review other typo errors.\n\nQ7. How robust is the model to adversarial graph perturbations?\n\nA7. Adversarial robustness is a promising research area across NLP, CV, and graphs [4-7]. While not our primary focus, preliminary results on noisy graphs (A2) suggest robustness through large-scale training. A deeper study would bridge GraphGPT’s transformer architecture with adversarial graph defenses, an encouraging future direction.\n\nQ8. Can GraphGPT generate graphs that match real-world constraints (e.g., chemical validity)?\n\nA8. While generation is not the primary focus, preliminary experiments show GraphGPT can generate valid molecules after pre-trained on PCQM4M-v2. \n\nHowever, generation quality depends on hyperparameters (e.g., temperature, top-p, iteration count T). Unconditional/conditional generation and diversity control require further study, which is planned for future work.\n\n\nReferences\n\n[1] Chen et al. GPTrans (IJCAI 2023), \n\n[2] Masters et al. GPS++ (TMLR 2023), \n\n[3] Hussain et al. Triplet Interaction (ICML 2024), \n\n[4] Guo et al. Gradient-based Adversarial Attacks (EMNLP 2021), \n\n[5] Shao et al. On the Adversarial Robustness of ViT (NeurIPS 2022 workshop), \n\n[6] Jin et al. Adversarial Attacks and Defenses on Graphs (SIGKDD Explorations), \n\n[7] Sun et al. Adversarial Attack and Defense on Graph Data (IEEE TKDE2023)", "claims_and_evidence": "C1: GraphGPT excels with large-scale data but there is a lack of how GNNs perform to compare with.\n\nC2: While GraphGPT enables a lossless and reversible graph-to-seq transformation, how well does it do this in real-world noisy graphs?", "methods_and_evaluation_criteria": "Yes, but it would be helpful to the readers if the authors could also include a runtime comparison with GNNs.", "theoretical_claims": "Yes, but there is very little validation to test if the lossless property will also hold true in large, noisy, real-world datasets.", "experimental_designs_or_analyses": "Yes, the authors have very clearly demonstrated the performance of GraphGPT against several large-scale datasets that align with real-world applications.", "relation_to_broader_scientific_literature": "GraphGPT has been demonstrated to overcome the typical GNN limitations of over-smoothing and over-squashing and reduce the need for computing adjacency matrices by using Eulerian paths. It brings graph pre-training closer to transformers but may require more rigorous evaluation in interpretability, robustness, and graph generation quality.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "This is a novel approach with scalability to billions of params. However, there is lack of clarity on the computational costs given the scalability and the robustness of the proposed model/method.", "other_comments_or_suggestions": "There are some minor typos in the paper, for example, in Section 1, \"(semi- \\n )Eulerian paths\" that can be revised.", "questions_for_authors": "Q1: How robust is the model to adversarial graph perturbations?\n\nQ2: Can GraphGPT generate graphs that match real-world constraints (e.g., chemical validity)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "200DbzQ9sO", "reviewer": "Reviewer_RhxR", "summary": "The paper presents GraphGPT, a novel self-supervised generative pre-trained model for graph learning that utilizes a new architecture called the Graph Eulerian Transformer (GET). The GET integrates a transformer architecture with a graph-to-sequence transformation method based on Eulerian paths, allowing for the reversible conversion of graphs into sequences of tokens representing nodes, edges, and attributes. The model is pre-trained using two self-supervised tasks: next-token prediction (NTP) and scheduled masked-token prediction (SMTP). GraphGPT is then fine-tuned for various downstream tasks, including graph, edge, and node-level predictions. The experimental results indicate that GraphGPT achieves state-of-the-art performance on multiple large-scale Open Graph Benchmark (OGB) datasets, particularly excelling in molecular property prediction and protein-protein interaction tasks. Notably, the model can scale to 2 billion parameters while maintaining performance increase, addressing scalability issues faced by traditional Graph Neural Networks (GNNs) and prior graph transformers. However, the paper could benefit from improved clarity in presentation, theoretical grounding, and methodological details to enhance its impact and applicability in diverse domains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your questions. Our responses are as follows:\n\nQ1. Why did you choose the Eulerian path for graph-to-sequence transformation? How does it ensure lossless and reversible mapping?\n\nA1: The Eulerian path was selected for its ability to traverse each edge exactly once, enabling a sequential representation that preserves graph topology without redundancy.\n\nAs detailed in Section 2.2.1, this approach guarantees lossless and reversible mapping by construction, ensuring the sequence-to-graph conversion retains full structural fidelity. Theoretical justification is provided in the final paragraph of Section 2.2.1.\n\n\nQ2. How do you conduct the re-index and cyclic re-index in the graph-to-sequence transformation? What are their differences?\n\nA2: Implementation details for both methods are outlined in Section 2.2.1 (Node Re-indexing) and visualized in Figure 1. Re-indexing assigns fixed node IDs based on traversal order, while cyclic re-indexing dynamically rotates starting nodes to ensure uniform appearance frequencies of node-index tokens during training.\n\nTheir comparative impacts on model performance are analyzed in A2 and A8 in our response to the 4th review.\n\n\nQ3. Can you provide more details on the tokenization process used, particularly for non-text-based features?\n\nA3: Non-text attributes (e.g., numerical or categorical features) are discretized into tokens via binning or directly, as described in Section 2.2.1 (Attribute Handling). Appendix D provides concrete examples. Notably, the benchmark datasets (OGB) lack text features, so our focus centers on structured numerical/categorical attribute processing.\n\n\nQ4. Can you compare with other pre-trained-based graph models to highlight the advantages of GraphGPT?\n\nA4: While models like GraphBERT [1], GraphMAE [2], and GCC [3] employ graph pre-training, they primarily target small-scale datasets. GraphGPT’s evaluation focuses on large-scale OGB leaderboard benchmarks, where existing pre-trained models lack competitive entries. Our comparisons align with state-of-the-art baselines dominating these leaderboards, emphasizing scalability and performance on real-world graph tasks.\n\n\n[1] Zhang et.al, GRAPH-BERT. arxiv:2001.05140\n\n[2] Hou et.al, GraphMAE. (KDD2022)\n\n[3] Qiu et.al, GCC. (KDD2020)", "claims_and_evidence": "The authors provide extensive experimental results demonstrating that GraphGPT outperforms existing methods on various benchmark datasets. However, the claims regarding the theoretical underpinnings of the model needs to be further elaborated. For example, why the author choose the Eulerian path for graph-to-sequence transformation and how it ensures lossless and reversible mapping.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem at hand. The use of self-supervised pre-training and the task-agnostic fine-tuning approach aligns well with the objectives of improving graph representation learning. The benchmarks selected for evaluation, including PCQM4Mv2 and ogbl-ppa, are relevant and widely recognized in the field.", "theoretical_claims": "The paper does not present formal proofs for its theoretical claims, particularly regarding the lossless and reversible mapping of graphs to sequences using Eulerian paths.", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound, with a variety of datasets used to evaluate the model's performance across different tasks.", "relation_to_broader_scientific_literature": "The contributions of this paper are well-positioned within the broader scientific literature on graph learning. The authors reference key prior works, situating their approach as an advancement in the adaptation of transformer architectures to graph data.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Paper Strength**\n1. Innovative methodology to convert graphs into sequences using Eulerian paths and using transformer architecture effectively capture graph structure.\n2. Strong empirical findings demonstrating state-of-the-art performance across various benchmarks.\n3. Comprehensive evaluation and analysis of the model's scalability and performance on large-scale datasets.\n\n**Paper Weakness**\n1. Presentation needs to be improved for clarity and readability. For example, how to conduct the re-index and cyclic re-index in the graph-to-sequence transformation?\n2. The theoretical bounding of the model's claims is not sufficiently detailed. Why Eulerian path is chosen and how it ensures lossless and reversible mapping?\n3. Methodology lacks clarity in certain aspects, particularly regarding tokenization and feature usage. How to handle non-text-based features is unclear.\n4. The significance of transferability across different domains is not convincingly established.\n5. The model did not compare with other pre-trained-based graph models to highlight the advantages of GraphGPT.", "other_comments_or_suggestions": "- A preliminary section is recommended to provide a background and connection to the proposed method, which can help readers understand the motivation and significance of the work.\n- The presentation of the methodology can be improved to better illustrate the proposed method.", "questions_for_authors": "1. Why did you choose the Eulerian path for graph-to-sequence transformation? How does it ensure lossless and reversible mapping?\n\n2. How do you conduct the re-index and cyclic re-index in the graph-to-sequence transformation? What are their differences?\n\n3. Can you provide more details on the tokenization process used, particularly for non-text-based features?\n\n4. Can you compare with other pre-trained-based graph models to highlight the advantages of GraphGPT?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xbGpQQIuvA", "reviewer": "Reviewer_xS7R", "summary": "The paper introduces a self-supervised generative pre-trained model called GraphGPT based on a transformer architecture, which is called Graph Eulerian Transformer and employs a graph to sequence method which uses Eulerian paths. This method ensures reversibility in the graph-to-sequence translation. The transformer is first pre-trained for two kinds of self supervised tasks (next token prediction and scheduled masked token prediction) and then fine-tuned on downstream graph-, edge- and node-level tasks. The paper claims that the proposed method outperforms SOTA on Open Graph Benchmark datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the constructive feedback. Please let us know if there are any concerns that have not been addressed.\n\nQ1. Explanation of why there is a need to introduce stochasticity when doing path identification.\n\nA1: GraphGPT lacks some inductive biases inherent to GNNs (e.g., node permutation invariance).\n\nRandomly sampling Eulerian paths per epoch forces the model to learn invariance between different paths of the same graph, akin to how ViT (lacking CNN’s inductive biases) benefits from large-scale data and data augmentation.\n\nEmpirically, this reduced overfitting on molecular datasets. While we did not include an explicit ablation study due to space constraints, we acknowledge its importance and will clarify this in the final version.\n\nQ2. Unclear relation between index frequencies and token training, and how re-indexing helps.\n\nA2: Without cyclic re-indexing, Eulerian paths would always start with low-index tokens (e.g., 0, 1, 2), leading to skewed token frequency distributions.\n\nCyclic re-indexing randomizes starting indices (e.g., selecting from {0,1,…,255} for N=256), ensuring uniform training across all index tokens.\n\nThis is critical for datasets like Triangles, where test graphs have significantly more nodes than training graphs (e.g., test graphs up to 100 nodes vs. training graphs ≤25 nodes). Without re-indexing, higher-index tokens (e.g., 25–255) remained undertrained, degrading performance.\n\nWe will expand these details in the appendix.\n\nQ3. Criteria for choosing nodes for synthetic edges?\n\nA3: Synthetic edges are added to connect disconnected components. For example, if a graph has disconnected components A, B, and C, we connect A-B via a random node pair, then B-C similarly.\n\nThe synthetic edges are tagged with a special token <edge_jump> to distinguish them from real edges. This ensures the graph becomes connected, enabling Eulerian path generation.\n\nWe will clarify this in the text.\n\nQ4. Is SMTP linear scheduling function implementation the same as Chang et al.?\n\nA4: Yes, we use the linear scheduling function from MaskGIT (Chang et al., 2022), defined as γ(r) = 1 − r, where r ∈ [0, 1) and is uniformly distributed.\n\nWe will explicitly state this in the final version.\n\nQ5. Size to performance ratio.\n\nA5: We clarify parameter counts and performance across datasets below:\n\n- Graph-Level (Tables 1–2): GraphGPT’s parameter counts are comparable to prior SOTA (e.g., 113.6M vs. 86M for GPTrans).\n\n- Edge-Level (Tab.4): For ogbl-ppa, GraphGPT-B (145.3M) is a bit worse than Refined-GAE (295.8M), but GraphGPT-XXL (2B) achieves the highest performance. For ogbl-citation2, GraphGPT-M (46.8M) and GraphGPT-B (133.1M) outperform MPLP (749.8M).\n\n- Node-Level (Tab.5): GraphGPT requires larger parameters on ogbn-proteins and ogbn-arxiv. This may reflect insufficient pre-training data for these tasks, leading to suboptimal parameter utilization.\n\nQ6. In the benchmark only specific types of GraphGPT models are used. For instance in table 2, only GraphGPT B is used and not GraphGPT M.\n\nA6: We tested models of size S/M/B for most datasets (e.g., PCQM4M-v2 and ogbl-ppa). Omitted results were excluded from the main text due to space constraints but did not alter the conclusions.\n\nThese results will be added to the Appendix in the final version.\n\nQ7. model scalability-section 3.3.\n\nA7: Our investigation of model scalability serves two critical purposes:\n\n1. Studying performance limits  reveals fundamental insights of data. Even small performance gains can reduce real-world validation costs [3].\n\n2. This study aligns with foundational NLP scaling law research [1,2], aiming to catalyze similar investigations for graph-structured data.\n\nQ8. Why node reindexing in sec. 2.1?\n\nA8: Re-indexing nodes reduces overfitting. Ablation experiments confirm its effectiveness: re-indexing increased the training loss but improved validation/test performance.\n\nThis ablation study, initially omitted for brevity, will be included in the Appendix.\n\nQ9. k value in sec. 2.2.3/C.4?\n\nA9: For the datasets ogbl-ppa/citation2, ogbn-proteins/arxiv, we set k=2, resulting in vocabulary sizes of 41,634, 25,687, 31,360, and 25,600, respectively.\n\nQ10. How is the prompt structured? How do you express the task to solve?\n\nA10: We do not use prompts. Instead, tasks are encoded via specialized tokens appended to the input sequence and processed by an additional MLP head during fine-tuning as discussed in sec. 2.3.2.\n\nA figure illustrating the implementations will be added to the Appendix in the final version.\n\nQ11. Correctness of the response? Iterative querying?\n\nA11: The model directly outputs predictions via the task head during inference. Results are evaluated using standard metrics (e.g., MAE, accuracy) for the downstream task. Each test/valid instance is processed once; no iterative querying is performed.\n\n[1] Kaplan et al., arxiv:2001.08361\n\n[2] Hoffmann et al., arxiv:2203.15556\n\n[3] https://ogb.stanford.edu/docs/linkprop/#ogbl-ppa", "claims_and_evidence": "- “Randomly sample one valid path from possible candidates, introducing stochasticity as a data augmentation strategy akin to computer vision techniques (Perez & Wang, 2017).” -> It lacks explanation of why there is a need to introduce stochasticity when doing path identification\n\n- “Introduce cyclic re-indexing:... where r is a random integer and N (hyperparameter) exceeds the maximum node count. This ensures uniform token training by distributing index frequencies.” -> It is not clear what is the relation between index frequencies and token training and not clear how re-indexing helps with that.\n\n- “Connect components by adding synthetic edges between randomly selected nodes.” -> What is the criteria with which nodes are chosen for synthetic edges?", "methods_and_evaluation_criteria": "It makes sense. However it would be better to expand and specify how the linear scheduling function for Mask Scheduling (SMTP) is implemented. Is it the same as the one proposed by Chang et. al? In case, specify.", "theoretical_claims": "They seem to be correct.", "experimental_designs_or_analyses": "Please check the comments.", "relation_to_broader_scientific_literature": "I think this work contributes to some novel mechanisms to solve graph problems.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n- Performance: demonstrates performance across multiple graph tasks, being close and sometimes higher than SOTA.\n- Generalization: The experimental results are strong in many settings including datasets from different domains.  \n\nWeaknesses:\n\n- Size to performance ratio: while it is able to compare to and sometimes beat SOTA, it needs to be noted that other models achieve the same performance with a number of parameters two orders of magnitude smaller than GraphGPT.\n\n- In the benchmark only specific types of GraphGPT models are used. For instance in table 2, only GraphGPT B is used and not GraphGPT M. \n\n- It is unclear what is intended by model scalability - section 3.3; additionally scalability seems to not be the answer to the problem if we take into account costs and computational resources required to solve the tasks.", "other_comments_or_suggestions": "Please see below.", "questions_for_authors": "- In section 2.1 when talking about the implementation steps: why is there a need to do node reindexing?\n\n- How do you choose which nodes to connect when adding synthetic edges?\n\n- What linear scheduling function do you use for mask scheduling in SMTP?\n\n- In section 3.3. What do you mean by “scales [...] to 2 billion parameters”? Can you elaborate on that?\n\n- In section C.4, which k do you use for your token vocabulary (also with respect to section 2.2.3)\n\n- How is the prompt structured? How do you express the task to solve?\n\n- How do you evaluate the correctness of the response? Do you query the Transformer model again with additional information if the response is not correct?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a self-supervised generative pre-trained model based on a transformer architecture that employs a graph to sequence method based on Eulerian paths. This helps overcoming the common problem of GNNs to capture long-range dependencies. All reviewers agree that the approach is innovative and that the results are convincing, and overall recommend to accept the paper.
Decision: Accept (poster)