Graph Generative Pre-trained Transformer
Authors: Xiaohui Chen
Yinkai Wang
Jiaxing He
Yuanqi Du
Soha Hassoun
Xiaolin Xu
Liping Liu
Venue: ICML 2025 poster (2025)
Keywords: Graph generation
transformer
graph representation learning
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: A novel approach for modeling graph using GPT-like model

Abstract: Graph generation is a critical task in numerous domains, including molecular design and social network analysis, due to its ability to model complex relationships and structured data. While most modern graph generative models utilize adjacency matrix representations, this work revisits an alternative approach that represents graphs as sequences of node set and edge set. We advocate for this approach due to its efficient encoding of graphs and propose a novel representation. Based on this representation, we introduce the Graph Generative Pre-trained Transformer (G2PT), an auto-regressive model that learns graph structures via next-token prediction. To further exploit G2PT's capabilities as a general-purpose foundation model, we explore fine-tuning strategies for two downstream applications: goal-oriented generation and graph property prediction. We conduct extensive experiments across multiple datasets. Results indicate that G2PT achieves superior generative performance on both generic graph and molecule datasets. Furthermore, G2PT exhibits strong adaptability and versatility in downstream tasks from molecular design to property prediction.

Graph Generative Pre-trained Transformer
Xiaohui Chen
Yinkai Wang
Jiaxing He?
Yuanqi Du 3
Soha Hassoun
Xiaolin Xu Li-Ping Liu'
Abstract
sequential models. These approaches employed sequential frameworks (c.g., RNNs LSTMs (Sherstinsky, 2020)) to generate graphs sequentially. For instance, GraphRNN gen- erates adjacency matrix entries step by step. For undirected graphs, it only needs to generate the lower triangular part of the adjacency matrix. DeepGMG frames graph genera- tion as a sequence of actions (e.g.. add-node, add-edge add-edge), and utilizes an agent-based model to learn the action trajectories.
Graph generative models, which can produce complex structures that resemble real-world data, serve as essential tools in domains such as molec- ular design and network analysis. While many existing generative models rely on adjacency ma- trices, this work introduces token-based ap- proach that represents graphs token sequences and generates them via next-token prediction, offering more efficient encoding. Based on this methodology. we propose the Graph Gen- erative Pre-trained Transformer (G2PT), an auto- regressive Transformer architecture that learns graph structures through this sequence-based paradigm. To extend G2PT's capabilities a general-purpose foundation model, we further de- velop fine-tuning strategies for two downstream tasks: goal-oriented generation and graph prop- erty prediction. Comprehensive experiments on multiple datasets demonstrate G2PT's superior performance in both generic graph generation and molecular generation. Additionally, the good ex- periment results also show that G2PT can be ef- fectively applied to goal-oriented molecular de- sign and graph representation learning. The code of G2PT is released at http:://gitubb.com/uti ml/G2PT.
Recent advances in graph generative models have primar- ily focused on permutation-invariint methods, particularly biffusion-based approaches (Ho et al., 2020; Austin et al., 2021). For example, models like EDP-GNN (Niu et al., 2020) and GDSS (Jo et al., 2022a) learn from adjacency matrices as continuous values. DiGress (Vignac et al., 2022) and EDGE (Chen al., 2023) employ discrete diffusion, treating node types and all node pairs (edges and non-edges) as categorical variables. These models start from a random or a fixed adjacency matrix and run "denoising" steps to sam- ple an adjacency matrix from the target graph distribution. They specify exchangeable (permutation-invanl butions over graphs by assigning the same probability to adjacency matrices of the same graph. However, achieving the crrmutation-invarian property has price: the under- lying neural network needs to be permutation-invarie well, limiting the architecture choice to graph neural net- works only. Discrete diffusion has an additional limitation: it samples matrix entries independently at each denoising step, making it challenging to learn the true distribution when the number of denoising steps is insufficient (Lezama et al., 2022; Campbell et al., 2022).
distri-
as
1. Introduction
Graph generation has emerged as a crucial task across di- verse fields such as chemical discovery and social network analysis, thanks to its ability to model complex relationships and produce realistic, structured data (Du et al., 2021; Zhu al., 2022).
In recent years, the revolutionary success of large language models (Achiam et al., 2023; Dubey et al., 2024) shows the power of autoregressive Transformers and also inspired the application of these models in other fields such as image generation (Esser al., 2021). In this work, we revisit the sequential approach to graph generation and introduce a novel token-based encoding scheme for representing sparse graphs as sequences. This new encoding strategy unlocks the potential of Transformer architectures for graph gen- eration. We train autoregressive Transformers to generate graphs by predicting token sequences, resulting in our pro- posed method: Graph Generative Pre-trained Transformer (G2PT). While G2PT does not maintain permutation in- variance, argue that its capacity to learn accurate graph
Early generation methods such as DeepGMG (Li et al., 2018) and GraphRNN (You et al., 2018b) model graphs with
'Tufts University NNrtheastee University *Cornell University. Correspondence to: Xiaohui Chen <xiaohui.chen @ tufts.edu>> Li-Ping Liu <liping. Saaping.liuutuun iiu@tufts.edu>.
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Graph Generative Pre-trained Transformer
Model (Rep.)
Likelihood
Illustration
#Network Calls
#Variables
p(A) p(E)
Diffusion(A)
p(A")
(A
O(Tn²)
Intractable
Scquential(A)
IIII
p(A ca Ages)
O(12)
O(x2)
Full factorization
=
Scquential(E)
p(e) p(alec))
O(m)
((m)
Full factorization
Table 1. Families of graph generative models and their graph representations. Here is the number of nodes, and m is the number of edges. The illustration use solid and dash lines to represent edges and non-edges respectively. The (non-)edges generated at current step are in blue. Our proposed G2PT is autoregressive model and learns with Sequential(E).
distributions from large-scale data outweighs this limitation. Table gives a comparison of diffusion-based models, se- quential models based adjacency matrices, and our model based on edges.
distributions via next-token prediction;
We explore fine-tuning techniques to adapt G2PT for downstream tasks, such goal-oriented graph genera- tion and graph property prediction;
We conduct an extensive empirical study of G2PT, which achieves strong performance across diverse graph generation and prediction tasks.
This new approach offers an additional advantage: it al- lows us to leverage recent training techniques developed for Transformers in NLP. Specifically, we apply our model to goal-oriented generation tasks, such as optimizing molecular properties. To this end, we explore both rejection sampling fine-tuning and reinforcement learning, each designed to increase the probability mass assigned to desirable graphs. Additionally, our model can be used to learn graph repre- sentations when provided with the entire graph as input- a capability not feasible with diffusion-basee generative models. With fine-tuning and supervised objective, G2PT can also be adapted for graph-level tasks such as graph classification.
2. Related work
Permutation-invarian vs sequential models. Neural graph generative models are first developed as autoregres- sive models. These models represent graphs as sequences and learn to generate such representations. A variety of mod- els such as GraphRNN (You et al., 2018a), GRAN (Liao et al., 2019), GraphDF (Luo et al., 2021), and DAGG (Han et al., 2023) generate entries of the lower triangular of the adjacency matrix of the target graph, while DeepGMG (Li et al., 2018 8) and BiGG (Dai et al., 2020) directly generate edges through node representations learned from partially generated graphs.
We evaluate G2PT on series of graph generation tasks: generic graph generation, molecule generation, and goal- oriented molecular generation. Without excessive architec- tural engineering or training tricks, G2PT performs better than or on par with previous state-of-the-un art (SOTA) base- lines over seven datasets. By fine-tuning G2PT towards showcase generating molecules with target properties,
In parallel, another research direction considers permutation- invariant models to predict logits of adjacency matrices and then sample the adjacency matrix in one step (Madhawa et al., 2019; Liu et al., 2019). These models define ex- changeable distributions over graphs. This approach has been further advanced with diffusion-based methods (Ho et al., 2020; Austin et al., 2021), which can be divided into continuous diffusion models (Niu et al., 2020; Jo al., 2022b) and discrete diffusion models (Vignac al., 2022: Chen al., 2023; Qin al., 2024). Among these, discrete diffusion models have shown superior performance due to their alignment with the discrete nature of graphs. These models begin with a fixed or random adjacency matrix and iteratively update matrix entries over multiple steps to con- verge to the target adjacency matrix. Each update step, referred as a "denoising" step. modifies a subset of the adjacency matrix entries.
that G2PT can be easily adapted to various generative tasks that require additional alignment. We also fine-tune G2PT for molecular property prediction on MoleculeNet datasets. The results demonstrate the effectiveness G2PT's learned representations for classification tasks. Finally, we analyze the G2PT model and show its performance with data aug- mentation.
Contributions. Our main contributions are as follows:
We propose novel token-based graph representation that enables efficient graph generation;
We introduce G2PT, a Transformer decoder trained on the new graph representation to model sequence
2

---

Graph Generative Pre-trained Transformer
Discrete diffusion graph models rypically require many de- noising steps for graph sampling. This is because entries are sampled independently conditioned on the previous step for each step. Such sampling scheme introduces the com- pounding decoding error (Lezama et al., 2022), leading to a poor approximation to the true distribution (Campbell et al., 2022).
Algorithm Degree-Based Edge Removal Process
Input: Graph G = (V, E), neighborhood function Nei(.)
Output: Sequence of removed edges os
Initialize Øg +-
while É + 0 do
Select tue E V with the minimum degree. Select Udes E Nei(Use) with the minimum degree.
Remove edge e = (Var, Vsea) from E.
Append to gg. Update the degrees of Une and Vaest- end weileReverse dE
Generating adjacency matrices vs edge lists. Most graph generative models, including the majority of sequential approaches (You et al., 201 88;; Liao et al., 2019) and all diffusion-basse methods (Jo et al., 2022b; Niu et al., 2020: Vignac et al., 2022; Chen et al., 2023; Qin et al., 2024), gen- erate graphs via their adjacency matrix, with a few excep- tions (Li et al., 2018; Dai et al., 2020) that operate directly on the edge list of the target graph.
(uil yid ater desit e²), where the first two elements define the edge
connection and e E {1...., Ke} is the edge type. For a graph without node or edge labels, the above representation
can be simplified by removing the node or edge labels from
the sequence. A graph G with n nodes and m edges can be represented as
Despite the popularity, learning adjacency matrices requires modeling all node pairs in graph, and thus needs com- putation quadratic in the number of nodes n. For autore- gressive models, creates long sequences for the model to Iearn, making it a challenging learning problem (Hihi & Bengio, 1995). Therefore, generating adjacency matri- ces are computationally intensive for both autoregressive and diffusion-based models. In contrast, edge lists only define variables over actual edges. Compared to adjacency matrices, models that use edge lists usually require less com- putation for both learning and sampling, especially when dealing with sparse graphs.
[uf ofd ...... on oid as
uid id dest ei fir...
li sree
dess E
nx2
x
Here as is the special token separating node tokens from edge tokens. We illustrate it in Figure 1.
The sequence depends on node order and an edge or- der. Nodes are ordered randomly. Nodes in the first part of the sequence follow the node order, and thus ufd = i. The edge order determined by reversing a degree-based edge-removal process, as described in Alg. 1. The removal process prioritizes the removal of edges connected to low- degree nodes, its reverse constructs compact, relatively dense core first, followed by the addition of edges with fewer connections. Edges in the second part of the sequence follow this construction order. We also explore the effectiveness of using other edge orderings such as breadth-firs search (BFS) and depth-first search (DFS) (details are presented in Appendix D.1).
Despite the advantage mentioned above, modeling edge lists has received limited attention because of the difficulty of modeling structurally heterogeneous discrete events (c.g., adding a node and deciding the two ends of an edge). Li et al. (2018) propose an action-bases framework and model a graph with pre-defined actions such as "add-node" and "add-edge". With a special design of the model architecture, the model is challenging to train. Additionally, the "add- edge" action cannot be treated as label prediction because the number of nodes increases with the generation procedure. Previous methods rely on node representations learned from graph neural networks, which are shown to have limited expressiveness in general (Li & Leskovec, 2022). This work takes a different approach and designs a baken-based sequences that enable generic Transformer architectures.
3.2. Learning Graph Sequences via Transformer
We utilize a Transformer decoder (Vaswani, 2017) for mod- cling the graph sequences. Before this, we construct uni- fied vocabulary containing all previously introduced tokens. A tokenizer is defined to map each token to unique integer.
Tokenization. Let Temux be the maximum number of nodes of a graph dataset. The unified vocabulary is then defined as
3. Graph Generative Pre-trained Transformer
3.1. Representing Graph as Sequence
tokenize (1²d)) vid, wid E (I,...,, :Pmax
We consider modeling a graph as a sequence that first lists all nodes and then all edges. Let G = (V, E) denote graph where nodes and edges are each associated with a type label. Here v E V is represented as a tuple v : (uS, vd)) where gid Z+ is the node index and of E {I,..., K.) is the node type. And E E is represented as triple e :=
tokenize (v²) ]- = of Tmmax J E (l....,Ky))
tokenize (ee) = c + Dmax + Kn, e (1, K.):
tokenize(aa) (ax) = 7'max + K + Ka+ +
We additionally introduce special tokens [SOG] and [EOG], representing the start and the end of the sequence generation.
3

---

Graph Generative Pre-trained Transformer
Graph representation
Training Transformer on Sequence representation
32
3
S3
41
R
52
Vocabulary
Index mode
Node Index
Node Type
Edge Type
Special Token
A
C
II
III
80G
BOG
Token
31
32
33
41
42
43
51
52
S0G
B
...
%
8
DOG
Encode
Sample edge order
Node
Edge
Node definition
Edge definition
(1,2), (2,3), (7.8)
Figure 1. Illustration our proposed graph sequence representation. This representation can viewed as sequence of actions: first generating all nodes (node type, node index), then explicitly adding edges (souree node index, destination node index, edge type) step by step until completion. unified vocabulary used map different types of actions into shared token space.
We denote the tokenized sequence s = [s1,.... $1). which is used in the following sections.
not directly compute, providing more sequences from the same graph G can reduce the condition KL. Intuitively, in- creasing the number of sequences increases the diversity of the training data, thus improving the model's generalization. Combining graph datasets and random sequences, we use the objective (1) to pre-train Transformer model as the generative model.
Training objective. With the autoregressive Transformer, we minimize the negative log-likelihood of the sequence.
Cpt (s: 0) == log (s)
loggo(s:(s=1),
(1)
where 9 denotes model parameters. With the rules of form- ing sequences from graphs, not all tokens are legal for sı at prediction step (o(s))s2)). For example, when the current token is a node type, the next token can only be one of the node indices. Illegal tokens can be avoided by masking out their logits in the model output. However, our exper- iments show that unconstrained logits also yield superior performance thanks to the learning power of Transformers.
4. Fine tuning
After pre-training a model, we further fine-tune it for down- stream tasks. We consider generative ($4.1) and predic- tive ($4.2) downstream tasks, where the former aims to generate graphs with desired properties, and the latter uti- lizes the graph embeddings learned from the Transformer to predict properties.
Relationship with graph likelihood. We show that maxi- mizing the sequence likelihood maximizes a lower bound of the graph likelihood. Denote training of N graphs (G Gr). ). The log-likelihood of the graph dataset is L(0) Ei=1 log pe (G.). Let Pa(G) denote the data dis- tribution, and pa Gs] denote the distribution of drawing sequences from graph G. We have
4.1. Goal-oriented Generation
Let 2 ((G) denote the function that estimates property of graph G. In goal-oriented generation, our objective is to train new model that generates graphs with property values closer to a target value =" than those produced by the pre-trained model. This problem has a broad application such as drug discovery. this work, we fine-tune the pre- trained model to improve its ability generate graphs that better meet the specified property criteria. To this end, explore two strategies: rejection sampling fine-tuning (RFT) and reinforcement learning (RL).
L(0) + (G)] = KK (LL (pa(C))lpo
2 (p(()(())l) - E (T((p((((C)))pd(s(I))
-KL(pa())lloo(s) Σ Lpe (s:; Hl)pa(s))
Rejection sampling fine-tuning. This approach fine-tunes the model using its generated samples that have the de- sired property values. Here we consider the case that the property a scalar and specify and an acceptance function ma (G) = 1 -(--)(())<w> where the distance tolerance hyperparameter.
In the third line, use the fact that p(G,s) = p((s) be- cause s determines G. Note that both the entropy terms are constant with respect to the model parameters 0.
From the lower bound, we see that an accurate approxima- tion Po G) can tighten the bound. Although Gs is

---

Graph Generative Pre-trained Transformer
Algorithm 2 RFT Dataset Construction
is the expectation of the undiscounted future return:
Input: Model pe, acceptance function m data size B.
V*((<1) = (p(ezz)sc)) [r(s)]-
Output: Fine-tuning dataset D
Initialize D +- }
A critic model (8<() is then learned to approximate the true value function (P(s<c) via minimizing the mean abso- lute error Laiii (v+). We parameterize the critic model using Transformer with the same architecture as the pre-trained model, except that the logits head is replaced with a value head. The parameters of the critic model are also initialized from the pre-trained model.
while [D=
ب do Generate G ~ pe-
if G is valid and m (G) 1 then
Append G to D
end end while
Algorithm 3 SBS combined with RFT
We use the clipped surrogate objective Spg-olip (o) in PPO to optimize the actor model. Moreover, to mitigate possible model degradation, we incorporate the pre-training loss (p(() following Zheng et al. (2023) and Liu et al. (2024).
Input: Model pe, thresholds list [w) ,h),..] Wu].
Output: Fine-tuned model pea
Set 0g 0.
for i 1....,h k do
Use p as input model, obtain D <- Alg. 2. Finc-tune 0,-1 D obtain new parameters 0..
All terms combined, we minimize the objective: Lpolds v; 0) -pg + p2
end for
Here pl, P2, P3 are loss coefficients. We provide preliminar- ies of PPO and details of each loss term in Appendix A.
We run the generative model and collect valid graphs that satisfy the acceptance function to construct the fine-tuning dataset DE = [- The algorithm is shown in Alg. 2. Note that we expect the learned pre-trained model to have the ability to generate decent fraction of graphs with the desired property.
4.2. Property Prediction
In a graph-level learning task, we often need to predict the label y of a graph G. With the pre-trained model, we fine- tune it to address the graph-level learning task. After the sequence is generated from G, we extract the activation h of the final token Sy, from the last Transformer block as the graph representation. The rationale is that the model must have learned the information of the entire graph in order to expand it with further edges. We replace the token- predicting layer with an MLP to predict y based on h:
RFT becomes inefficient if the pre-trained model can rarely generate acceptable samples. To address this, we run the self-bootstrap (SBS) version of RFT to approach the target distribution in multiple rounds. With l tolerances w1 > w2 > > We = w, we obtain a sequence fine-tuned models by iteratively constructing fine-tuned datasets using the model trained from the previous tolerance. The SBS algorithm combined with RFT is shown in Alg. 3.
ply)s) = Softmax((ropootte (Dropou((Linear(I))
We minimize the cross-entropy loss -C log PGy over a graph classification dataset C to fine-tune the model. Compared to freezing the whole Transformer during training and only updating parameters of the linear layer, we found that unlocking the latter half of the Transformer blocks significantly enhances the performance.
Reinforcement learning. Denote target-relevant reward function r=. (G), we consider a KL-regularized reinforce- ment learning problem:
= arg max (S) ([))))))))())
Here r- (s) Tg.. (G) uniquely decides G. The KL divergence KT.(-II-) prevents the target model from deviating too much from the pre-trained model.
5. Experiments
5.1. Setup
We choose Proximal Policy Optimization (PPO) (Schulman et al., 2017) to effectively train the target model ps without sacrificing stability. The token-level reward is r-. (s) only at the last token and zero otherwise:
Datasets. We consider both generative tasks and predictive tasks our experiments. We use three molecule datasets: QM9 (Wu al., 2018b), MOSES (Polykovskiy et al., 2020), and GuacaMol (Brown et al., 2019); and four generic graph datasets: Planar, Tree, Lobster, and stochastic block model (SBM), which are widely used to benchmark graph gen- erative models. In predictive tasks, we fine-tune models pre-trained from GuacaMol datasets on molecular proper- ties with the benchmark method MoleculeNet (Wu et al., 201 1a). Further details are in Appendix B.4,
-{r(leess (((ssct,s))) si])
10
+ [EOG]
R([sel. s())
81 = [EOG]
Here Szi is the state of the 1-th step in a finite trajectory (sequence). The value function of state Set under model p
5

---

Graph Generative Pre-trained Transformer
Planor
Tree
Model
Deg.4
Clus.4
Orhit
Spec.
Wavelet
VUN.f
Deg.4
Cluc.
Orbit.
Spec.
Wavelet
V.U.N.
GRAN (Lian al., 2019)
7e-4
4.3e-2
9e-4
7.5e-3
1.9e-3
0
1.9e-1
8e:3
2e-2
2.8e-1
3.3e-1
BiGG (Dai 2020)
7e.4
5.7e-2
3.7e-2
1.
5.2e.3
5
1,4e.3
0.00
0.00
1.2e-2
5.8e-3
75
DiGress (Vignac ot al, 2022)
7o-4
7.8c-2
7.9c-3
9.8c-3
3.1e-3
77.5
2o-4
0.00
0.00
1.1e-2
4.3e-3
90
BwR (Diamant
2023)
2.3c-2
2.6c-1
4.4c-2
1.3c-1
1.60-3
1.2c-1
3c-4
4.8c-2
3.9c-2
HSpectre (Bergmeeister at., 2023)
Se-4
6.3e-2
1.7e-3
7.5e-3
1.3e-3
95
le-4
0.00
0.00
1.2e-2
4.7e:3
100
GEEL (Fang et al., 2023)
le.3
le.2
27.5
1.5e-3
1.5e-2
DeFoG (Qin et al., 2024)
Se-4
Se-2
6e-4
7.2e-3
14e-3
99.5
2e-4
0.00
0.00
1.1e-2
4.6e-3
96.5
G2PT.uatl
4.7o-3
2.4c-3
0.00
L-c-2
1.40-2
95
20-3
0.00
0.00
7.4o33
3.9e-3
G2PT
1.8e3
4.76-3
0.00
8.1e-3
5.1e-3
100
4.3c-3
0.00
le-1
7.3e-3
5.7e-3
99
Lobster
SBM
Model
Deg.U
Clus.
Orbit
Spec.
Wavelet, V.N.T
Deg.4
Clus.
Orbit]
Speo.
Wavelet, V.U.N.
ino 2019)
3.8e-2
2.7e-2
1.1e:2
5.5e-2
5.4e-2
5,4e-3
BIGG (Dai et al., 2020)
0.00
0.00
0.00
96-3
1.2c-3
6.0c-2
6.7c-2
5.9c-3
3.7c-2
10
DiGress (Vignac et al., 2022)
2.1e-2
0.00
4o-3
1.8c-3
4,9c-2
4.2o-2
4.56-3
1.4e-3
60
BwR (Diamant al., 2023)
3.2e-1
0.00
2.5e-1
4.8e-2
6.4e-2
1.le-1
1.7e-2
8.9e-2
7.5
HSpectro (Bergmeister et al., 2023)
1.2c-2
5,20-2
6.76-2
6,70-3
2.2c-2
45
GEEL (Jang et al., 2023)
2e-3
1o-3
72.7
2.5e-2
3e-3
2.6e-2
DeFoG (Qin et al., 2024)
6e-4
5.2e:2
5.6e
5.4e-3
8e.3
90
G2PT.ral
2e-3
0.00
0.00
3e:3
8.5e-3 le.2
100
3.5e-3
1.2e-2
Te-4
7.6e-3
9.8e-3
100
G2PTosee
le.3
0.00
0.00
4e-3
100
4.2e 3
5.3e
3e.4
6.1e-3
6.9e.
100
Table 2. Generative performance on generic graph datasets.
Model specifications. We train Transformers with three different sizes: (1) the small Transformer has 6 layers and attention heads, with dmeael = 384, leading to approxi- mately 10M parameters; (2) the base Transformer has 12 layers and 12 attention heads, with dmoodel = 768, leading to approximately 85M parameters; (2) the large Transformer has 24 layers and 16 attention heads, with dimode! = 1024, leading to approximately 300M parameters. We use differ- ent specifications for different experiments according to the task complexity.
Rep.
#Tokens]
Deg.u
Clus.)
Orbit.
Spec...
Wavelet.
V.U.N.1
2018
8e-3
6.1e-2
(Ous)
737
4.7e-3
2.4e-3
0.00
1.6e-2
1.4-2
95
(Ours)
Table 3. Generative performance comparison between the pro- posed edge sequence and adjacency matrix representations.
5.2. A Case Study with Planar Graphs
considers degree (Deg.), clustering coefficient (Clus.), orbit counts (Orbit), spectral properties (Spec.), and wavelet statis- tics. Morcover, we report the percentage of valid, unique, and novel samples (V.U.N.) (Vignac et al., 2022). For this task, we trained the G2PT small and G2PT models.
We compare our token-based representation against the ad- jacency matrix and validate its effectiveness in the genera- tion task. We train Transformer decoders on planar graphs using our boken-based representation and adjacency matri- ces, and then evaluate their generative performance. For the adjacency representation, planar graphs are encoded as sequences of Os and 1s derived from the strictly lower triangular matrix, with rows and columns permuted using BFS orderings to augment the training dataset. Table 3 presents the quantitative and qualitative results of the gen- erated samples. Our proposed representation demonstrates superior generative performance with a much smaller set of tokens. In contrast, the model trained with adjacency matrices struggles to capture the topological rule of planar graphs.
As shown in Table 2, G2PT demonstrates superior perfor- mance compared to the baselines. The details about baseline and metric are introduced in appendix B.5 The base model achieves 11 out of 24 best scores and ranks in the top two for 17 out of 24 metrics. The small model also demonstrates competitive results, indicating that lightweight model can effectively capture the graph patterns in the datasets.
5.4. Molecule Generation
De novo molecular design is a key real-world applica- tion of graph generation. We assess G2PT's performance on the QM9, MOSES, and GuacaMol datasets. For the QM9 dataset, we adopt the evaluation protocol in Vignac et al. (2022). For MOSES and GuacaMol, we utilize the evaluation pipelines provided by their respective toolk- (Polykovskiy et al., 2020; Brown et al., 2019).
5.3. Generic Graph Generation
We evaluate G2PT on four generic datasets using Maximum Mean Discrepancy (MMD) to compare the graph statistics distributions of generated and test graphs. The evaluation
6

---

Graph Generative Pre-trained Transformer
MOSES
GuacaMol
Model
Validity
Unique:
Novelty
Filterst
FCD4
SNN?
Scaft
Validityt
Unique.1
Novellyf
Kl. KivDiv.
FCD1
LigGPT (Bagal Aggurwal)
90.0
99.9
94.1
98.6
99.8
100
DiGress (Vignac al. 2022)
85.7
100
95.0
97.1
1.19
0.52
14.8
85.2
100
99.9
92.9
68
GEEL (Jang al,, 2023)
92.1
100
81.1
97.5
1.28
0.52
3.6
88.2
98.2
89.1
93.1
71.5
DisCo (Xu al., 2024)
88.3
100
97.7
95.6
86.6
86.6
86.5
92.6
59.7
Cometh (Siraudin et al., 2024)
90.5
99,9
92.6
99.1
1.27
0.54
16.0
98,9
98.9
97.6
96.7
72.7
(Qin 2024)
99.0
G2PTomot onlll
95.1
100
91.7
97.4
1.10
0.52
5.0
90.4
100
99.8
92.8
86.6
G2PT
96.4
100
86.0
98.3
0.55
94.6
100
99.5
96.0
93.4
97.2
100
79.4
98.9
1.02
0.55
2.9
95.3
100
99.5
95.6
92.7
Model
QM9
MOSES
GuacaMol
Validity
Unique.1 FCDL
Train
G2PT.mal
G2PT besc
G2PT Tha
DiGress (Vignac ct al., 2022)
99.0
96.2
DisCo al., 2024)
99.6
0.25
of Torch
Cometh (Siraudin et al., 2024)
99.2
96.7
0.11
(Qin et al., 2024)
99.3
G2PT uncil G2PT buse G2PTiape
99.0
96.7
0.06
Exot
Puods
99.0
96.8
0.06
98.9
96.7
0.06
Table Generative performance on molecular graph datasets
10
Date
RFT(C
RFTewlo D.4
Prestralood
RFT(= 0.5)
Pre-tralnod RPTTC S.0
RFT 188
10
Presrained
RF RFTeee (6)
RITT2
RFFT(=00:))
RPT( 0.8)
RF RF 0.8)
Density
0.0
0.2
0.4
0.8
0.8
1.0
10
0.0
0.2
0.4
0.6
1.0
QED Score
SA Score
GSK3 Score
(a) Rejection sampling fine-tuning (with self-bootstrap)
0.8
3.0
Pnsirained
Protraine: PPO
Prostaited
2.5
0.6-
10
PPO
2.0
Density
1.0
4
0.2
0.0
0.0
0.0
0.2
0.1
0.5
0.8
1.0
6
10
0.0
0.2
0.1
0.6
0.8
1.0
QED
GSK3 Score
(b) Reinforcement learning framework (PPO)
Figure Gral-ori molecule generation using QED, SA and GSK3,3 scores. Top row (a) shows the results using RFT, and bottom row (b) shows the results using RL.
The quantitative results are presented in Table 4. On MOSES, G2PT surpasses other state-of-the-an art models in va- lidity, uniqueness, FCD, and SNN metrics. We introduce the details for metrics in appendix B.6. Notably, the FCD, SNN, and scaffold similarity (Scaf) evaluations compare gener- ated samples to held-out test set, where the test molecules have scaffolds distinct from the training data. Although the scaffold similarity score is relatively low, the overall perfor- mance indicates that G2PT achieves a better goodness of fit on training set. G2PT also delivers strong performance on the GuacaMol and QM9 datasets. We additionally pro- vide qualitative examples from the MOSES and GuacaMol
datasets in the table.
5.5. Goal-oriented Generation In addition distribution learning which aims draw inde- pendent samples from the learned graph distribution, goal- oriented generation is major task in graph generation that aims to draw samples with additional constraints or pref- erences and is key to many applications such as molecule optimization (Du et al., 2024).
We validate the capability G2PT on goal-oriented gen- eration by fine-tuning the pre-trained model. Practically,

---

Graph Generative Pre-trained Transformer
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
Avg.
AttrMask (Hu et al., 2020a)
70.2:0.55
7..2::0.8
62.5+0.4
60.410.6
68.6:1966
73.911..
1.
17.2
70.2
InfoGruph (Sun et al., 2020)
69.2+0.8
73.040.7
62.0=0.3
59.240.2
75.145.0
74.0=11.5
74.5=18
73.942.5
Contex(Pred (Hu ct al., 2020a)
71.2-0.9
73.3+00.5
62.8=0.3
59.311.4
73.744.0
72.5#2.2
75.8=111
78.6+
70.9
GraphCL (You et al., 2021)
67.5+2.5
75.0::0.5
62.8=0.2
60.1
78,944.2
77.1=1.0
75.0=0.4
68.747.8
70.6
GraphMVP (Liu et al., 2022a)
68.510.2
74.5::0.0
62.711.11
6..3
79.0:2.5
75.0::1.4
7..8
76.11
GraphMAE (Hou et al., 2022b)
70,910.9
75.0:1044
64.1
59.910.55
81.52288
76.91:266
76.7:00.9
81.414
73.3
G2PT sml (No pre-training)
60.7+0.3
66.4++0.5
57.0+0.3
61.6+0.2
67.8+111
45.8+8.5
70.1=7.55
68.8+1.3
62.3
G2PT l (No pre- training)
56,510.2
67.410.4
57.910.1
60.212.8
71.0:5.6
60.1
72.71111
73.410.3
64.9
G2PT small G2PT
68.5+0.5
74.7+0.2
61.2+0.1
61.7+1.0
82.3+2.2
74.9+H.1
75.7+0.4 76.3+0.4
81.3+0.5
72.5
71.0+0.4
7..0++.3
63.0+0.5
61.9+002
82.1+1.1
74.5+0..
82.3+
73.3
Table 5. Results for molecule property prediction in terms of ROC-AUC. We report mean and standard deviation over three runs.
we employ the model pre-trained on GuacaMol dataset and select three commonly used physiochemical and binding- related properties: quantitative evaluation of druglikeness (QED), synthesis accessibility (SA), and the activity against target protein Glycogen synthase kinase 3 beta (GSK33),
100
100
90
Validity (%)
detailed in Appendix B.3. The property oracle functions are provided by the Therapeutics Data Commons (TDC)
90
package (Huang et al., 2022).
MCBES
MOSES
60
Guscaaal
GearaMil
As discussed $4.1, we employ two approaches for fine- tuning: (1) rejection sampling fine-tuning and (2) reinforce- ment learning with PPO. Figure 2 shows that both methods effectively push the Iearned distribution to the distri- bution of interest. Notably, RFT, with up to three rounds of SBS, significantly shifts the distribution towards de- sired one. In contrast, PPO, despite biasing the distribution, suffers from the over-regularization from the base policy, which aims for training stability. In the most challenging case (GSK3,9), PPO fails sampling data with high rewards. Conversely, RFT overcomes the barrier in the second round (nF ), where its distribution becomes flat across the range and quickly transitions to high-reward distribution.
1M
10M
SSM
308M
707M
155
100
Model
ISSqqeecess per graph
Figure 3. Model and data scaling effects.
as 3D information or chirality, G2PT is trained exclusively on 2D graph structural information. Overall, these results indicate that G2PT not only excels in generation but also learns effective graph representations.
5.7. Scaling Effects
We analyze how scaling the model size and data size will affect the model performance using the three molecular datasets. We use the validity score to quantify the model performance. Results are provided in Figure 3.
5.6. Predictive Performance on Downstream Tasks
We conduct experiments eight graph classification bench- mark datasets from MoleculeNet (Wu et al., 2018a), strictly following the data splitting protocol used in GraphMAE (Hou et al., 2022a) for fair comparison. A detailed descrip- tion of these datasets provided Appendix B.4.
For model scaling, we additionally train G2PTs with 1M, 707M, and 1.5B parameters. We notice that as model size increases, validity score generally increases and saturates at some point, depending on the task complexity. For in- stance, QM9 saturates at the beginning (IM parameters) while MOSES and GuacaMol require more than 85M (base) parameters to achieve satisfying performance.
For downstream fine-tuning, we initialize G2PT with param- eters pre-trained on the GuacaMol dataset, which contains molecules with up to 89 heavy atoms. We also provide results where models are not pre-trained.
For data scaling, we generating multiple sequences from the same graph to improve the diversity of the training data. The number of augmentation per graph is chosen from {1, 10, 100). As shown, one sequence per graph is insufficient to train Transformers effectively, and improving data diversity helps improve model performance. Similar to model scaling, performance saturated at some point when enough data are used.
As summarized in Table 5, G2PT's graph embeddings demonstrate consistently strong (best or second-best) per- formance on seven out of eight downstream tasks, achieving an overall performance comparable to GraphMAE, lead- ing self- supervised learning (SSL) method. Notably, while previous SSL. approaches leverage additional features such
8

---

Graph Generative Pre-trained Transformer
6. Conclusion
References
This work revisits the sequential approach to graph gen- eration and proposes novel token-based representation that efficiently encodes graph structures via node and edge tokens. This representation serves the foundation for the proposed Graph Generative Pre-trained Transformer (G2PT), an auto-regressive model that effectively models graph sequences using next-token prediction. Extensive evaluations demonstrated that G2PT achieves remarkable performance across multiple datasets and tasks, including generic graph and molecule generation, well as down- stream tasks like goal-oriented graph generation and graph property prediction. The results highlight G2PT's adapt- ability and scalability, making it a versatile framework for various applications. One limitation of our method is that G2PT is srder-sensitive, where different graph domains may prefer different edge orderings. Future research could be done by exploring edge orderings that are more universal and expressive.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Austin, J., Johnson, D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34: 14:17981-1799 2021.
Bacciu, D., Micheli, A., and Podda, M. Edge-based se- quential graph generation with recurrent neural networks. Neurocomputing, 116:177-189 2020.
Bagal, V. and Aggarwal, R. Liggpt: Molecular generation using a transformer-decoder model.
Bergmeister, A.. Martinkus, K., Perraudin, N., and Wat- tenhofer, R. Efficient and scalable graph genera- tion through iterative local expansion. arXiv preprint arXiv:2312.1 11599, 2023.
Impact Statement
Bergmeister, A., Martinkus, K., Perraudin, N., and Wat- tenhofer, R. Efficient and scalable graph generation through iterative local expansion, 2024. URL https: //arxiv. org abs 221222122122 11529.
This paper introduces a framework that models graphs a similar vein to GPT (Generative Pre-trained Transformer). The G2PT framework allows seamless plantation of training techniques that have developed in other domains based on GPT. Besides performing generative tasks such as drug dis- covery, G2PT also can be easily extended for discriminative tasks such as graph property prediction. We hope this work will advance the field of graph learning. As a powerful tool, G2PT may also be used as one step in a complex system to create molecule structures harmful to humans the en- vironment, but we don't to see immediate hazards from our study.
Brown, N., Fiscato, M., Segler, M. H., and Vaucher, A. C. Guacamol: Benchmarking models for de novo molec- ular design. Journal of Chemical Information and Modeling, 19(6): 1108, March 2019. ISSN 1549- 960X. doi: 10. http:///cc.ccm..bnnn.... //dx. org/o0ggg//..o 10t. acc jeim 8b00839.
URL http:
Camphell, A., Benton, J., De Bortoli, V., Rainforth, T., Deli- giannidis, G., and Doucet, A. continuous time frame- work for discrete denoising models. Advances Neural Information Processing Systems, 35:28266-28279 2022.
Acknowledgment
We thank all reviewers for their insightful feedback. Chen and Liu's work was supported by NSF 2239869, He and Xu's work was support by NSF 2239672.
Campbell, A., Yim, J., Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402. 04990, 2024.
Chen, D., O'Bray, L., and Borgwardt, K. Structure-aware transformer for graph representation learning. In Interna- tional Conference on Machine Learning, pp. 3469-3489, PMLR, 2022a.
Chen, X., Han, X., Hu, Ruiz, J., and Liu, L. Order mat- ters: Probabilistic modeling of node sequence for graph generation. arXiv preprint arXiv:21066 10Xx:2106.0611 201.0006.199 06189,6666600 2021.
Chen, X., Li, Y., Zhang, A., and Liu, L.-p. Nvdiff: Graph generation through the diffusion node vectors. arXiv preprint 1. 2022b.

---

Graph Generative Pre-trained Transformer
Chen, X., He, Han, X., and Liu, L.-P. Efficient and degree- guided graph generation via discrete diffusion modeling. arXiv preprint 7xXx::2305504111,, 2023.
Esser, Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 18873-12883, 2021.
Chen, X., Wang, Y. Du, Y., Hassoun, S., and Liu, L. On separate normalization in self-supervised transformers. Advances in Neural Information Processing Systems, 36, 2024.
Gao, Z., Dong. D., Tan, Xia, J., Hu, B., and Li, S. Z. A graph is worth k words: Euclideanizing graph using pure transformer. arXiv preprint arXiv:2402.02444
2024.
Dai. H., Nazi, A.. Li, Y., Dai, B., and Schuurmans, D. Scalable deep generative modeling for sparse graphs. In International conference on machine learning, pp. 2302- 2312. PMLR, 2020.
Gat, I,, Remez, T., Shaul, N., Kreuk, F., Chen, R. T., Syn- naeve, G., Adi, Y., and Lipman, Y. Discrete flow match- ing. arXiv preprint arXiv:2407.15544 2024.
Haefeli, K. K., Martinkus, K., Perraudin, N., and Wat- tenhofer, R. Diffusion models for graphs benefit from discrete state spaces. arXiv preprint arXiv:2210001549, 2022.
De Cao, N. and Kipf, T. Molgan: An implicit genera- tive model for small molecular graphs. arXiv preprint orXiv:1805.11973. 2018.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 1iXiv:1810.0455 2018.
Han, X., Chen, X., Ruiz, F. J. R., and Liu, L.-P. Fit- ting autoregressive graph generative models through maximum likelihood estimation. Journal of Machine Learning Research, 24(97):1-30, 2023. URL http: html.
Diamant, N. L., Tseng, A. M., Chuang. K. V., Biancalani, T., and Scalia, G. Improving graph generation by restrict- ing graph bandwidth. In International Conference Machine Learning, pp. 7939-7959. PMLR, 2023.
/ l//mmr wwt..o///pppppp/////ee.e.
Hihi, S. and Bengio, Y. Hierarchical recurrent neural net- works for long-term dependencies. Advances in neural information processing systems, 8, 1995.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Du, Y., Wang, S., Guo, X., Cao, H., Hu, S., Jiang, J., Var- ala, A., Angirekula, A., and Zhao, L. Graphgt: Machine learning datasets for graph generation and transformation. In Thirty-fitth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2).
Hou, Z., Liu, X., Cen, Y., Dong, Y., Yang, H., Wang, C., and Tang, J. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 594 604, 2022a.
2021.
Du, Y., Jamasb, A. R., Guo, J., Fu, T., Harris, C., Wang, Y., Duan, C., Liò, P., Schwaller, P., and Blundell, T. Ma- chine ceaning-aided generative molecular design. Nature Machine Intelligence, pp. 1-16, 2024.
Hou, Z.,, Liu, X., Cen, Y., Dong, Y., Yang, H., Wang, C., and Tang, J. Graphmae: Self-supervised masked graph autoencoders, 2022b. URL https://arxiv. abs/2200 abs/2205.1082
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., and Leskovec, J. Strategies for pre-training graph neural networks, 2020a. URL https: /ttp:s/////xvviioia /bs/ 1905.12263
Dwivedi, V. P. and Bresson, X. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.
Hu, Z., Dong, Y., Wang, K., and Sun, Y. Heterogeneous graph transformer. In Proceedings of the web conference 2020, pp. 2704-2710, 2020h.
Eijkelboom, F., Bartosh, G., Naesseth, C. A., Welling, M., and van de Meent, J.-W. Variational flow matching for graph generation. arXiv preprint axXiv:2406.04843, 2024.
Huang, K., Fu, T., Gao, W., Zhao, Y., Roohani, Y., Leskovec, J., Coley, C. W., Xiao, C., Sun, J., and Zimik, M. Artificial intelligence foundation for therapeutic science. Nature chemical biology, 18(10): 2022.
10

---

Graph Generative Pre-trained Transformer
Jang, Y., Lee, S., and Ahn, S. A simple and scalable representation for graph generation. arXiv preprint arXiv:2312.02230 2023.
Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., and Tang. J. Pre-training molecular graph representation with 3d geometry, 2022a. URL https: 2110 07728.
abs
Jo, J., Lee, S., and Hwang, S. J. Score-based generative modeling of graphs via the system of stochastic differen- tial equations. In International conference on machine learning, pp. 10362-10333 PMLR, 2022a.
Liu, X., Gong, C.,, and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint N.Niv:2200000000 2022b.
Jo, J., Lee, S., and Hwang, S. J. Score-based generative modeling of graphs via the system of stochastic differen- tial equations, 2022b. URL abs/ 2bs/2202..0 02514.
Liu, Z., Lu, M.. Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overopti- mization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405. 16436, 2024.
Loshchilov, and Hutter, Decoupled weight decay regu- larization, 2019. https:// 1711.0511 Luo, Y., Yan, K., and Ji, S. Graphdf: A discrete flow model for molecular graph generation. In International conference machine learning, pp. 7192-7203. PMLR, 2021.
Jo, J., Kim, D., and Hwang, S. J. Graph generation with dif- fusion mixture. arXiv preprint arXiv:2302.03596, 2023.
abs//
Kim, J., Nguyen, D., Min, S., Cho, S., Lee, M., Lee, H., and Hong, S. Pure transformers are powerful graph learners. Advances in Neural Information Processing Systems, 35: 14582-14595, 2022.
Kreuzer, D., Beaini, D., Hamilton, W., Léturrneau, V., and Tossou, Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34:21618-2166 216299, 2021.
Madhawa, K., Ishiguro, K., Nakago, K., and Abe, M. Graph- nvp: An invertible flow model for generating molecular graphs. arXiv preprint arXiv: 1905: 1600, 2019.
Martinkus, K., Loukas, A., Perraudin, N., and Wattenhofer, R. Spectre: Spectral conditioning helps to overcome the expressivity limits of one-shot graph generators, 2022. URL https: /ttp///axvvv////////////e/.//aaa org/ / abs / 2004.000000 01613.
Lezama, J., Salimans, T., Jiang. L., Chang, H., Ho, J., and Essa, I. Discrete predictor-cone cooreccor diffusion models for image synthesis. In The Eleventh International Confer- ence on Learning Representations, 2022.
Min, E., Chen, R., Bian, Y., Xu, T., Zhao, K., Huang, W., Zhao, P., Huang, J., Ananiadou, S., and Rong, Y. Trans- former for graphs: An overview from architecture per- spective. arXiv preprint arXiv:2202.08455: 2022.
Li, P. and Leskovec, J. The expressive power of graph neural networks. Graph Neural Networks: Foundations, Frontiers, and Applications, pp. 63 33-98,77 2022.
Li, Y., Vinyals, O., Dyer, C., Pascanu, R., and Battaglia, P. Learning deep generative models of graphs. arXiv preprint arXiv:180.3 0803.03324, 2018.
Niu, C., Song, Y., Song, J., Zhao, S., Grover, A., and Ermon, S. Permutation invariant graph generation via score-based generative modeling, In International Conference on Ar- tificial Intelligence and Statistics, pp. 4474-4484. PMLR, 2020.
Liao, R., Li, Y., Song, Y., Wang, S., Hamilton, W., Du- venaud, D. K., Urtasun, R., and Zemel, R. Efficient graph generation with graph recurrent attention networks. Advances in neural information processing systems. 32, 2019.
Olivecrona, M., Blaschke, T., Engkvist, O., and Chen, H. Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics, 9:1-14, 2017.
Lipman, Y., Chen, R. T., Ben- Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.020 02.07777 2022.
Polykovskiy, D., Zhebrak, A., Sanchez-Lengeling, B., Golo- vanov, S., Tatanov, O., Belyaev, S., Kurbanov, R., Arta- monov, A., Aladinskiy, V., Veselov, M., Kadurin, A., Jo- hansson, S., Chen, H., Nikolenko, S., Aspuru- Guzik, A., and Zhavoronkov, A. Molecular sets (moses): A bench- marking platform for molecular generation models, 2020. URL https: wwwpaawwwwwww...aaaaaaaaeaeeeeee 128233
Liu, J., Kumar, A., Ba, J., Kiros, J., and Swersky, K. Graph normalizing flows. Advances in Neural Information Pro- cessing Systems, 32, 2019.
Liu. Q., Allamanis, M., Brockschmidt, M., and Gaunt, A. Constrained graph variational autoencoders for molecule design. Advances in neural information processing sys- tems, 31, 2018.
Preuer, K., Renz, P., Unterthiner, T., Hochreiter, S., and Klambauer, G. Fréchet chemnet distance: A metric for generative models for molecules in drug discovery, 2018 URL https://p//// /awxiv..vvvv. .evvvorgg hts//bb//eee wtabs///0...m 0803.09518. 09518.

---

Graph Generative Pre-trained Transformer
Qin, Y., Madeira, M., Thanou, D., and Frossard, P. Defog: Discrete flow matching for graph generation, 2024. URL https: / /awxxiixvvv..//. org/ abs /4100 04263.
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Ge- niesse, C., Pappu, A. S., Leswing, K., and Pande, V. Moleculenet: a benchmark for molecular machine learn- ing. Chemical science, 9()):513-534 2018a.
Radford, A. Improving language understanding by genera- tive pre-training. 2018.
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Ge- niesse, C., Pappu, A. S., Leswing, K., and Pande, V. Moleculenet: A benchmark for molecular machine learning, 2018b. URL https: hatp:://axxiv.ooo org abs/ 1 1703. 00564.
Rampäšek, L., Galkin, M., Dwivedi, V. Luu, A. T., Wolf, G., and Beaini, D. Recipe for a general, powerful, scal- able graph transformer. Advances in Neural Information Processing Systems, 35:14501-14515, 2022.
Wu, Z., Jain, P., Wright, M., Mirhoseini, A., Gonzalez, J. E., and Stoica, I. Representing long-range context for graph neural networks with global attention. Advances in Neural Information Processing Systems, 34:1 12:66 13279, 2021.
Schulman, J., Moritz, P.. Levine, S., Jordan, M., and Abbeel, P. nigh-dimeessioal continuous control using generalized advantage estimation. arXiv preprint 1eXiv:1506.02-55 2015.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprini arXiv: (atl::077006444 2017.
Xu, Z., Qiu, R., Chen, Y., Chen, H.,, Fan, X., Pan, M., Zeng, Z., Das, M., and Tong, H. Discrete-state continuous- time diffusion for graph generation. arXiv preprint urXiv:2405: 1141, 2024.
Sherstinsky, A. Fundamentals of recurrent neural network (rnn) and long short-term memory (Istm) network. Phys- ica D: Nonlinear Phenomena, 404:132306, 2020.
Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., and Liu, T.-Y. Do transformers really perform badly for graph representation? Advances in neural information processing systems, 34:28877-28888,, 2021.
Simonovsky, M. and Komodakis, N. Graphvae: Towards generation of small graphs using variational autoencoders. In Artificial Neural Networks and Machine Learning- ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part I 27, pp. 412-422. Springer, 2018.
You, J., Liu, B., Ying, Z., Pande, V., and Leskovec, Graph convolutional policy network for goal-directed molecu- lar graph generation. Advances in neural information processing systems, 31, 2018a.
You, J., Ying, R., Ren, X., Hamilton, W., and Leskovec, J. Graphmn: Generating realistic graphs with deep auto- regressive models. In International conference mu- chine learning, pp. 5708-5717, PMLR, 2018b.
Siraudin, A., Malliaros, F. D., and Morris, C. Cometh: A continuous- discrete-state graph diffusion model. arXiv preprint 2406::2400.044444 2024.
Sun, F-Y.,, Hoffmann, J., Verma, V., and Tang. J. Info- graph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximiza- tion, 2020. URL https: / arxiv. org/ abss//b//// /bbs////... 01000.
You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., and Shen, Y. Graph contrastive learning with augmentations, 2021. URL https: / arxiv org/ abs 2010..3902.
Zang. C. and Wang, F. Moflow: an invertible flow model for generating molecular graphs. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 617-626, 2020.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Vignac, C., Krawczuk, I., Siraudin, A., Wang, B., Cevher, V., and Frossard, P. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.1477 2022.
Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., Liu, Y., Jin, S., Liu, Q., Zhou, Y., et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint 2023.
arXiv:2307.00
Wang, Y.. Chen, X., Liu, L., and Hassoun, S. Madgen mass-spec attends to de novo molecular generation, 2025.
7hu, Y., Du, Y., Wang, Y., Xu, Y., Zhang, J., Liu, Q., and Wu, S. survey on deep graph generation: Methods and applications. In Learning on Graphs Conference, pp. 47-1. PMLR, 2022.
URL httht: http:://axxvvvvi org/ abs/ 2501. 01950.
Wu, M., Chen, X., and Liu, L.-P. Edge++: Improved training and sampling of edge. arXiv preprint arXiv:2310.14411, 2023.
Zhu, Y., Chen, D., Du, Y., Wang, Y., Liu, Q., and Wu, S. Molecular contrastive pretraining with collaborative
12

---

Graph Generative Pre-trained Transformer
featurizations. Journal of Chemical Information and Modeling, 64(4):1112-1122, 2024.
13

---

Graph Generative Pre-trained Transformer
A. Reinforcement Learning Details
A.1. Preliminaries on Proximal Policy Optimization (PPO)
Generalized Advantage Estimation. In reinforcement learning, the Q function Q(sci, si) captures the expected returns when taking an action 8t at current state Scl, and the value function V(scr) captures the expected return following the policy from a given state S-1-
The advantage function A<))) defined as the difference between the Q function and the value function, measures whether taking action is better or worse than the policy's 's default behavior. In practice, the Q function is estimated using the actual rewards and the estimated future returns (the value function). There are two commonly used estimators, one is the one-step Temporal Difference (TD):
88) = = (V(8ct+1)
i(sc1, 81) = = + +V(( - V(sai),
and the full Monte Carlo (MC):
(1))) =
1' -rrr,,
A(s<1,81)
Σ 'r V(sci),
assuming finite trajectory with L steps in total. However, the TD estimator exhibits high bias and the MC estimator exhibits high variance. The Generalized Advantage Estimation (GAE) (Schulman al., 2015) effectively balances the high bias and high variance smoothly using trade-off parameter 7. Let d; Ti (5((1=++1)) (S<z)), the definition of GAE is:
L Σ (T)))" 5p = dr + Á 8111-11)1))
(7 81)
GAE plays an important role in estimating the policy gradient, and will be used in the PPO algorithm.
Proximal Policy Optimization. PPO (Schulman et al., 2017) is a fundamental technique in reinforcement learning, designed to train policies efficiently while preserving stability. It is built on the principle that gradually guides the policy towards an optimal solution, rather than applying aggressive updates that could compromise the stability of the learning process.
In traditional policy gradient methods, the new policy should remain close to the old policy in the parameters space. However, proximity in parameter space does not indicate similar performance. A large update step in policy may lead to falling "off the cliff", thus getting bad policy. Once is stuck in bad policy, it will take very long time to recover.
PPO introduces two kinds of constraints on policy updates. The first kind add KL-regularization term to the policy gradient "surrogate" objective
Cpp-penatty (0) E, (o((()) A (-I Pers (s1(8<2)
Here E- is the empirical average over finite batch of samples where sampling and optimization alternates. 8 is the penalty factor. A, == (s s1) is GAE, which is detailed in last section.
The second type is the clipped surrogate objective, expressed as
(P((s))<<)) A,, clip Penu (s1|s
( (o(())B<1) - 1011-,,11 A
c,1+c) ),1,), )))++)))))
)
Cpp-cti (0)
E, min
Pona (st (1))))))
where Pa(ai)az)) Paay [2]) is the probability ratio between the and the old policy. And decides how much the new policy can deviate from the one policy. The clipping operations prevent the policy from changing too much from the older one within one iteration. In the following, we elaborate on how the critic model is optimized.
14

---

Graph Generative Pre-trained Transformer
QM9
MOSES
GuacaMol
Planar
Tree
Lobster
SBM
#Node Types
4
8
12
1
1
#Edge Types
4
4
1
1
Avg. #Nodes
8.79
21.67
27.83
64
53
104.01
Min. #Nodes
1
8
64
64
10
44
Max. #Nodes
9
27
88
64
64
100
187
#Training Sequences
9,773,200
141,951,200
111,863,300
12,800,000
10,000,000
12,800,000
12,800,000
Vocabulary Size
27
60
120
73
73
110
195
Max Sequence Length
85
207
614
737
383
599
3950
Table 6. Dataset statisties.
Value Function Approximation. The critic model (s<x) PPO algorithm is used approximate the actual value function (*(s<)). We the mean absolute value loss to minimize the difference between the predicted values and the actual return values. Specifically, the objective is
Leritic (v)) (V/(V(() - V (sa)))
Here the actual return value is estimated using GAE balance the bias and variance:
((s<i) A(sę1, Vern (sci),
where Vilu (8<1) is collected during the sampling step in PPO. The critic loss is weight by factor
A.2. KL-regularization
As mentioned in $4.1, we adopt KL-regularized reinforcement learning approach. Unlike the KL penalty in Cpe-penatty (Ø), this regularizer ensures that the policy model Pş does not diverge significantly from the reference model po. Instead of optimizing this term directly, we incorporate it into the rewards rı- Specifically, we define:
r = ri - PiKL(pp (r)) (i)))
where Pi is the penalty factor. In practice, P1 is set to small value, such 0.03, to promote exploration.
A.3. Pre-training loss
Following Zheng et al. (2023) and Liu et al. (2024), we incorporate the pre-training loss Lp(()) o mitigate potential degeneration in the model's ability produce valid sequences. This is partieularly beneficial for helping the actor model recover when it "falls off the grid" during PPO. The pre- pre-training data sourced from the dataset used to train the reference model, and the loss (x) is weighted by the coefficient P3.
B. Additional Experimental Details
B.1. Graph Generative Pre-training
Generative pre-training leverages graph-structured data to learn foundational representations that can be fine-tuned for downstream tasks.
Sequence conversion. We convert graphs into sequences of tokens that represent nodes and edges. This transformation involves encoding the molecular structure in sequential format that captures both the composition and the order of assembly, For instance, we iteratively process the nodes and edges, and insert special tokens to mark key points in the sequence, such as the start and end of generation. Additionally, we apply preprocessing steps like filtering molecules by size, removing hydrogens, or addressing dataset-specific constraints to ensure consistency and suitability for the target tasks.
Data splitting. We divide generic datasets into training, validation, and test sets based on splitting ratios 6:2:2. For the molecular datasets, we follow the default settings of the datasets.
15

---

Graph Generative Pre-trained Transformer
10M
85M
300M
Architecture
#layers
6
12
24
#heads
6
12
16
mmddel
384
768
1024
dropout rate
Training
Lr
le-4
Optimizer
AdamW (Loshchilov & Hutter, 2019)
Lr scheduler
Cosine
Weight decay
le-1
#iterations
300000
Batch size
60
60
30
#Gradient Accumulation
8
8
Grad Clipping Value
#Warmup Iterations
2000
Table 7. Hyperparameters for graph generative pre-training.
Dataset statistics. The vocabulary size, maximum sequence length, and other parameters vary across datasets due to their distinct molecular characteristics. We summarize the specifications in Table 6, which includes details the number of node types, edge types, and graphs for each dataset.
Hlpperaarameters Table provides hyperparameters used for training three distinct model sizes, corresponding to approximately 10M, 85M, and 300M parameters, respectively.
B.2. Demonstration Experiment
We elaborate on how to represent adjacency matrix as sequence and train a transformer decoder on it. We choose planar graphs as the investigation object as requires model to be able to capture the rule embedded in the graph. We use G2PT small for this experiment.
Sequence conversion. We convert 2-D adjacency matrix into 1-D sequence before training the models. Similar to GraphRNN (You et al., 2018b), we consider modeling the strictly lower triangle of the adjacency matrix. To obtain sequence, we flatten the triangle by concatenating the rows together. The i-th row has I entries, where each entry is either or 1. We employ BFS to determine the node orderings, which is used to permute the rows and columns of the adjacency matrix to reduce the learning complexity (as uniform orderings are generally harder to fit (Chen et al., 2021)).
Training transformers adjacency matrices. After obtaining the sequence representation, prepend and append two special tokens, [SOG] and [EOG], to mark the start and end of the generation of each sequence. The sequence is then tokenized using vocabulary of size 4, and the transformer is trained on these sequences. Note that no additional token is needed to indicate transitions between rows, as the flattened sequence maintains fixed correspondence between positions and the referenced node pairs. Specifically, the original row and column indices in the adjacency matrix for the i-th entry in the sequence can determined as: 1 +
/1 8i
(row 1)(oow 2)
.col
2
2
Here [-] is the ceiling operation. Such correspondence is agnostic to graph size and can be inferred by transformers by using positional embeddings.
16

---

Graph Generative Pre-trained Transformer
QED
SA
GSK33
7
1.0
A
1.0
P1
0.5
P2
0.03
0.03
0.05
0.03
Advantage Normalization and Clipping
Yes
No
No
Reward Normalization and Clipping
No
Yes
Yes
Ratio Clipping (e)
[0.2]
Critic Value Clipping
[0.2]
Entropy Regularization
No
Gradient Clipping Value
1.0
Actor Lr
1.0
Critic Lr
0.5
0.5
1.0
#lterations
6000
Batch size
60
Table 8. Hyperparameters used for PPO training.
B.3. Fine-tuning G2PT for Goal-oriented Generation
For the goal-oriented generation, we fine-tune G2PT to generate molecules with desired characteristics. Specifically, we consider three properties that are commonly used for molecule optimization whose functions are easily accessed through the Therapeutics Data Commons (TDC) package (Huang et al., 2022).
Quantitative evaluation of druglikeness (QED): range 0-1, the higher the more druglike.
Synthesis accessibility (SA) score: range 1-10, the lower the more synthesizable.
GSK333 activity against target protein Glycogen synthase kinase 3 beta, range 0-1, the higher the better activity.
We use the 85M model pre-traine pre-trained on GucaMol dataset for all experiments. Below elaborate how the RFT and RL algorithms implement each optimization task (property).
Rejection-sampling fine-tuning. For RFT algorithm without SBS, we begin by generating samples using the pre-trained model and retain only those that meet the criteria defined by the acceptance function m (.). We collect 200,000 qualified samples from the generations. Then, we fine-tune the model by initializing it with pre-trained parameters. When combining RFT with SBS, we repeat this process iteratively, using the fine-tuned model from the previous iteration for both sampling and parameter initialization.
For QED score, we retain samples with scores exceeding thresholds of 0.4, 0.6, 0.8, or 0.9. We do not use the SBS algorithm here, as the pre-trained model generates samples efficiently across all QED score ranges.
For SA score, we consider thresholds of { < 3.0, < 2.0, < 1.5]. We find that the pre-trained model efficiently generates molecules with SA scores below 2.0 and 3.0 but struggles with scores below 1.5. address this, we bootstrap the fine-tuned model from the 2.0 threshold to the 1.5 threshold.
For GSK38, consider thresholds in {> 0.2,> 0.4,> 0.6,> 0.8). We observe that the pre-trained model's score distribution is skewed towards 0, making it challenging to generate satisfactory samples. To resolve this, we fine-tune the model at the 0.2 threshold and progressively bootstrap it through intermediate thresholds (0.4, 0.6) up to 0.8, performing three bootstrapping steps in total.
All models are trained for 6000 iterations, with batch size of 120 and learning rate of le-5. The learning rate gradually decay using Cosine scheduler.
17

---

Graph Generative Pre-trained Transformer
Reinforcement learning. We use the PPO algorithm further optimize the pre-trained model. In practice, the token-level reward R((s<1,)1) st]) is set 0 except when 8; [EOG]. The final reward r(s) for the three properties are designed as follow:
QRD (s) = G max max (0.2, x (QED(G) 0.5)),
(2)
(s) = L max(0.2, 0.2 x (5 - SA(G)),
(3)
y GSK38 (s) (5 (GSK3,3(G))
(4)
The indicator function s- assigns 0 to the final reward when the generated sequence s is invalid. We show the PPO hyperparameters for each targeted task in Table
B.4. Fine-tuning G2PT for Graph Property Prediction
Datasets. We use eight classification tasks in MoleculeNet (Wu et al., 2018a) following Zhu et al. (2024) to validate the predictive capability of our learned representations.
The datasets cover two types of molecular properties: biophysical and physiological properties.
Biophysical properties include (1) the HIV dataset for HIV replication inhibition, (2) the Maximum Unbiased Validation (MUV) dataset for virtual screening with nearst neighbor search, (3) the BACE dataset for inhibition of human 3 secretase (BACE-I), and (4) the Side Effect Resource (SIDER) dataset for grouping the side effects of marketed drugs into 27 system organ classes
Physiological properties include (1) the Blood-brain barrier penetration (BBBP) dataset for predicting barrier perme- ability of molecules targeting central nervous system, (2) the Tox21, (3) the ClinTox, and (4) the ToxCast datasets that are all associated with certain type of toxicity of the chemical compounds.
We adopt the scaffold split that divides train, validation and test set by different scaffolds, introduced by Wu et al. (2018b).
Fine-tuning details. We fine-tune G2PT small and G2PT base pre-trained on GuacaMol dataset for the downstream tasks. We setup the dropout rate to 0.5 and use learning rate of le-4 for training the linear layer. For the half transformer blocks, we use a learning rate of 1e-6. We use batch size of 256 and train the models for 100 epochs. Test result with best validation performance reported.
B.5. Baselines
We evaluate our proposed method against a variety of baselines across different datasets. The baselines include models that span diverse methodologics, ranging from graph neural networks to transformer-base architectures.
Generic graph datasets. The performance baseline models on Planar, Tree, Lobster, and SBM datasets is shown in Table 2. We consider baselines mainly from categories: auto-regressive and diffusion graph models. Among them, GRAN (Liao al., 2019), BiGG (Dai et al., 2020), and BwR (Diamant al., 2023) are auto-regressive models that sequentially generate graphs. GRAN uses attention-based GNNs to perform block-wise generation, focusing on dependencies between components within the graph. In contrast, BiGG addresses the challenges of efficiency by leveraging the sparsity of real-world graphs to avoid constructing dense representations. Unlike GRAN and BiGG, BwR simplifies the generation process further by restricting graph bandwidth. On the other hand, DiGress (Vignac et al., 2022) and HSpectra (Bergmeister et al., 2023) are built based on diffusion frameworks. DiGress the first approach that uses a discrete diffusion model to iteratively modify graphs, while HSpectra focuses on multi-scale graph construction by progressively generating graphs through localized denoising diffusion.
Molecule generation datasets. We compare G2PT against four baselines: DiGress (Vignac et al., 2022), DisCo (Xu et al., 2024), Cometh (Siraudin et al., 2024), and DeFoG (Qin et al., 2024). Among them, DisCo and Cometh are both based on a continuous- ciinuuuu-timeeeee discrete diffusion framework, with Cometh additionally incorporating positional encoding for nodes and separate noising processes for nodes and edges. DeFoG adopts a discrete flow matching approach with a linear interpolation noising process.
18

---

Graph Generative Pre-trained Transformer
Graph pre-training methods. We compare against several pre-training approaches for molecular property prediction, as summarized in Table 5. The goal ofGraph pre-training methods is to learn robust graph representations via exploiting the structural information. AttrMask (Hu et al., 2020a) uses attribute masking at both node and graph levels to capture local and global features simultaneously. ContextPred (Hu et al., 2020a) builds on this idea by predicting subgraph contexts, enabling the model to understand patterns beyond individual attributes. Similarly, InfoGraph (Sun et al., 2020) focuses on multi-scale graph representations by maximizing mutual information between graph-level embeddings and substructures. Moving to contrastive learning approaches, GraphCL (You et al., 2021) applies graph augmentations to generate positive and negative samples for representation learning. Building on this idea, GraphMVP (Liu et al., 2022a) incorporates both 2D molecular topology and 3D geometric views, aligning them within contrastive framework to enhance feature representation. In contrast to these methods, GraphMAE (Hou et al., 2022b) adopts generative approach, using a masked graph auto-encoder to reconstruct node features and capture structural information.
B.6. Evaluation
Metrics for molecule datasets. As MOSES and GuacaMol are established benchmarking tools, they provide predefined metrics for evaluating and reporting results. These metrics are briefly outlined as follows: Validity assesses the percentage of molecules that satisfy basic valency constraints. Uniqueness evaluates the fraction of molecules represented by distinct SMILES strings, indicating non-isomorphism. Novelty quantifies the proportion of generated molecules absent from the training dataset. The filter score represents the percentage of molecules that satisfy the same filtering criteria applied during test set construction. The Frechet ChemNet Distance (FCD) (Preuer et al., 2018) quantifies the similarity between molecules in the training and test sets based neural network-derived embeddings. SNN computes the similarity to the nearest neighbor using the Tanimoto distance. Scaffold similarity compares the distributions of Bemis-Murcko scaffolds, and KL divergence measures discrepancies in the distribution of various physicochemical descriptors.
For QM9 dataset, the validity metric reported in this study is calculated by constructing a molecule using RDKit and attempting to generate a valid SMILES string from it, as this approach is commonly employed in the literature. However, as explained by Jo et al. (2022b), this method has limitations, as it may classify certain charged molecules present in QM9 as invalid. To address this, they propose a more lenient definition of validity that accommodates partial charges, offering a slight advantage in their computations.
Metrics for generic graph datasets. We adopt the evaluation framework outlined by (Martinkus et al., 2022) and (Bergmeister et al., 2024), incorporating both dataset-agnostic and dataset-specific metrics. The dataset-agnostic metrics evaluate the alignment between the distributions of the generated graphs and the training data by analyzing general graph properties. Specifically, we characterize graphs based on their node degrees (Deg.), clustering coefficients (Clus.), orbit counts (Orbit), eigenvalues of the normalized graph Laplacian (Spec.), and statistics derived from wavelet graph transform (Wavelet). To quantify the alignment, compute the distances between the empirical distributions of these statistics for the generated and test graphs using Maximum Mean Discrepancy (MMD).
Subsequently, we evaluate the generated graphs using dataset-specific metrics under the V.U.N. framework, which measures the proportions of valid (V), unique (U), and novel (N) graphs. Validity is determined by datasct-specific criteria: graphs must be planar, tree-structured, or statistically consistent with a Stochastic Block Model (SBM) for the planar, tree, and SBM datasets, respectively. Uniqueness evaluates the proportion of non-isomorphic graphs among the generated samples, while novelty quantifies the proportion of generated graphs that are non-isomorphic to any graph in the training set.
B.7. Computation Resources.
We ran all pre-training tasks and all goal-oriented generation fine-tuning tasks run on 8 NVIDIA A 1000SXXM480GBB GPU with distributed training. For PPO training and graph property prediction tasks, we ran experiments using A100 GPU.
C. Extended Related works
C.1. Auto-regressive Graph Generative Models
Even though graph is naturally an unordered set, auto-regressive models generate graphs sequentially, one node, edge, or substructure at time. GraphRNN and DeepGMG (You et al., 2018b; Li et al., 2018) prefix a canonical ordering (e.g.. breath- first search) for the nodes and edges and generates nodes and edges associated with them step by step. On the contrary,
19

---

Graph Generative Pre-trained Transformer
Algorithm 4 Depth-First search edge order generation
Input: Graph G (V; E), neighborhood function Nei.(-).
Output: Sequence of traversed edges dg-
Initialize dE []. sample to from
DES helper (:): for of E Nei(v) do e= (v,v)). if o' is not visited then Append e to gg. Call DFS DFS_DFSSSpeeeee helper(e").
else
if og then
Append to dg. end if end if
end for
Run DFS helppr((o))
Algorithm 5 Breadth-First Search edge order generation
Input: Graph (V. E), neighborhood function Nei(-). Output: Sequence of traversed edges dg-
Initialize oæ + [. sample from V, initialize queue +- [vo]-
while queue is not empty do v < queue.popfirst()
for v' E Ne)(v)) do e = vv)
if is visited then append to dg. append v' queue. else
e og then append to ag. end if end if end for end while
Bacciu et al. (2020) propose to generate edges first then the connected nodes subsequently. These auto-regressive models are also broadly adapted into applications such as molecule generation. GCPN (You et al., 2018a), and REINVENT (Olivecrona al., 2017) both leverage pre-trained auto-regressive models fine-tune with reward model to generate molecules with desired properties.
C.2. reggesvt-re Graph Generative Models
In addition auto-regressive models, -oo--uuo-reeressive graph generative models can be categorized into two branches: (I) one-shot generation and (2) iterative refinement. One-shot generation aims to generate graph in single step including methods such generative adversarial networks (De Cao & Kipf, 2018), variational auto-encoders (Simonovsky & Komodakis, 2018; Liu et al., 2018) and normalizing flows (Madhawa et al., 2019; Zang & Wang, 2020). Nevertheless, one-shot graph generative models often suffer from the decoding strategies such that it requires an expressive decoder to map from latent vectors to graphs. On the other side, iterative refinement methods generate the entire graph in the first step and then iteratively refine the generated graph to be close to realistic graph, including diffusion (Niu et al., 2020; Jo et al., 2022a; Vignac al., 2022; Chen al., 2022b; 2023; Jo al., 2023; Haefeli et al., 2022; Wu al., 2023; Siraudin et al.,
20

---

Graph Generative Pre-trained Transformer
Algorithm 6 Uniform edge order genration
Input: Graph = EV
Output: Sequence of edge ordering dØ Initialize dE x-1
while E' is not empty do
sample from E, append to dE Remove from E end while
Model
Edge Orderings
Validity?
Unique.t
Noveltyt
Filterst
FCD
SNN1
Scaft
Degree-based
95.1
91.7
97.4
0.52
DFS BFS
91.6
100
87.1
98.0
1,2
0.55
8.9
G2PT small
96.2
100
86.8
98.3
1.0
0.55
10.6
Uniform
62.9
100
99.4
52.0
7.0
0.38
9.5
Degree-hased
96.4
100
86.0
98.3
0.97
0.55
3.3
DFS
91.9
100
83.7
98.1
1.13
0.55
7.5
G2PT base
BFS
96.9
84.6
98.7
0.55
11.1
Uniform
80.9
100
97.0
83.9
2.14
0.46
10.3
Table 9. Sensitivity analysis on edge orderings.
2024; Xu et al., 2024; Wang et al., 2025) and flow matching models (Qin al., 2024; Eijkelboom et al., 2024; Lipman et al., 2022; Liu et al., 2022b; Campbell et al., 2024; Gat et al., 2024). As discussed in ??, they often require prefixed number of refinement steps and they need to maintain an adjacency matrix over the trajectory which is computationally intensive.
C.3. Pre-training Transformers for Graphs
Transformers are now dominating domains of natural language processing (NLP) and computer vision (CV) (Radford, 2018;
Devlin, 2018; Dosovitskiy, 2020). Several works also attempt to applying transformers in the field of graph learning (Ying et al., 2021; Hu et al., 2020b; Dwivedi & Bresson, 2020; Rampášek et al., 2022; Chen et al., 2022a; Wu et al., 2021; Kreuzer al., 2021; Min al., 2022; Chen et al., 2024; Gao et al., 2024). Those approaches propose several methods to encode the graph structure information into sequences, specifically, the key research problem lies in how add identifiers nodes and tokenize the edges. For instance, Kim et al. (2022) type embeddings distinguish node and edge tokens. A more recent work, Gao et al. (2024), which focuses molecule representation learning, uses vocabulary to store all atom types and all possible bond types (same bonds with different atoms as endpoints are considered as different type in their case). In contrast, by introducing node index into the vocabulary, G2PT easily implements the edge tokenizations and node identifications.
positional embedding help transformer identify nodes, and
D. A Additional Results
D.1. Sensitivity Analysis of Edge Orderings
We investigate how the employed edge orderings will affect the generative performance of G2PT. Specifically, we consider four orderings: the reverse of edge-removal process (Alg. 1), DFS ordering (Alg. 4), BFS ordering (Alg. 5), and uniform ordering (Alg. 6). We train G2PT small and G2PT base on MOSES dataset and evaluate the performance.
Result. Table 9 reports the performance of different edge orderings. BFS and dagree-based edge-removal orderings both exhibit superior results, while DFS orderings show moderate performance. Particularly, uniform ordering shows poor performance in capturing the sequence distribution. This result highlights the importance of choosing the right ordering families for generating sequences.
D.2. Additional Visualizations
We further visualize the generic graph in Figure 4, and molecular graph in Figure The results show that both G2PT small and G2PT base have the ability to capture the topological rules of the training graphs.
21

---

Graph Generative Pre-trained Transformer
Train
G2PT small
G2PT base
Tree
Lobster
Planar
SBM
Figure 4. The visualization of generic graph datasets
MOSES
goot
Train
G2PT
G2PT hase
GuacaMol
0010k ooo
Train
G2PT small
G2PT
Figure 5, The visualization of molecular datascts
22


Reviews:
[{"review_id": "XBsoimfFnN", "reviewer": "Reviewer_TGsg", "summary": "Authors introduce a new way to represent graph as sequence of tokens, that contains both node definitions and edge definitions. They use this representation and standard transformer architecture trained on next-token prediction task to generate new graphs. The method is competetive to SOTA diffusion and non-autoregressive graph generative methods, which is not the case for older autoregressive approaches. Authors further expand their methods by showing its utility for downstream molecule property prediction tasks as well as RL finetuning for goal oriented generation.\n\n## update after rebuttal\n\nThe authors adressed my main concerns raised, and overall I think this is a good paper. Thus I recommend acceptance", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive review, below we address the raised question/comment.\n\n---\n**Q1**. _Discussion with GEEL (https://arxiv.org/pdf/2312.02230) and comparison_\n\n**A1**. We first provide a discussion with GEEL then provide the experimental result of G2PT and GEEL on 6 graph datasets.\n\n- Discussion: \n\nG2PT and GEEL both transform graphs into edge lists. Typically, an edge list is viewed as a sequence of node pairs, such as $[(s_1, t_1), (s_2, t_2), …, (s_m, t_m)]$. The efficiency of this representation hinges on the construction of the vocabulary. A basic method treats each pair as a distinct “word,” leading to a vocabulary size on the order of $O(N^2)$ for a graph with N nodes, which becomes unscalable and directly depends on the graph size.\n\nTo address this, GEEL reduces the vocabulary size by drawing inspiration from the concept of graph bandwidth B. This idea is based on the observation that, after an appropriate node permutation, only the entries near the diagonal of the adjacency matrix are nonzero, thereby shrinking the vocabulary from $O(N^2)$ to $O(B^2)$.\n\nIn contrast to GEEL, G2PT represents each node pair using two tokens rather than a single token. This multi-token approach provides better flexibility and avoids any assumptions about the graph’s structure, reducing the vocabulary size significantly to $O(N)$.\n\nIt is also important to note that for any graph, a trivial lower bound for the graph bandwidth is given by $bw(G) ≥ \\Delta/2$, where $\\Delta# is the maximum degree of the graph. Thus, while GEEL’s vocabulary size may vary depending on the graph’s pattern, G2PT’s approach remains pattern-agnostic.\n\n- Result Comparison: We compare G2PT with GEEL on generic graphs (Planar, Tree, Lobster, SBM) and molecular graphs (MOSES, GuacaMol). \n\n_Planar_\n\n|      | Deg | Clus   | Orbit  | Spec   | Wavelet | V.U.N. |\n|------|--------|--------|--------|--------|---------|--------|\n| GEEL | **1e-3** | 1e-2 | 1e-3 | - | - | <27.5 |\n| G2PT | 1.8e-3 | **4.7e-3** | **0.00** | 8.1e-3 | 5.1e-3 | **100** |\n\n\n_Tree_\n\n|      | Deg | Clus   | Orbit  | Spec   | Wavelet | V.U.N. |\n|------|--------|--------|--------|--------|---------|--------|\n| GEEL | **1.5e-3** | **0.00** | 2e-4 | 1.5e-2 | **4.6e-3** | 90 |\n| G2PT | 4.2e-3 | **0.00** | **1e-4** | **7.3e-3** | 5.7e-3 | **99** |\n\n\n_Lobster_\n\n|      | Deg | Clus   | Orbit  | Spec   | Wavelet | V.U.N. |\n|------|--------|--------|--------|--------|---------|--------|\n| GEEL | 2e-3 | **0.00**  | 1e-3 | - | - | <72.7 |\n| G2PT | **1e-3** | **0.00** | **0.00** | 4e-3 | 1e-2 | **100** |\n\n\n_SBM_\n\n|      | Deg | Clus   | Orbit  | Spec   | Wavelet | V.U.N. |\n|------|--------|--------|--------|--------|---------|--------|\n| GEEL | 2.5e-2 | **3e-3** |2.6e-2 | - | - | <42.5 |\n| G2PT | **4.2e-3** | 5.3e-3 | **3e-4** | 6.1e-3 | 6.9e-3  | **100** |\n\n\n_MOSES_\n\n|      | Validity | Unique   | Novelty  | Filters   | FCD | SNN | Scaf |\n|------|--------|--------|--------|--------|---------|--------|--------|\n| GEEL | 92.1 | **100** | 81.1 | 97.5 | 1.28 | 0.52 | **3.6** |\n| G2PT | **96.4** | **100** | **86.0** | **98.3** | **0.97** | **0.55** | 3.3 |\n\n_GuacaMol_\n\n|      | Validity | Unique   | Novelty  | KL Div.   | FCD |\n|------|--------|--------|--------|--------|---------|\n| GEEL | 88.2 | 98.2 | 89.1 | 93.1 | 71.5 |\n| G2PT | **94.6** | **100** | **99.5** | **96.0** | **93.4** |\n\nWe will include the discussion as well as the experiment result in our next version.\n\n---\n\n**Q2**. _V.U.N reference correction_\n\n**A2**. Thanks for correcting this! We will update the reference in the next version.\n\n---\n\n**Q3**. _Can the authors also expand on tie-breaking in Algorithm 1? How is the problem solved when multiple nodes have the same degree? Are ties broken randomly and ordering is different in each epoch for the same sample or are they broken in a fixed manner based on original graph IDs._\n\n**A3**. Excellent question! Yes we break the tie randomly and will obtain a different order every epoch. We observe that using more than 10 orders for every graph leads to better generalization and performance (similar to data augmentation). We kindly refer to the reviewer in Figure 3 for how using more orders affects the overall validity of molecular graphs.\n\n---\n\nLet us know whether we address your concerns, thanks!", "claims_and_evidence": "I find all claims to be well validated experimentally using established benchmarks.", "methods_and_evaluation_criteria": "Authors use a wide range of well established graph generation benchmarks that are good for testing graph generation approaches.", "theoretical_claims": "The theoretical claims are quite standard and rely on well known methods and thus are sound.", "experimental_designs_or_analyses": "All experiments follow standard practices and use standard datasets.", "relation_to_broader_scientific_literature": "The paper revisits autoregressive graph generation and shows that with modern neural netrowks and smarter data representation the autoregressive methods can be competetive to the current SOTA diffusion-based graph generative methods. They do skip one very relevant related work as per below.", "essential_references_not_discussed": "Authors failed to discuss and compare to probably the most comparable work in the literature to theirs: https://arxiv.org/pdf/2312.02230 which also proposes a new way to represent graphs for autoregresive generation.", "other_strengths_and_weaknesses": "No other weaknesses. The method is quite straight forward, the validation is quite extensive and the results look good and are in line with other SOTA approaches, which is good enough for a method resurecting a much older way of doing things (autoregressive generation).", "other_comments_or_suggestions": "A small note is that in lines 327-328 authors quote DiGress as source for valid, unique and novel samples (V.U.N.) metrics, while they were originally introduced in SPECTRE (https://arxiv.org/abs/2204.01613).", "questions_for_authors": "I'd like to see a comparison to https://arxiv.org/pdf/2312.02230.\n\nCan the authors also expand on tie-breaking in Algorithm 1? How is the problem solved when multiple nodes have the same degree? Are ties broken randomly and ordering is different in each epoch for the same sample or are they broken in a fixed maner based on original graph IDs.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gy8M0oxZKM", "reviewer": "Reviewer_rB8b", "summary": "This paper proposes Graph Generative Pre-trained Transformer (G2PT) as a novel approach to molecular graph generation models. While conventional graph generation models are primarily adjacency matrix-based, this method treats node and edge lists as token sequences and employs an autoregressive Transformer (Transformer Decoder) for efficient learning. To extend G2PT as a general-purpose foundation model, it explores fine-tuning techniques for two downstream tasks: goal-oriented generation and graph property prediction. Computational experiments demonstrate that G2PT achieves state-of-the-art (SOTA) performance in molecular generation (QM9, MOSES, GuacaMol), general graph generation (Planar, Tree, Lobster, SBM), and downstream tasks (MoleculeNet).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We believe the suggestion are very constructive in improving the quality of our draft, below we address the raised concern.\n\n---\n\n**Q1**. _On the other hand, the way of representing graphs as token sequences is very straightforward and cannot be said to build upon previous research. The extent to which \"learning as a token sequence\" and \"learning through autoregressive next-token prediction\" individually contribute remains unclear, and the evidence for each aspect appears somewhat insufficient._\n\n**A1**. Yes, the way we represent graphs as tokens is both straightforward and novel as it is not built upon previous research. This is because previous token-based approaches have focused on **learning representations for graph/node/edge**, while ours focus on **generating graphs**. \n\nRegarding the contributions of each component, obtaining the token sequence is a necessary first step for applying next-token prediction. As a result, it is challenging to separate their individual impacts. Moreover, none of the earlier token-based methods are suitable for next-token prediction learning, which is why we developed this new tokenization approach. We will further elaborate on this in our response to your next question.\n\n---\n\n**Q2**. _At the very least, there should be a discussion on how this method relates to existing token-based approaches - such as those highlighted in well-cited reviews like https://arxiv.org/abs/2302.04181._\n\n**A2**. Thank you for sharing the comprehensive survey covering previous graph-based transformer methods. We have thoroughly reviewed these approaches. Below we summarize what distinguish our method from them.\n\n- **Task**: G2PT is designed specifically for graph generation, whereas the token-based approaches discussed in the review primarily target graph representation learning. This fundamental difference necessitates distinct tokenization strategies for our task.\n\n- **Tokenization**: Correspondingly, G2PT’s tokenization is **invertible** between graph and token sequence. That said, we can obtain the graph by de-tokenizing the graph. While the previous token-based approaches only address how to represent graphs into tokens (node / node+edge / subgraph), translating token sequence back to graph remains unclear.\n\nGPT-like models have demonstrated tremendous success across various domains. However, attempts on utilizing them in graph generation remain inadequate. We believe the barrier lies in how to efficiently tokenize and de-tokenize graphs, which our work directly addresses.\n\n---\n\n**Q3**. _or, if specializing in molecular graphs, a comparison with simpler methods that feed traditional symbol-sequence representations of molecules like SMILES or SELFIES to Transformers._\n\n**A3**. G2PT is more generic in representing graphs, making it  more flexible thus more suitable for various molecule-related applications compared to traditional symbol-sequence. Such applications include **constrained generation**, **molecule in-painting**, **retrosynthesis**, etc. (https://arxiv.org/pdf/2502.09571, https://arxiv.org/abs/2308.16212)\n\nIn contrast, since symbol-sequences like SMILES/SELFIES are canonical for graphs, modifying any node or edge on the graphs will lead to a transformative change on the sequence's representation. Methods that operate on SMILES/SELFIES need to employ a seq-to-seq approach to generate a new molecule from scratch given the condition one. \n\nG2PT's principle of tokenizing a graph is more general and can be further extended beyond the one we define in the paper. By introducing addition/deletion/replace actions, alteration can be easily performed on the original molecule via defining a sequence of action. Such action trajectory well aligns with the process of lead optimization. And we believe such agentic paradigm would be more universal and scalable.\n\n---\n**Q4**. _The experimental datasets and tasks are well-balanced, incorporating molecular generation benchmarks such as MOSES and GuacaMol, as well as generic datasets. However, the QM9 dataset contains molecular data with 3D coordinates, which may not be the best fit for the proposed method. A more careful consideration of this choice may be necessary._\n\n**A4**. We agree that QM9 contains 3D information. The reason we chose QM9 is to follow prior works (DiGress, DeFoG, Cometh) to provide a more comprehensive comparison. \n\n---\n\nWe hope that we have addressed your concern. Let us know if you have any following-up questions.", "claims_and_evidence": "The paper presents experimental results demonstrating that the proposed token sequence-based graph generation approach is more efficient than existing adjacency matrix-based methods. It also confirms that G2PT achieves performance comparable to or exceeding state-of-the-art (SOTA) models based on generation quality metrics such as MMD, Validity, Novelty, and FCD. Additionally, for goal-oriented generation, fine-tuning G2PT using rejection sampling (RFT) and PPO (reinforcement learning) effectively enhances target graph properties such as QED, SA, and GSK3β.\n\nOn the other hand, the way of representing graphs as token sequences is a very straightforward and cannot be said to build upon previous research. The extent to which \"learning as a token sequence\" and \"learning through autoregressive next-token prediction\" individually contribute remains unclear, and the evidence for each aspect appears somewhat insufficient. At the very least, there should be a discussion on how this method relates to existing token-based approaches—such as those highlighted in well-cited reviews like https://arxiv.org/abs/2302.04181—or, if specializing in molecular graphs, a comparison with simpler methods that feed traditional symbol-sequence representations of molecules like SMILES or SELFIES to Transformers.", "methods_and_evaluation_criteria": "While traditional graph generation models have primarily been adjacency matrix-based, this approach treats node and edge lists as token sequences and learns them efficiently using an autoregressive Transformer (Transformer Decoder). The paper notes that while early research on graph generation started with naive autoregressive methods, many recent high-performance models, such as those based on discrete diffusion, generate adjacency matrices. This makes it technically interesting to examine whether the autoregressive approach remains effective. Additionally, the study evaluates the method using two tasks: goal-oriented graph generation and property prediction. The evaluation also extends beyond molecular datasets to include generic datasets.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental datasets and tasks are well-balanced, incorporating molecular generation benchmarks such as MOSES and GuacaMol, as well as generic datasets. However, the QM9 dataset contains molecular data with 3D coordinates, which may not be the best fit for the proposed method. A more careful consideration of this choice may be necessary.", "relation_to_broader_scientific_literature": "While traditional graph generation models have primarily been adjacency matrix-based, this approach treats node and edge lists as token sequences and learns them efficiently using an autoregressive Transformer (Transformer Decoder). The paper notes that while early research on graph generation started with naive autoregressive methods, many recent high-performance models, such as those based on discrete diffusion, generate adjacency matrices. This makes it technically interesting to examine whether the autoregressive approach remains effective.", "essential_references_not_discussed": "There should be a discussion on how this method relates to existing token-based approaches—such as those highlighted in well-cited reviews like https://arxiv.org/abs/2302.04181—or, if specializing in molecular graphs, a comparison with simpler methods that just feed widely-established symbol-sequence representations of molecules like SMILES or SELFIES to Transformers.\n\nAttending to Graph Transformers\nhttps://arxiv.org/abs/2302.04181\n\nSELFIES and the future of molecular string representations\nhttps://doi.org/10.1016/j.patter.2022.100588\n\nTransformer-based models for chemical SMILES representation: A comprehensive literature review\nhttps://doi.org/10.1016/j.heliyon.2024.e39038\n\nWhile this paper also evaluates generic tasks, its main focus is on molecular graph generation. Given that SMILES (and SELFIES) representations have a long history and are widely used for molecular representation, a natural question arises: why not simply input these representations into a Transformer? In fact, similar proposals have been repeatedly explored in the machine learning field, and discussions from the following paper and its peer reviews could provide useful insights. (**This doesn't mean the paper should cite the following papers; Just for information to make sure the main discussion points along this kind of research.**)\n\nSmilesFormer: Language Model for Molecular Design\nhttps://openreview.net/forum?id=VBQZkYu22G\n\nSELFIES-TED : A Robust Transformer Model for Molecular Representation using SELFIES\nhttps://openreview.net/forum?id=uPj9oBH80V", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PHEzg0kgIN", "reviewer": "Reviewer_jm1Y", "summary": "This paper introduces the graph generative pre-trained transformer (G2PT) for graph generation using auto-regressive transformers. The method introduces a sequence-based graph representation approach, which fits well to transformer architectures originally developed for NLP. The paper explores fine-tuning methods for downstream applications, e.g., goal-oriented graph generation and property prediction. Experiments demonstrate that G2PT achieves SOTA performance across several benchmarks, including generic graphs and molecular graphs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive suggestion! We address the concerns below. \n\n---\n\n**Q1**. _I would suggest adding a more detailed introduction/explanation for Table 2. From the title and text, it's not straightforward to know what tasks are performed there._\n\n**A1**. Thanks for pointing this out! Since the experiment settings are standard and due to the page limit, we move most of the experiment details to the appendix. We will provide add a more detailed introduction for Table 2 in the next version. \n\n---\n\n**Q2**. _​​There are a lot of zeros in Table 2. Is it possible that the model overfits?_\n\n**A2**. We report the V.U.N metric in our table. Specifically, this is computed by counting the percentage of generated graphs being valid, unique and novel. The novelty metric indicates whether the samples are different from the training graphs. Therefore, the model performs well not just by memorizing the training graph. Moreover, they are not exact zeros but we round it to 4 decimals (potentially a value of 1e-5  during evaluation). The values are so small may be due to the nature of  the graph statistics or simply because they are very easy to be captured by the model.\n\n---\n\nWe hope that we have addressed your concerns, thanks!", "claims_and_evidence": "Claims made in the submission look good to me.", "methods_and_evaluation_criteria": "Yes, they make sense to me.", "theoretical_claims": "The major theoretical claim is that maximizing the sequence likelihood serves as maximizing a lower bound on the true graph likelihood, which is pretty clear to me.", "experimental_designs_or_analyses": "1. I would suggest to add more detailed introduction/explanation for Table 2. From the title and text, it's not straightforward to know what tasks are performed there.\n\n2. There are a lot of zeros in Table 2. Is it possible that the model overfits?", "relation_to_broader_scientific_literature": "The graph generation task targeted by the paper is a big domain, including a few classic works such as GraphRNN, GRAN. Tokenizing graph is well motivated given the success of LLMs.", "essential_references_not_discussed": "I don't see essential references not discussed by the paper.", "other_strengths_and_weaknesses": "The paper is easy to follow. The method proposed by the paper is straightforward yet effective, and complexity could be reduced by sequential representation compared to using adjacency matrix. The fine-tuning approaches (e.g., rejection sampling, reinforcement learning) are thoroughly explored and clearly demonstrated through experiments on goal-oriented molecule generation.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please see my comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on graph generation.  The paper proposes a straight-forward yet effective method for graph generation.  The approaches centers on a new way to represent a graph as sequence of tokens, that contains both node definitions and edge definitions.  Generation is achieved via an auto-regressive framework on this representation.

The paper had divergent reviews after the rebuttal phase.  The reviewers appreciated the simplicity and effectiveness of the framework.  The results on standard benchmarks (Planar, Tree, Lobster, and SBM) were solid.  Concerns were about the results on molecular datasets and comparisons to standard string-based graph representations.

On balance, the contribution is sufficient for ICML, though with concerns over the overall magnitude of contribution.
Decision: Accept (poster)