Title: Flatten Graphs as Sequences: Transformers are Scalable Graph Generators
Venue: NeurIPS 2025
Accepted: yes
Keywords: graph generation

transformers

autoregressive modeling

language models

LLMs
Abstract: We introduce AutoGraph, a scalable autoregressive model for attributed graph generation using decoder-only transformers. By flattening graphs into random sequences of tokens through a reversible process, AutoGraph enables modeling graphs as sequences without relying on additional node features that are expensive to compute, in contrast to diffusion-based approaches. This results in sampling complexity and sequence lengths that scale optimally linearly with the number of edges, making it scalable and efficient for large, sparse graphs. A key success factor of AutoGraph is that its sequence prefixes represent induced subgraphs, creating a direct link to sub-sentences in language modeling. Empirically, AutoGraph achieves state-of-the-art performance on synthetic and molecular benchmarks, with up to 100x faster generation and 3x faster training than leading diffusion models. It also supports substructure-conditioned generation without fine-tuning and shows promising transferability, bridging language modeling and graph generation to lay the groundwork for graph foundation models. Our code is available at https://github.com/BorgwardtLab/AutoGraph.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of **scalable and structurally valid graph generation**, a task that remains challenging for existing diffusion-based and autoregressive (AR) models due to either quadratic-to-cubic complexity or limited long-range consistency. The authors propose **AUTOGRAPH**, a framework that bridges graph generation with large language models (LLMs) through a novel **graph-to-sequence transformation**:\n\n1. **Segmented Eulerian Neighborhood Trails (SENTs)** – an extension of segmented Eulerian trails that augments each trail segment with local neighborhood tokens, ensuring complete edge coverage while preserving subgraph-induced structures.\n2. **Lossless flattening and tokenization** – a formal procedure that converts SENTs into token sequences with special delimiters, accompanied by theoretical proofs guaranteeing one-to-one recoverability between graphs and sequences.\n3. **Autoregressive language modeling** – any decoder-only transformer can now treat graph generation as next-token prediction, yielding *O(n)* time and memory complexity via an efficient SENT sampling algorithm.\n4. **Empirical validation** – using a 12-layer LLaMA backbone, AUTOGRAPH achieves:\n   - Up to **100× inference speed-up** and **3× training speed-up** over diffusion baselines on Planar and SBM datasets.\n   - Competitive performance on transfer learning and substructure-conditioned generation, demonstrating the method’s flexibility.\n\nOverall, AUTOGRAPH offers a principled and efficient pathway to leverage modern LLMs for graph generation, effectively mitigating negative transfer issues and outperforming prior diffusion and AR approaches in both accuracy and scalability.\n\nSTRENGTHS AND WEAKNESSES:\n------\n\n### Strengths\n\n- **S1. Lossless graph-to-sequence transformation**  \n  The paper moves beyond prior random-walk-based graph representations by introducing a **lossless** conversion—Segmented Eulerian Neighborhood Trails (SENTs)—that preserves complete edge coverage and subgraph-induced structures. This theoretically grounded mapping enables direct, reversible use of language models for graph generation without sacrificing structural validity.\n\n-----\n\n- **S2. Comprehensive empirical evaluation**  \n  The authors conduct meticulous head-to-head comparisons against both diffusion-based and conventional autoregressive baselines across four benchmarks encompassing small, medium, and large graphs. Results consistently demonstrate AUTOGRAPH’s superiority in accuracy (MMD, VUN) and efficiency (up to 100× faster inference), underscoring the method’s robustness to graph size and domain diversity.\n\n-----\n\n- **S3. Formal guarantees and linear-time sampling**  \n  Theoretical proofs establish a one-to-one correspondence between SENT sequences and their source graphs, while the proposed sampling algorithm operates in linear time $\\(O(m)\\)$ with respect to edge count—crucial for scalability.\n\n---\n\n## Weaknesses\n\nThese are elaborated under the Questions section below.\n\nQUESTIONS:\nWhile the manuscript exhibits many notable strengths, I would appreciate clarification on the following points:\n\n-----\n\n1. **Positioning of prior work on graph-to-sequence representations**  \n   Introduction section occasionally reads as if no effective graph-to-sequence approach existed before this work. However, random-walk-based representations are acknowledged later in the Related Work section. It would be clearer—and fairer to prior art—if Introduction section explicitly positioned SENT as an improved (lossless and generation-oriented) extension of these earlier methods, rather than implying the absence of any sequence paradigm.\n\n-----\n\n2. **Interpretability of evaluation metrics**  \n   Although Maximum Mean Discrepancy (MMD) and VUN metrics are standard, readers unfamiliar with graph generation benchmarks may struggle to **intuitively grasp what each metric indicates**. **A brief high-level explanation** of the intuition behind each metric, perhaps in an appendix footnote or a short paragraph, would greatly improve result interpretability.\n\n-----\n\n3. **Speed comparison beyond diffusion baselines**  \n   The efficiency analysis highlights impressive speed-ups over diffusion-based models (Table 2). Is it feasible to provide analogous wall-clock comparisons against other baselines? If not, could the authors explain the practical obstacles? Such numbers would further substantiate AUTOGRAPH’s scalability claims across paradigms.\n\n-----\n\n4. **Validity scores in molecular generation (Table 4)**  \n   In the molecular benchmarks, AUTOGRAPH achieves strong uniqueness and novelty but appears to match—or slightly underperform—some baselines in validity. To the best of my knowledge, the validity is the most critical criterion for molecule generation, so the authors need to report the additional discussion.\n\n---\n\nAddressing these points would further strengthen the paper’s clarity and persuasiveness.\n\nLIMITATIONS:\nAll key limitations have been included within the Questions above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a graph generation method called AutoGraph, which serializes graph structures by treating sampled graph trials as tokens, enabling the use of decoder-only Transformer models for graph generation tasks. The authors introduce a newly designed structure named SENT (Segmented Eulerian Neighborhood Trail) to ensure that the prefix of the generated sequence corresponds to an induced subgraph. Thanks to these innovations, the proposed model demonstrates efficient generation, strong transferability, and controllable substructure conditioned generation capabilities across multiple synthetic and real-world graph datasets. Compared to diffusion-based models, the proposed approach achieves 100× faster generation and 3× faster training.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The authors propose AutoGraph, the first method that serializes graph structures to enable direct application of language models to graph generation.\n2. The authors carefully design the SENT structure and theoretically prove that sampling SENTs allows for graph serialization, while ensuring that the prefix of the resulting sequence corresponds to an induced subgraph.\n3. The chosen method of serializing and tokenizing graphs using SENT guarantees that isomorphic graphs are mapped to identical token sequences, which facilitates better generalization.\n4. The proposed random trial sampling algorithm with breaks is cleverly designed and enables efficient sampling of SENTs.\n5. AutoGraph achieves significant inference and training speedups while maintaining almost state-of-the-art generation quality, demonstrating strong scalability.\n6. AutoGraph also shows impressive transferability and substructure conditioned generation capabilities without requiring additional finetuning.\n### Weaknesses\n1. The main advantage of the proposed AutoGraph method lies in its generation speed, while it shows some limitations in generation quality and does not consistently outperform other methods across all evaluation metrics:\n* As shown in Table 1, AutoGraph achieves the best VUN performance on the SMB dataset. However, it underperforms compared to methods like DiGress and GRUM in metrics such as node degree distributions (Deg.), clustering coefficients (Clus.), and orbit count statistics (Orbit). This may indicate that AutoGraph emphasizes the structural feasibility of the generated graphs.\n* According to Table 4, AutoGraph scores lower than DiGress and JT-VAE on the FCD metric in the MOSES dataset, and lower than MCTS on the VALID metric in the GuacaMol dataset. This suggests that AutoGraph still has room for improvement in generating realistic molecular distributions.\n2. The interpretability of the token sequences generated by AutoGraph is less intuitive compared to representations like SMILES, and may not be suitable for all types of graph structures.\n3. For dense graphs, where each node may have a large number of neighbors, the authors’ strategy of explicitly encoding adjacent nodes could lead to an excessive number of tokens, potentially affecting generation efficiency.\n\nQUESTIONS:\n1. What is the difference between AutoGraph-Graph and AUTOGRAPH (Pretrained) in the last two rows of Table 4?\n2. Currently, AutoGraph uses a 12-layer LLaMA model to align with GPT-2’s smallest variant, which is still relatively small in scale. Have the authors evaluated how AutoGraph scales with larger language models?\n3. Does the proposed method support generation conditioned on combinations of different motifs as substructure condition? Currently, the experiments only demonstrate generation conditioned on repetitive occurrences of the same motif.\n\nLIMITATIONS:\nThe authors note that their experiments are still conducted on relatively small-scale benchmarks and attribute this limitation to the current lack of comprehensive graph generation benchmarks and well-curated pre-training datasets in the community.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is focused on the topic of graph generation using Transformers and uses an existing principle of recent interest that transforms graphs to sequences. It presents AUTOGRAPH, an autoregressive graph generation framework that converts graphs into token sequences to enable direct use of decoder-only Transformers for efficient and scalable generation. The key to their approach in this paper is the concept of Segmented Eulerian Neighborhood Trails (SENTs) which is a reversible graph to sequence transformation that preserves topological structure while producing token sequences applicable or similar to language modeling. This method also ensures subgraph-aware information. The paper demonstrates the effectiveness of AUTOGRAPH on synthetic and molecular benchmarks, showing improved efficiency, performance gains, and support for substructure-conditioned generation without fine-tuning. As such, this work aims to bridge the gap between graph generation and language modeling and perhaps addresses improvements on methods concerned with flattening graphs to sequences which can be well-defined and lossless.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- A key strength of this paper is their progress on graph to sequence conversion methods which has been of much interest in recent literature. \n- The flattening/tokenization process involves a more principled encoding using eulerian trails where each token corresponds to a structurally interpretable graph event. This is in contrast to multiple prior works which may use heuristics, neighborhood or hop encoding and templates that are manually engineered and target for use of those sequences as prompts in LLMs.\n- The SENT sequences based method provides improved validity in graph generation.\n- The construction of SENT sequences are transparent and interpretable, and is complemented with theoretical analysis.\n- The proposed method achieves significantly faster inference and training speedups compared to previous methods.  \n\nWeaknesses:\n- The paper includes an extension of SENT tokenization to (discrete) attributed graphs, but many graph tasks may be non-discrete features.\n- While this is not a weakness per se and maybe not the focus of this paper, the proposed graph sequence approach has not been evaluated on general graph tasks which prior graph tokenization approaches seem to apply to.\n- Closely related recent works on graph to sequence methods such as [1], [2] are not discussed.\n\n[1] Fatemi, B., Halcrow, J. and Perozzi, B., 2023. Talk like a graph: Encoding graphs for large language models. \n[2] Chen, R., Zhao, T., Jaiswal, A., Shah, N. and Wang, Z., 2024. Llaga: Large language and graph assistant.\n\nQUESTIONS:\n- Can the proposed SENT based graph to sequence representation be adapted for general graph modeling/understanding/downstream tasks (e.g., classification, regression)?\n- If yes, how can the SENT sequencing apply to large size graphs where generally subgraph or partial graph is presented in a GNN network in existing works?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents AUTOGRAPH, an autoregressive framework for generating graphs using decoder-only transformers. The core contribution lies in a reversible flattening procedure that transforms graphs into sequences, enabling the use of language modeling. The authors claim that this approach is scalable and demonstrate its effectiveness across various synthetic and molecular graph generation benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\n- The paper is clearly written and well-organized.\n- It provides comprehensive experimental evaluations across four general graph generation benchmarks and two molecular graph generation benchmarks.\n\n[Weaknesses]\n\n- **Overstatement of novelty**: The claim that “AUTOGRAPH establishes the first fundamental connection between graph and language modeling” (in the conclusion) appears overstated. Prior works such as [1,2,3] have already employed decoder-only transformers for attributed graph generation. In particular, GEEL [1] shares a similar motivation and framework to AUTOGRAPH.\n- **Omission of baseline metrics**: The omission of MMD metrics for GEEL in Tables 1 and 3 is concerning, especially since the original GEEL paper includes these values for the same benchmarks. Given that GEEL is a highly relevant work as it shares the same motivation and is a walk-based autoregressive graph generative model, its complete evaluation should be included for fair comparison.\n- **Insufficient implementation details**: Key implementation details are missing, such as which version of LLaMA was used, the number of training epochs, and whether the codebase will be made publicly available.\n\nQUESTIONS:\n- **Clarification on token definition**: The paper states that graphs are “composed of tokens” (line 37) and that it “explicitly represents graphs as token sequences” (lines 230–232). Could you clarify what is meant by “tokens” in this context? For instance, GEEL [1] represents each edge as a token—arguably a more direct analogy to tokens in language modeling—without requiring specialized architectures. How does AUTOGRAPH’s node-based representation differ from or improve upon this? Additionally, what does it mean to “interpret the sequences as language” (lines 187–189)?\n- **Pre-training data scale**: The graph generation benchmarks used for training involve relatively small datasets to train LLMs. Was this data volume sufficient for pre-training a large model like LLaMA? How did you address the challenges posed by the limited scale of training data?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}